.

New web application crawler

<<

caissyd

User avatar

Hero Member
Hero Member

Posts: 894

Joined: Thu Dec 31, 2009 11:20 am

Location: Ottawa, Canada

Post Tue May 25, 2010 6:24 pm

New web application crawler

Hey,

I am about 60% done creating a new web application crawler. My goal is not to replace Nikto or any other tool for that matter, but I am more thinking about an application that we can use in the reconnaissance/information gathering phase.

So far, all my prototypes have been successful. Some of tasks it performs are already done by existing tools, and I know that. My goal is to put some of these existing functionalities plus many new ones in one single tool.

It will hae a GUI and a command line interface. Could be multi-threaded later if people like it. Finally, I will give it for free!

So here are the tasks it can/will do against a web site:
- Create a wordlist
- Find all emails, telephone numbers, fax numbers, etc
- Find names and guest possible usernames based on email address
- Find broken links
- Create a site structure
- Display robot.txt file details
- Find typical html files
- Identify all forms
- Search for login screens

My questions are:
1) What else would you find useful from a tool like this?
2) Output format from the command line version?

I want to launch a usefull tool. Not a script or two...

Thanks for you comments!
OSCP, GPEN, GWAPT, GSEC, CEH, CISSP
(aka H1t.M0nk3y)
<<

former33t

Full Member
Full Member

Posts: 226

Joined: Sat Feb 14, 2009 12:33 am

Post Tue May 25, 2010 10:10 pm

Re: New web application crawler

Well, if you are finding forms, I'd like it to enumerate information about those forms.  I like to know going in whether the form is POST or GET (a single GET in a mess of POST forms may indicate someone moved development code to production).  I also like to know all the fields, names, types, and values (especially hidden fields).  I'd like it to crawl any page with a form a couple of times some user defined time interval apart to see if any pre-defined values for hidden fields change (these are great fuzzing targets).  That's about all I can think of at this late hour.
Certifications: CREA, MCSE: Security, CCNA, Security+, other junk
<<

caissyd

User avatar

Hero Member
Hero Member

Posts: 894

Joined: Thu Dec 31, 2009 11:20 am

Location: Ottawa, Canada

Post Wed May 26, 2010 6:46 am

Re: New web application crawler

Thanks former33t for your comment.

It makes sense and I had not thought about this before. I have an idea on what I could do to report this kind of data.

Anyone else?
OSCP, GPEN, GWAPT, GSEC, CEH, CISSP
(aka H1t.M0nk3y)
<<

former33t

Full Member
Full Member

Posts: 226

Joined: Sat Feb 14, 2009 12:33 am

Post Wed May 26, 2010 7:40 pm

Re: New web application crawler

Let me know when it is ready for testing.
Certifications: CREA, MCSE: Security, CCNA, Security+, other junk
<<

caissyd

User avatar

Hero Member
Hero Member

Posts: 894

Joined: Thu Dec 31, 2009 11:20 am

Location: Ottawa, Canada

Post Thu May 27, 2010 6:35 am

Re: New web application crawler

I will post on this forum when it will be in Beta. Probably August or September.

Thanks former33t.
OSCP, GPEN, GWAPT, GSEC, CEH, CISSP
(aka H1t.M0nk3y)

Return to Web Applications

Who is online

Users browsing this forum: No registered users and 1 guest

.
Powered by phpBB® Forum Software © phpBB Group.
Designed by ST Software