New web application crawler

Viewing 4 reply threads
  • Author
    • #5099


      I am about 60% done creating a new web application crawler. My goal is not to replace Nikto or any other tool for that matter, but I am more thinking about an application that we can use in the reconnaissance/information gathering phase.

      So far, all my prototypes have been successful. Some of tasks it performs are already done by existing tools, and I know that. My goal is to put some of these existing functionalities plus many new ones in one single tool.

      It will hae a GUI and a command line interface. Could be multi-threaded later if people like it. Finally, I will give it for free!

      So here are the tasks it can/will do against a web site:
      – Create a wordlist
      – Find all emails, telephone numbers, fax numbers, etc
      – Find names and guest possible usernames based on email address
      – Find broken links
      – Create a site structure
      – Display robot.txt file details
      – Find typical html files
      – Identify all forms
      – Search for login screens

      My questions are:
      1) What else would you find useful from a tool like this?
      2) Output format from the command line version?

      I want to launch a usefull tool. Not a script or two…

      Thanks for you comments!

    • #32366

      Well, if you are finding forms, I’d like it to enumerate information about those forms.  I like to know going in whether the form is POST or GET (a single GET in a mess of POST forms may indicate someone moved development code to production).  I also like to know all the fields, names, types, and values (especially hidden fields).  I’d like it to crawl any page with a form a couple of times some user defined time interval apart to see if any pre-defined values for hidden fields change (these are great fuzzing targets).  That’s about all I can think of at this late hour.

    • #32367

      Thanks former33t for your comment.

      It makes sense and I had not thought about this before. I have an idea on what I could do to report this kind of data.

      Anyone else?

    • #32368

      Let me know when it is ready for testing.

    • #32369

      I will post on this forum when it will be in Beta. Probably August or September.

      Thanks former33t.

Viewing 4 reply threads
  • You must be logged in to reply to this topic.

Copyright ©2021 Caendra, Inc.

Contact Us

Thoughts, suggestions, issues? Send us an email, and we'll get back to you.


Sign in with Caendra

Forgot password?Sign up

Forgot your details?