Directory Bruteforcing

One thing you learn when you start a career pentesting is:

Never assume anything.

In my experience hacks aren’t always elegant and elaborate. Sometimes something simple and effective is your avenue of penetration. Which brings us to today’s topic: directory bruteforcing.

Directory bruteforcing is a favorite of mine.  I can’t tell you how many times a directory listing has broken open a pentest for me.  Whether it be that all elusive web admin panel, or a directory listing containing a database with passwords, there’s almost always something hiding beneath that tidy little web server.

Before we start bashing away, and let’s be honest here that’s what we’re doing, we have to mention that this kind of enumeration can get you blacklisted. In fact if your scope doesn’t have you whitelisted for the engagement, we recommend you scan low and slow to get a feel for the targets response. It never hurts to have a backup IP (or a few) to scan from as well.  Dealing with customer blacklisting is a pain.

Today, we’re showcasing python based Wfuzz by Edge-Security and Java based Dirbuster maintained by the OWASP project. Both are excellent directory and file brute forcing tools that come complete with lists of common (and sometimes not so common) directories or files. Both support recursion, multi-threading, and output to useful file formats. They are also great about inherent false positive detection and support proxies… excellent. We use Wfuzz on our *nix boxes and Dirbuster from Windows. We interchange lists frequently.

We hate to regurgitate verbatim but Wfuzz actually gives pretty good usage and feature documentation that can be seen on their website here.

“Wfuzz is a tool designed for bruteforcing Web Applications, it can be used for finding resources not linked (directories, servlets, scripts, etc), bruteforce GET and POST parameters for checking different kind of injections (SQL, XSS, LDAP,etc), bruteforce Forms parameters (User/Password), Fuzzing,etc.”

Usage:

# wfuzz.py -c -z file -f wordlists/commons.txt --hc 404 --html http://www.mypentesttarget.com/FUZZ 2> results.html

This does a basic directory bruteforce against http://mypentesttarget.com/ throwing http GETs to the web server matching every line in the wordlists/commons.txt file. It strips out the 404 not found responses and sends the output to an HTML file for later usage.

Wfuzz is actually a far more robust tool allowing you to fuzz web parameters to identify SQL injection, XSS, and bruteforce usernames and passwords. The lists for these injection strings are included with wfuzz. We will showcase Wfuzz in more detail in a future write-up.

Dirbuster is very similar. It uses a pretty java GUI that allows you to specify number of threads and tune the amount of threads on the fly (which is actually really handy). It also supports pausing which is useful. Another great feature it offers is selective recursion. If Dirbuster finds a  directory it will automagically queue it for recursive scanning, but if we want to skip that directory  we can un-check the tick box next to it and change this on the fly. The GUI itself is pretty self explanatory and you can see basic usage in the video.

So what are we looking for?

Some of our favorites are:

    * Jboss admin panels
    * Backend web administration (think VPN, firewall, and website management logins)
    * OWA servers
    * Frontpage Config Files
    * Citrix Portals
    * Directories with databases
    * Directories with readable web app scripts
    * Webcam portals
    * Development/stage versions of software/sites
    * Default PHP Config files
    * 401 credential protected directories
    * Directories containing documents we can mine for metadata
    * Scripts we can fiddle with (list below)

For file types we wanna look for things like scripts we might be able to manipulate, log files, etc:

    * .log, .phtml, .php, .php3, .php4, .php5, .inc, .asp, .aspx, .pl, .pm, .cgi, .lib, .jsp, .jspx, .jsw, .jsv, .jspf, .cfm, .cfml, .cfc, .dbm, .mdb

Even resources that give you 403 Forbidden responses are valuable in identifying the web server’s structure and the apps that run on it.

Earlier we said that we interchange lists. Here’s why. Below are the sizes, in words, of the lists supplied with Wfuzz and Dirbuster (as well as another favorite tool of ours Grendel Scan). The whopping difference here is Dirbuster’s lists are huge comparatively. The reason for this is that Dirbuster uses a large number of numeric only resource requests. Dirbuster also seems to really take the word “bruteforce” to heart requesting less than technical directory names. We’d love to say “Use X list over Y list” but we really can’t. We have garnered valuable findings from all these lists. If you aren’t under the blacklisting/shunning gun per-se you can cat these into a “masterdirs” file and then sort and uniq it. Just be aware that these lists are unordered on purpose to be optimized, if we have the time to complete the full list then it doesn’t matter, but if you have a short testing time frame it will.

Wfuzz    common 947    medium 1660    big 3037    
Dirbuster    small 81643    medium 207631    big 1185252    

Grendel Scan    Small 100    Medium 300    Large 500    XL 819

In our video we show the basic usage of both tools.

If you’re working with a scope that limits tools you can install, or you want to comb over some could-be false positives from a tools output you can do this by using a bash script (be easy on our Bash foo!)

# cat dircurl.sh

#!/usr/bin/bash

if [[ $# -ne 2 ]]; then
echo "usage: $0 directorylist www.target.com"
exit
fi

for i in $(cat $1)
do echo -ne "directory: "
echo -ne $i
echo -ne "\t"
echo -ne "count: "
echo -ne `curl $2/$i 2> /dev/null | wc -l`
echo
done

This does a Curl request to each line in the supplied “directorylist” to the “target.com” and then does a wordcount (wc -l) on it. Look at the output, what is the most common response?

# bash dircurl.sh scanneroutput www.securityaegis.com

directory: sitemap      count: 266
directory: archives     count: 266
directory: wp-admin     count: 7
directory: links        count: 0
directory: login        count: 266
directory: articles     count: 266
directory: support      count: 266
directory: keygen       count: 266
directory: article      count: 266
directory: help count: 266
directory: events       count: 266
directory: archive      count: 266
directory: register     count: 266
directory: en   count: 266
directory: forum        count: 266
directory: wp-includes  count: 7
directory: software     count: 266
directory: downloads    count: 266
directory: security     count: 0
directory: category     count: 266
directory: content      count: 266
directory: main count: 266
directory: press        count: 266
directory: media        count: 266
directory: templates    count: 266
directory: services     count: 266
directory: icons        count: 266
directory: wp-content   count: 7
directory: resources    count: 0
directory: info count: 0
directory: overnment    count: 266
directory: corrections  count: 266
directory: ajax count: 266
directory: icom_includes        count: 266
directory: rules        count: 266
directory: tr   count: 266
directory: server       count: 266
directory: mirrors      count: 266
directory: government    count: 266
directory: corrections  count: 266

Looks like my error page (or in some cases my redirects) have about 266 newlines. Lets pipe that into grep -v 266, removing all lines containing 266:

# bash dircurl.sh scanneroutput www.securityaegis.com |grep -v 266

directory: wp-admin     count: 7
directory: links        count: 0
directory: wp-includes  count: 7
directory: security     count: 0
directory: wp-content   count: 7
directory: resources    count: 0
directory: info count: 0

This gives us a good place to start poking.

Thanks go to David, Paul, and Nate from the Redspin Team, and of course Mike Kelly (Laz3r) for his contributions on the video =)

NMAP Database Output : XML TO SQL

SQL support has been a much requested feature of NMAP in the Redspin office. While a number of tools exist to support NMAP SQL output, their database format has left much to be desired. Using SQLite, Perl’s DB and the NMAP Parser module, our tool extracts all supported fields in an NMAP XML file and creates the following database format:

TABLE nmap (
    sid INTEGER PRIMARY KEY AUTOINCREMENT,
    version TEXT,
    xmlversion TEXT,
    args TEXT,
    types TEXT,
    starttime INTEGER,
    startstr TEXT,
    endtime INTEGER,
    endstr TEXT,
    numservices INTEGER)

TABLE hosts (
    sid INTEGER,
    hid INTEGER PRIMARY KEY AUTOINCREMENT,
    ip4 TEXT,
    ip4num INTEGER,
    hostname TEXT,
    status TEXT,
    tcpcount INTEGER,
    udpcount INTEGER,
    mac TEXT,
    vendor TEXT,
    ip6 TEXT,
    distance INTEGER,
    uptime TEXT,
    upstr TEXT)

TABLE sequencing (
    hid INTEGER,
    tcpclass TEXT,
    tcpindex TEXT,
    tcpvalues TEXT,
    ipclass TEXT,
    ipvalues TEXT,
    tcptclass TEXT,
    tcptvalues TEXT)

TABLE ports (
    hid INTEGER,
    port INTEGER,
    type TEXT,
    state TEXT,
    name TEXT,
    tunnel TEXT,
    product TEXT,
    version TEXT,
    extra TEXT,
    confidence INTEGER,
    method TEXT,
    proto TEXT,
    owner TEXT,
    rpcnum TEXT,
    fingerprint TEXT)

TABLE os (
    hid INTEGER,
    name TEXT,
    family TEXT,
    generation TEXT,
    type TEXT,
    vendor TEXT,
    accuracy INTEGER)

The resulting database can then be queried directly using SQLite in order to extract the relevant information. The tool also prints output in the following format sorted by IP and PORT as show below:

$ nmap -A -T4 scanme.nmap.org -oX scanme >/dev/null
$ nmap_xml2sql.pl scanme | grep -v "^#"
64.13.134.52 (scanme.nmap.org)   53/tcp   domain
64.13.134.52 (scanme.nmap.org)   80/tcp   http

Compare this database structure with the other similar XML2SQL approaches:
NMAP-SQL: http://sourceforge.net/projects/nmapsql/ outdated – only supports NMAP 3.75 and MySQL:

table portstat, table runlist, table targets

NMAP-Parser (nmap2db.pl): http://search.cpan.org/dist/Nmap-Parser/ MySQL and SQLite support:

table hosts(
  ip              VARCHAR(15) PRIMARY KEY NOT NULL,
  mac             VARCHAR(17),
  status          VARCHAR(7) DEFAULT 'down',
  hostname        TEXT,
  open_ports      TEXT,
  filtered_ports  TEXT,
  osname         TEXT,
  osfamily        TEXT,
  osgen           TEXT,
  last_scanned    TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE (ip))

PBNJ -x option: http://pbnj.sourceforge.net/ MySQL and SQLite support:

table machines(
  mid             PRIMARY KEY AUTOINCREMENT,
  ip                TEXT,
  host             TEXT,
  localh           INTEGER,
  os            TEXT,
  machine_created TEXT,
  created_on      TEXT)

table services(
  mid             INTEGER,
  service         TEXT,
  state             TEXT,
  port           INTEGER,
  protocol         TEXT,
  version       TEXT,
  banner         TEXT,
  machine_updated TEXT,
  updated_on      TEXT)

The tool has been released under a NMAP equivalent license as well as a Fyodor may-use-as-he-pleases license, and can be download here: nmap_xml2sql.pl

Requirements:

    * Recent Version of Perl: http://www.perl.org/
    * Nmap Parser: http://search.cpan.org/dist/Nmap-Parser/
    * Perl DB: http://dbi.perl.org/
    * SQLite: http://www.sqlite.org/

            Our hope is that SQL support is added to NMAP eventually; until then, this script serves our purposes. Let us know if you use it for your project or make any changes.