Two days ago OWASP (http://www.owasp.org) announced the release of a new version of their DirBuster tool.  DirBuster is a Java based web application scanner.  Basically you give it a host and it scans that host for directories on the host.  DirBuster can utilize a list of directories and files or it can brute force them.  DirBuster is nice because it can find files directories that might not be directly linked to.  This can be used to expose information on the host that you might not find otherwise.  DirBuster will also parse the HTML of files that it does discover, allowing it to follow links present in discoverable files as well.  You can find more information about DirBuster at the OWASP site at https://www.owasp.org/index.php/Category:OWASP_DirBuster_Project.