Hi Guys

I didnt know where to post this so I forgive me if this the wrong section. Anyway, over the last few days my bro's win7 was doing all sorts of crazy things so I decided to wipe the drive and do a fresh install. Today I left my PC on whilst doing some work (winXP x64), shortly afterwards it also acted in a strange manner. I suspect my comp has been hacked since I have some dodgy files on my desktop. Ive done a virus/spyware scan and it hasnt revealed anything interesting. I checked my wireshark logs and they showed alot of 'encrypted' data being sent around. There were some plaintext links too to rapidshare sites etc. Im in the process of find out exactly what has happened and how it happened.
Having said that I was wondering if there are any tools which I can use to analyse the system for rootkits/file modification and what system calls are being made etc ?

I have to second Helix. I used to use it at work, but haven't had to do any forensic work for a while. The user manual is really useful too.

I don't know if it will help you, but I usually try "netstat -b" in the cmd prompt. This will list executables that create connections or listen on ports. Of course that won't clean up your computer, but you might be able to close the connections and stop the downloads.

My general opinion is that if you truly think you've got a rootkit, you should be fdisk'ing and reinstalling.  As Ketchup noted, there are some rootkits that hide themselves VERY well.  I ran into one, not long ago, that I was asked to 'try to remove' before wiping the disk.  Not only did it really fight me, even spotting it, but it was morphing itself at a pretty quick clip, before I could even begin removal.  I ended up showing the 'owner' what was going on, and they quickly understood the gravity of it, and yielded to a disk wipe.

The only really successful way I wipe rootkits (and I STILL feel safer wiping and starting over, as you never know what signatures might not have been included yet in rootkit finding tools, etc) is to put the hard disk or media in an external enclosure or connection, and scan it from a known good machine, pretty much as Ketchup noted in his post, and even then, there's not always a guarantee you'll clean it up.  I'm not saying you can't remove a rootkit, etc, with relative success, but I find it easier, personally, to just boot the box to a bootable Linux distro, get whatever data I feel is relevant and burn it or copy it to external media, then wipe the box and start over.  It's safer, and you run less risk of missing something, for which you might pay a heavier price later.

I VERY rarely suggest that someone simply remove a rootkit and go on about their business, without wiping the drive or media and starting over.  Just not worth the trouble and risk.

My opinion, anyway, for what it's worth.

I've heard of them, and talked to one guy who supposedly hit one, once...  He managed to flash the BIOS, and 'thinks' he cleared it.  But I have no experience with any that went to the BIOS.

there are rootkits that install themselves in either the bios or Master Boot Record. even a fresh install wont help against these. i guess switching to linux is the best choice . my advice would be to flash the bios, do a low level format and try again with a fresh install...or buy a new rig 

Regardless of infection, I would attempt to establish what family of malware was installed, if any. This will give you an indication of what type of data may be compromised. Some infections will steal your logins, msn logs, cookies etc. Some may steal your chinese game login info... Either way, you can then make an informed decision as to what type of data was compromised or will be in the future (bank details).

If the value of such data outweighs the value of data on PC. Boot to live cd, copy important files (unless it was a file infector!) and reformat...

Alternatively, disinfect with malwarebytes, AVs, onlive AVs other malware removers and continue with the thought that something is more than likely still there!