I've come across numerous servers over the past month which have been compromised by perl.Bossworm (CVE-2010-0738). All the servers had the same behaviour; they all had hundreds of instances of the pnscan port scanner running looking for JBoss in the HTTP headers of /16 address blocks and dumping them to /tmp. They all had the fly.pl, linda.pl, line.pl and other files associated with this worm all doing their thing.
But one server which was running RHEL 5.4 had some other strange behaviour which was not on the other servers and which I've never seen before. Unfortunately this was the first server that I'd seen with this worm and so I did not think this behaviour strange until I noticed that the others were different. By the time I got suspicious my customer had already formatted and rebuilt their server so I can't do any futher investigating.
The difference between the regular infections and this one all have to do with the passwd and shadow files;
[root@server1 bin]# cat /etc/shadow
root:$1$gm9ykhCF$5SclI2Qg5qAojg64vLhbG.:15253:0:99999:7:::
root:$1$gm9ykhCF$5SclI2Qg5qAojg64vLhbG.:15253:0:99999:7:::
daemon:*:14672:0:99999:7:::
adm:*:14672:0:99999:7:::
lp:*:14672:0:99999:7:::
sync:*:14672:0:99999:7:::
shutdown:*:14672:0:99999:7:::
halt:*:14672:0:99999:7:::
mail:*:14672:0:99999:7:::
news:*:14672:0:99999:7:::
uucp:*:14672:0:99999:7:::
operator:*:14672:0:99999:7:::
games:*:14672:0:99999:7:::
gopher:*:14672:0:99999:7:::
ftp:*:14672:0:99999:7:::
nobody:*:14672:0:99999:7:::
nscd:!!:14672:0:99999:7:::
vcsa:!!:14672:0:99999:7:::
pcap:!!:14672:0:99999:7:::
rpc:!!:14672:0:99999:7:::
mailnull:!!:14672:0:99999:7:::
smmsp:!!:14672:0:99999:7:::
rpcuser:!!:14672:0:99999:7:::
nfsnobody:!!:14672:0:99999:7:::
sshd:!!:14672:0:99999:7:::
dbus:!!:14672:0:99999:7:::
haldaemon:!!:14672:0:99999:7:::
avahi-autoipd:!!:14672:0:99999:7:::
avahi:!!:14672:0:99999:7:::
ntp:!!:14672:0:99999:7:::
xfs:!!:14672:0:99999:7:::
gdm:!!:14672:0:99999:7:::
sabayon:!!:14672:0:99999:7:::
hacluster:!!:14672::::::
admin:$1$5Mq6lUXu$h6E9E0Am3s.UwK82spdNn/:15253:0:99999:7:::
user:$1$UNIM0eKr$ysqyzTeEZ1MoK7UavDM1O.:15291:0:99999:7:::
egg:$1$kTJWkCDt$Ls8.Xyik.Ao4wQMdLlBXn.:15264:0:99999:7:::
[root@server1 bin]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/bin/bash
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
nscd:x:28:28:NSCD Daemon:/:/sbin/nologin
vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
pcap:x:77:77::/var/arpwatch:/sbin/nologin
rpc:x:32:32:Portmapper RPC user:/:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
nfsnobody:x:4294967294:4294967294:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
haldaemon:x:68:68:HAL daemon:/:/sbin/nologin
avahi-autoipd:x:100:101:avahi-autoipd:/var/lib/avahi-autoipd:/sbin/nologin
avahi:x:70:70:Avahi daemon:/:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
xfs:x:43:43:X Font Server:/etc/X11/fs:/sbin/nologin
gdm:x:42:42::/var/gdm:/sbin/nologin
sabayon:x:86:86:Sabayon user:/home/sabayon:/sbin/nologin
hacluster:x:90:90:heartbeat processes:/var/lib/heartbeat/cores/hacluster:/bin/bash
admin:x:501:500:Admin:/opt/admin:/bin/bash
user:x:0:504::/home/user:/bin/bash
egg:x:502:502::/home/egg:/bin/bash
1. A new user called "user" had been added with root privileges.
2. A new user called "egg" had been added. I presume this means an eggdrop had been installed.
3. The database user had been deleted. This is how I discovered that the server had been compromised in the first place. :-)
4. The application server user had been deleted.
5. The most curious thing (and the reason I opened this thread in the first place) is that the shadow file had 2 root users both with the same password hash and other properties, while passwd has only one root user.
I suspect that this server was being used as a C&C master instead of a zombie which is why the additional users had been added (in particular the egg user). I don't quite understand why the cracker would want to delete the database and application server users as this immediately caused things to malfunction and caused the customer to open the support case.
What I really want to know from the *NIX guru's here is what effect having 2 root users in the shadow file has on the servers functionality and user authentication?