A new Trojan horse is so good at hiding itself that some security researchers claim a new chapter has begun in their battle against malicious-code authors.

The new pest, dubbed "Rustock" by Symantec and "Mailbot.AZ" by F-Secure, uses "rootkit" techniques crafted to avoid the detection technology used by security software, Symantec and F-Secure said in recent analyses.

"It can be considered the first born of the next generation of rootkits," Elia Florio, a security response engineer at Symantec, wrote in a blog late last month. "Rustock.A consists of a mix of old techniques and new ideas that when combined make a malware that is stealthy enough to remain undetected by many rootkit detectors commonly used."

Rootkits are considered an emerging threat. They are used to make system changes to hide software, which may be malicious. In the case of Rustock or Mailbot.AZ, rootkit technology was used to hide a Trojan horse that opens a backdoor on an infected system, putting it at the beck and call of an attacker, according to Symantec.

In their continuing race with security software makers, the creators of this latest rootkit appear to have looked closely at the inner workings of detection tools before crafting their malicious code, said Craig Schmugar, virus research manager at McAfee, which calls the pest "PWS-JM."

"Security companies are trying to stay one step ahead of the bad guys, but the bad guys already have the technology that is available from the security vendors," he said. "A number of techniques have been combined to really strengthen and harden this particular threat. They have done a pretty good job at closing all the doors."

The mixture of cloaking methods makes Rustock "totally invisible on a compromised computer when installed," including on a PC running an early release of Windows Vista, Symantec's Florio wrote. "We consider it to be an advanced example of stealth by design malicious code."

To avoid detection, Rustock runs no system processes, but runs its code inside a driver and kernel threads, Florio wrote. It also uses alternate data streams instead of hidden files and avoids using application programming interfaces (APIs). Today's detection tools look for system processes, hidden files and hooks into APIs, according to Florio's post.