CEO Paul Maritz apologized to customers and said VMware was determined to get to the bottom of the problem that caused virtual machines to refuse to start.

By Charles Babcock
InformationWeek
August 13, 2008 04:00 PM


VMware has issued a patch for the bug that was contained in Update 2 for the ESX 3.5 hypervisor, a bug that caused ESX and ESXi 3.5 virtual machines to refuse to start up Tuesday morning. The company also is working to fix the Update 2 software and reissue it without the flaw.
The ESX 3.5 hypervisor's Update 2 was first released 17 days ago. The new, freely downloadable ESXi 3.5 hypervisor was released 16 days ago. The company said it expect the bug-free software to become available at later today.

A license limitation built into the Update 2 release caused ESX and ESXi 3.5 virtual machines to stop running Tuesday morning. VMware issued a patch for existing Update 2 users at about 9 p.m. Tuesday. It didn't have an estimate of how many customers were using Update 2 or what percentage of the customer base had adopted it. Because the update was issued only recently, it is possible only a fraction of the customer base has tested the update in isolation and then implemented it in production.
Nevertheless, new president and CEO Paul Maritz wrote a letter in which he apologized to customers and said VMware was determined to get to the bottom of the problem. "I want to apologize for the disruption and difficulty this issue may have caused our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly."

VMware hoped to have a new version of Update 2 available by noon on Wednesday, it said Tuesday as news of the bug spread. VMware spokesmen at noon revised that timing and said the new version will be ready by 6 p.m. on Wednesday.

"We are doing everything in our power to make sure this doesn't happen again," Maritz' letter, posted on the VMware Web site, said. "VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products. ... We have kicked off a comprehensive, in-depth review ... and will quickly make the needed changes."

The problem was created when ESX 3.5 Update 2 was sent to customers with a piece of code in it that caused the license governing use of the code to expire at 12 a.m. on Tuesday, Aug. 12. That wouldn't shut down a continuously running virtual machine, but it if had been decommissioned and stored Monday night, it wouldn't start again Tuesday morning. Likewise, use of the VMotion capability that shuts a virtual machine down on one physical server while starting a carbon copy on another server would also cause the virtual machine to fail.

A virtual machine that was left in suspension mode Monday night could not be brought out suspension on Tuesday morning, VMware officials said.

Maritz explained that the inadvertent code in Update 2 "was designed to ensure that customers are running on the supported, general available version of Update 2," not earlier, unsupported versions of Update 2. But the explanation was still likely to leave users of the free version of the ESX hypervisor, ESXi, disgruntled if they had been sold on its ease of use features.