Testing results from internal and customers showed SAP workloads: 85-95, Oracle OLTP: 80-92% bare metal. LAMP stack showed better than bare metal performance. Whitepapers will be published in how this was achieved. Java achieved up to 94% bare metal.

Frankly, I was surprised to hear this.  KVM is a hosted virtualization platform, equivalent to the free VMware Server, which runs on top of a host operating system.  VMware server is fine for a virtual machine or two, but you would not want it hosting your critical business applications.  The above KVM claim suggests that KVM possesses hypervisor-like performance.  So we ran a test with a few virtual machines to see what we could learn.  These tests confirmed my suspicions: KVM is a very long way from enterprise-class virtualization performance.

The thing to remember about virtualization benchmarking is that any vendor can provide virtualization software (hosted or hypervisor) that can virtualize a single application at better than 80% of native performance.  VMware has been doing this for a decade.  But it is extraordinarily difficult to build a hypervisor that can scale with many virtual machines.  Maybe this is one reason why you have never seen Microsoft or Citrix post results from a consolidated workload.  But I digress.

We decided that the easiest way to test this environment with a light/moderate enterprise workload is to use two or three VMs running SQL Server, as tested by DVD Store 2 (DS2).  We tried four configurations of these VMs:

• Case A: Two 4-way virtual machines.
• Case B: Two 3-way VMs and one 2-way.
• Case C: Three 3-way VMs.
• Case D: Three 4-way VMs.

Each virtual machine ran on an HP DL380 G5 and was given 4 GB.

Finding the right number of threads per virtual machine took some time.  Threads on the DS2 client determine the volume of transactions that are generated against the SQL Server.  We wanted to get the highest throughput for a reasonable latency, which we set at 33 ms.  Here are the best numbers I could produce for vSphere and KVM.

The very best performance that KVM could muster was only removed% of vSphere’s performance on the same configuration.  Notice that at 50% CPU over-commitment (1.5 vCPUs for each CPU), KVM’s performance removed.  It’s throughput fell to removed% of vSphere and its response time removed.  Increasing threads in this configuration actually made throughput and latency worse.

I had suspected that KVM would show hosted platform performance, as it relies on a host operating system.  It appears my suspicions were correct.  It will be tough for Red Hat to sell this product as part of an enterprise product.  To do so they will likely publish results based on single virtual machines and in environments where the CPUs are under-committed.

Lastly, this is the only workload that we have attempted.  I would expect KVM to do much worse when more virtual machines are part of the test or if network or storage throughput becomes significant.  But we have no plans to spend time on KVM benchmarking.  As I mentioned in my performance debate at Catalyst 2009, I think that each vendor should do its own benchmarking to best represent its products.  I challenge Red Hat to post a KVM number using TPC, SPECweb, VMmark, vConsolidate, or any enterprise-class workload.  Customers should expect nothing less of their virtualization vendor.

10/2/09 Update

I decided to remove the KVM results to allow Red Hat or a KVM enthusiast to show their own best results on a consolidated workload.  I recommend VMmark or vConsolidate.