All kidding aside, yesterday I wrote an article about a sneaky trick to leverage improved hardware assist performance on ESX 3.5 virtual machines that defaulted to binary translation. I learned very late in the day that the recommended guidance only works for AMD processors. The text below is from the original post but has since been updated to reflect my newly discovered information.

Begin Updated Article

Do you have any running instances of ESX 3.5 or older?  Are those instances running on AMD processors that are no more than a couple of years old?  If so, I have a tip for you: force hardware assist in those virtual machines.  In most situations application performance will improve by 10% or more.  Details follow.

ESX’s monitor presents virtual hardware to virtual machines’ guest operating systems.  VMware’s multi-mode monitor uses three technologies to do this: hardware assist, para-virtualization, and binary translation.  Hardware assist has gotten much faster over the years, as this figure demonstrates.

The latency of the VMEXIT instruction is shown on Intel VT systems. The longer this instruction takes to execute, the worse the virtual machine performs.

Johan De Gelas included his take on monitor mode performance when he reported “Virtualization Round Trip Latency” in a recent article on the new Xeon 5600.  His results reiterate the trend I have been sharing with my audiences for over a year now.

Because hardware assist was once so slow, older versions of ESX would utilize our faster-performing binary translation in many situations.  But virtualization assist in today’s processors–and here I am talking about Intel and AMD processors manufactured in the past two years–is generally faster than binary translation. This means your virtual machines running on ESX 3.5 on shiny new processors may not be reaching their full potential performance.

The fix is simple: force hardware assist for your ESX 3.0 and ESX 3.5 virtual machines running on newer AMD processors.  You can do this with the following lines in your virtual machines’ VMX files:

monitor.virtual_mmu = hardware

A reboot is then required. But this setting only works for AMD processors, where RVI (AMD’s hardware memory management unit) is available. With this setting both AMD-V and RVI are forced on. This setting is ignored on Intel processors, whose hardware MMU is not leveraged on ESX 3.5.

These changes are not needed with vSphere because its default monitor modes favor hardware assist more than ESX 3.5 and earlier. You can see vSphere’s default monitor modes in our wonderful monitor modes paper.

As one example of the magnitude of performance increase this small change can produce, look to the SQL Server performance paper we released last year.  Here is one graph I lifted from that document.

This figure shows AMD-V improving performance by 18% over binary translation on virtual machines running SQL Server 2008.  More gain is possible when the hardware memory management unit is utilized.

Because of ESX 3.5′s continued wide deployment, it may very well be running millions of virtual machines.  Many those virtual machines are running on newer AMD processors that can benefit from this change.  Go forth and reconfigure those virtual machines to claim the performance to which you are entitled!

Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

Newer processors are much more important to virtualized environments than the non-virtualized counterpart. Generational improvements have not just increased the raw compute power, they have also reduced virtualization overheads. This blog entry will describe three key changes that have particularly impacted virtual performance.

Hardware Assist Is Faster

In 2008, with the launch of the Opteron 1300, 2300 and 8300 parts, AMD became the first CPU vendor to produce a hardware memory management unit equipped to support virtualization. They called this technology Rapid Virtualization Indexing (RVI). This year Intel did the same with Extended Page Tables (EPT) on its Xeon 5500 line. Both vendors have been providing the ability to virtualize privileged instructions since 2006, with continually improving results. Consider the following graph showing the latency of one key instruction from Intel:

This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT’s wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.

In a recent white paper detailing SQL Server on vSphere, the following graph showed the gains derived by using AMD-V in the Opteron 8324 (Shanghai).

Binary translation, AMD-V, and AMV-V plus RVI are measured using SQL Server.

This graph shows the practical value of the great gains that CPU manufacturers have made with virtualization assist.  Hardware assist can now be regularly relied upon for great performance.

Pipelines Are Shorter

The longest pipelines in the x86 world were in Intel’s Netburst processors. These processor’s pipelines had twice as many stages at their counterparts at AMD and twice as many as the generation of Intel CPUs that followed. The increased pipeline length would have enabled support for 8 GHz silicon, had it arrived. Instead, silicon switching speeds hit a wall at 4 GHz and Intel (and its customers) were forced to suffer the drawbacks of large pipelines.

Large pipelines are not necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drove thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and efficiency fell.

Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel’s Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But you should use newer parts for applications that hold your highest performance expectations.

Caches Are Larger

A cache is highly effective when it fully contains the software’s working set. The addition from the hypervisor of even a small about of code will change the working set and reduce cache hit rate. I’ve attempted to illustrate this concept with the following simplified view of the relationship between cache hit rates, application working set, and cache sizes:

Performance drops with small cache systems for even small increases to working set size.

This graph is based on a model that greatly simplifies working sets and the hypervisor’s impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the decrease cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization code. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.

In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX’s small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.