Hyper-threading presents an interesting dilemma to any software running on Nehalem-based processors.  For some multithreaded workloads, an operating system scheduler can spread threads across multiple NUMA nodes or co-locate them to a single node.  Consider the following figure, which depicts a single 8-way virtual machine being scheduled to all of the eight physical cores on a server.

This figure depicts the eight vCPUs of a single virtual machine being spared across two NUMA nodes' eight cores.

In this case the threads (vCPUs for vSphere) are each given their own physical core.  The benefit is that the vCPUs get unfettered access to their physical cores and the resulting additional computational power.  The drawback is that common memory is remote for half the vCPUs and will have to go through the other NUMA node.  This means memory-intensive workloads might run slower.

This figure depicts the eight vCPUs of a single virtual machine being consolidated to one NUMA node's four cores.

This second configuration places the same virtual machine’s eight vCPUs on a single NUMA node.  This means physical cores are shared but all memory access is local.  The vCPUs are contending for fewer CPU cycles, although they are benefiting from Hyper-threading.  This will result in less computational power than dedicated physical cores.  On the other hand, assuming the virtual machine was sized to fit in a single node, 100% of memory access will go to fast, local memory.  This could produce better performance for memory intensive workloads.

vSphere will prefer to spread virtual CPUs across NUMA nodes (option one above) to gain the benefit of more physical cores.  But if you are running an application where memory throughput is more important than processor speed, you should consider testing a change vSphere’s default behavior. You can do this by setting the ESX 4.1  advanced parameter NUMA.preferHT to 1.  This will configure the scheduler to prefer consolidating threads on logical processors on a single NUMA instead of using more physical cores across multiple nodes.

It would be nice if VMware provided definitive guidance on when virtual machines should be configured to prefer more physical cores (the default setting) or local memory access (NUMA.preferHT=1). But this guidance would be dependent on application, CPU, virtual machine size, consolidation ratios and utilization. The complexity of this guidance likely means that we will not see an authoritative word on this any time soon. But that does not stop you from experimenting on your own and sharing results. I would love to see any results of experiments posted here.

Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.