I started thinking about this topic at Sydney’s vForum a month ago.  VMware’s Dan Anderson suggested that designs that implemented maximum cluster sizes (32 hosts per cluster) were the result of misguided reasoning.  Dan insisted that clusters need never be larger than eight hosts per cluster.  And on this subject we bantered for a few minutes.  Dan convinced me that there are few compelling reasons to implement large clusters.  And we could think of many reasons to avoid them.  I do not think it easy to assign one number as the “right” cluster size.  But there are many principles that suggest small to medium sized clusters being choices.

First, the argument for the largest clusters: DRS efficiency.  This was my primary claim in favor of 32-host clusters.  My reasoning is simple: with more hosts in the cluster there are more CPU and memory resource holes into which DRS can place running virtual machines to optimize the cluster’s performance.  The more hosts, the more options to the scheduler.

But on retrospect I think this is a weak argument.  Its not backed by data and in practice I cannot imagine a 16 host cluster being much more efficient than an eight host cluster.  Once vCenter is managing hundreds or more virtual machines per cluster, it has an astronomical number of combinations for VM placement.  So, doubling the host (and the virtual machine count) should have little impact to cluster efficiency.

More importantly, with respect to the efficiency argument, maximum CPU and memory utilization will be bound either by the failover capacity or the target utilization, which is usually about 80%.  With 20% reserved for resource spikes, the failover capacity is equal to the reserved resources at a 4+1 HA cluster.  Any any cluster larger than this, the failover capacity is less than 20%.  This means that only target utilization bounds resource efficiency.

The efficiency calculation is a little more tricky if you want to size your cluster for target resource utilization after a host failure.  In this case each additional host provides some incremental value to the cluster’s utilization.  To size a 4+1 cluster to 80% utilization after host failure, you will want to restrict CPU usage in the five hosts to 64%.  Going to a 5+1 cluster results in a pre-failure CPU utilization target of 66%.  The increases slowly approach 80% as the clusters get larger and larger.  But, you can see that the incremental resource utilization improvement is never more than 2%.  So, growing a cluster slightly provides very little value in terms of resource utilization.

Now why might you want to keep a cluster small?  I can think of a few reasons.

It is generally wise to avoid mixing different classes of servers in a single pool.  DRS does not make scheduling decisions based on the performance characteristics of the server so a new, powerful server in a cluster is just as likely to receive a mission-critical virtual machine as older, slower host.  This would be unfortunate if a cluster contained servers with radically different–although EVC compatible–CPUs like the Intel Xeon 5400 and Xeon 5500 series.  In the former case ESX would be using its software memory management unit which could perform as much as 40% worse than the hardware MMU in the Xeon 5500.

(I will momentarily digress to answer a question I often get in my performance talks: what is the impact of Enhanced vMotion Compatibility (EVC) on virtual machine performance?  Briefly: very little to none.  The instructions that are disabled on newer processors only benefit applications that were compiled to use those new instructions.  Those applications are rare in the enterprise space.)

Given my recommendation that servers in a cluster should be of a similar class of performance, you will soon find that your purchasing patterns will influence your cluster size.  If you are one of the few people lucky enough to work at a company that is buying servers by the truckload, you can size your clusters however you want.  But the vast majority of VMware’s customers make smaller purchases of anywhere from four to 16 servers at a time.  These will make nice, homogenous clusters of moderate size.

One more argument Dave offered for keeping clusters small is to use clusters for logical separation of applications of different class.  By putting your mission-critical applications in a cluster of their own your “server huggers” will sleep better at night.  They will be able to keep one eye on the iron that can make or break their job.  In my opinion, using physical separation in a virtual world is resisting the complete cloud and hardware independent virtualization that we are all striving for.  But I cannot begrudge an administrator that wants to hold onto some semblance of physical hardware best practices while traveling the multi-year journey to the private cloud.

Another of Dan’s arguments against large customers is the cumbersome nature of their change control.  Clusters have to be managed to a consistent state and the complexity of this process is dependent on the number of items being managed.  A very large cluster will present unique challenges when managing change.

So, have I given a recommendation?  I am not sure.  If anything I feel that Dave, Dan and I believe that a minimum cluster size needs should be set to guarantee that the CPU utilization target, and not the HA failover capacity, is the defining the number of wasted resources.  This means a minimum cluster of something like four or five hosts.  While neither of us claims a specific problem that will occur with very large clusters, we cannot imagine the value of a 32-host cluster.  So, we think the right cluster size is somewhere shy of 10.

I am quite interested to hear your thoughts on this.  Perhaps the best guidance will grow out of the crucible of debate.

Hyper-threading presents an interesting dilemma to any software running on Nehalem-based processors.  For some multithreaded workloads, an operating system scheduler can spread threads across multiple NUMA nodes or co-locate them to a single node.  Consider the following figure, which depicts a single 8-way virtual machine being scheduled to all of the eight physical cores on a server.

This figure depicts the eight vCPUs of a single virtual machine being spared across two NUMA nodes' eight cores.

In this case the threads (vCPUs for vSphere) are each given their own physical core.  The benefit is that the vCPUs get unfettered access to their physical cores and the resulting additional computational power.  The drawback is that common memory is remote for half the vCPUs and will have to go through the other NUMA node.  This means memory-intensive workloads might run slower.

This figure depicts the eight vCPUs of a single virtual machine being consolidated to one NUMA node's four cores.

This second configuration places the same virtual machine’s eight vCPUs on a single NUMA node.  This means physical cores are shared but all memory access is local.  The vCPUs are contending for fewer CPU cycles, although they are benefiting from Hyper-threading.  This will result in less computational power than dedicated physical cores.  On the other hand, assuming the virtual machine was sized to fit in a single node, 100% of memory access will go to fast, local memory.  This could produce better performance for memory intensive workloads.

vSphere will prefer to spread virtual CPUs across NUMA nodes (option one above) to gain the benefit of more physical cores.  But if you are running an application where memory throughput is more important than processor speed, you should consider testing a change vSphere’s default behavior. You can do this by setting the ESX 4.1  advanced parameter NUMA.preferHT to 1.  This will configure the scheduler to prefer consolidating threads on logical processors on a single NUMA instead of using more physical cores across multiple nodes.

It would be nice if VMware provided definitive guidance on when virtual machines should be configured to prefer more physical cores (the default setting) or local memory access (NUMA.preferHT=1). But this guidance would be dependent on application, CPU, virtual machine size, consolidation ratios and utilization. The complexity of this guidance likely means that we will not see an authoritative word on this any time soon. But that does not stop you from experimenting on your own and sharing results. I would love to see any results of experiments posted here.

It was during discussions with advanced VMware customers in Melbourne that a solution to this problem occurred to me. I have reasoned it through and I think it holds water. I have socialized it with more customers and my colleagues and we think it stands. So I want to introduce a system for implementing virtual machines with a better assurance of a performance SLA.

The key to this process is that minimum performance can be measured using limits and that performance can be assured using reservations. You can develop and document virtual machines with performance SLAs using the following procedure:

• First, as always, define a small number of strictly-sized virtual machines to be used by all applications in your environment. Often these look something like small VMs of 1 vCPU and 4 GB RAM, medium VMs of 2 vCPUs and 8 GB of RAM, and large VMs of 4 vCPUs and 16 GB of RAM. Tune these numbers for your environment, as needed.
• For any application, benchmark its maximum performance against each of these virtual machine configurations on an unloaded system. Chose an ISV-supplied benchmark or a well-known third party tool. This sets your high water mark for throughput for each application in its virtual machine.
• For each configuration, set a CPU limit at 50% of the available CPU and a memory limit of 50% of the available memory. Retest the application against this smaller, limited configuration.
• During the applications’ deployment, change the limits to reservations. That is, remove limits and set reservations equal to the limits’ previous values, in this case 50%.
• Your application now has a maximum performance defined in bullet two, and a “guaranteed” performance measured in bullet three.  This is your application’s performance SLA.

The concept is simple: limits can be used to measure the performance of an application in the presence of that degree of contention. Reservations ensure that those resource amounts are always present. Here are some notes on this process:

• This is not a true guarantee since network and storage throughput may drop. No tool can eliminate this risk entirely but SIOC and NetIOC can reduce the risk of a network- or storage-induced performance failure.
• The memory test is going to be highly dependent on the working set created by your load generation tool. Your mileage will vary depending on your application owners’ use of the virtual machine.
• vCenter will guarantee that the reservations are always available through a process called admission control, which checks the cluster to ensure that enough CPU or memory is available to run the virtual machine immediately and in the event of a server failure.

As I said above, this is not a true guarantee of application performance. But it is as close as we can get until AppSpeed or a replacement evolves into universal application latency measurement that is fed into vCenter. And this is another in a growing list of reasons why CPU and memory reservations should be part of all VMware deployments.

Calling attention to a subset of the performance features in this launch, I will augment the published documentation with my own comments.

Wide VM NUMA Support

A “wide VM” is defined by VMware as a virtual machine whose memory is too large for a single NUMA node.  In this case, some of the memory must be placed on a remote node, which has a relatively higher memory latency.  ESX 4.0 would place as much memory as possible on a single node, then arbitrarily spill the rest over to other nodes.  ESX 4.1 now recognizes memory locality of reference and places frequently accessed memory on the local node, potentially eliminating remote memory access penalties.  Expect big gains with wide virtual machines running Java or very active databases.

Memory Compression

When I wrote on Steve Herrod’s preview of memory compression at PEX, I am sure you knew this feature’s release was imminent.  VMware’s documentation is sufficient on the gains provided by this feature, so I will not repeat those gains here.  The key thing to remember about memory compression is that it greatly reduces the need for swap.  For years VMware administrators have feared the spectre of memory swapping and have left memory woefully underutilized, even in consolidated environments.  With memory compression in place, you should more confidently push active memory closer to 100%.

Storage IO Control (SIOC)

Before the new VMware documentation, you had my article on SIOC and a delightful video previewing the feature to whet your appetites.  Now that the feature is out, I want to repeat the moral of the story: SIOC will save your high priority applications’ in the event of storage contention.  But if you storage performance stinks before SIOC, it will continue to stink with SIOC.  SIOC just buys you time for your mission critical applications so you can correct that storage problem.

Faster vMotion

(I’ll take a break here and point out the change of spelling from VMotion to vMotion.  This innocuous change will surely be missed by the large numbers of people that misspell VMWare [sic].  In truth, the case of vMotion is not particularly critical, but those of you grammatical pedants like myself take note.)

Many customers, already happy with vMotion, will scratch their heads as to what is left to be improved in this feature.  But a large number of you have tried evacuating 100 virtual machines from a host.  At two virtual machines at a time, this evacuation would have taken tens of minutes.  VMware was not limiting the vMotion concurrency for no good reason; they wanted to guarantee 100% correctness.  Careful evaluation, experimentation, and critical code improvements allowed the vMotion engineering team to greatly improve the efficiency of a migration in vSphere 4.1.  The result is that virtual machines more efficiency use the vMotion network which means VMware can qualify and support more virtual machines being concurrently migrated.

Part of this efficiency change included a decrease in the virtual machine switchover time, during which the application is unresponsive.  In every production environment I have seen, this switchover time was quite small, resulting in no application downtime.  But as processor performance and memory access time improved, and with vMotion efficiency remaining flat, eventually pages would be touched faster than vMotion could migrate them. This would result in vMotion failures.

The new vMotion efficiency improvements have dropped application switchover times to minuscule levels, guaranteeing zero application downtime for many years to come.

Network IO Control

Missing from the recently published performance document is an overview on Network IO Control (NetIOC).  In truth, I may be responsible for its lack of inclusion.  Apologies.  But luckily performance engineering released a best practices document on this wonderful new feature.

NetIOC is the network version of SIOC and may be even more important than SiOC in 10 Gb network environments and infrastructure using converged network adapters.  Let us be honest: the best practice of giving dedicated network hardware to each vSphere network traffic stream is so 2007.  It’s time to consolidate network and put everything on fewer 10 Gb adapters.  But this is going to create occasional network contention that would benefit from the same resource prioritization that CPU and memory shares have provided for years.

NetIOC will help prioritize your network streams in such an environment. Converge your networks and investigate NetIOC.

While memory over-subscription is a critical feature for production environments, balancing the demands of heterogenous applications of varying demands in a resource starved environment is difficult.  Without guidance from administrators on the relative importance of the virtual machines running these applications, a hypervisor will be forced to make arbitrary decisions in assigning limited resources.  Effective use of over-commitment requires a sound resource control system.  The only product on the market that does this well is VMware vSphere.

Both Greg and my articles only talked of memory over-commitment, but the rules apply for CPU over-commitment, too.  Microsoft will realize how important resource controls are somewhere between year two and five of their product’s life.  I can only imagine where vSphere will be by then.

What do I mean by “drive over-commitment”? I mean that, when properly used, memory reservations allow a VI admin to optimally pack virtual machines across a cluster’s memory. With properly set reservations, an admin can continue to power on a cluster’s VMs until vCenter’s admission control refuses to allow more. At that point you can know that you the optimal number of virtual machines is on your hosts.

The first step in using vCenter’s admission control to drive consolidation is to properly reserve memory. You do this by summing the minimum memory needs for everything in the virtual machine and setting reservations to that number. Some thoughts on reservations follow:

This table will get you started in setting a virtual machine’s memory. For example, consider a virtual machine running Windows Server 2003 (500MB minimum), with SQL Server 2005 (500 MB), running a database that needs at least a 500 MB cache. This VM’s reservation should start at 1.5 GB. But for future growth, small spikes of peak usage, and for additional administrator tools, it is entirely possible that additional 2 GB will be needed. So, the VM should be sized to at least 3.5 GB with a 1.5 GB reservation.

Given the wide variation in memory usage from OS to OS, application to application, and instance to instance, I would by lying if I told you a formula worked right without later revision. The truth is you can only estimate these values before deployment. You will have to monitor your memory hungry applications and fine tune reservations over time.

You do this by checking vCenter’s stats for each VM’s active memory. The active memory should be less than your selected reservations except for rare occasions. So, if active memory is much higher than your reservations, increase reservations. If your VM is consistently using less memory than you have reserved, reduce the reservations.

When care has been taken to properly size and reserve memory, the administrator can power on virtual machines confidence that memory is available to support the application. When vCenter is unable to reserve enough memory to support your reservations, it will refuse to power on VMs. This means that memory management is no longer about trend analysis and capacity management. Instead you will investigate application needs with their owners and trust vCenter to give you a simple “yes” or “no”.

To get the most out of your servers’ memory you have to over-commit it.  Go too far and the host will swap.  This fact is immutable in enterprise memory management.  Because rotating disks have seek times that are six orders of magnitude larger than memory, when your host swaps your applications’ performance suffers catastrophically.  Solid state drives (SSD) can mitigate the performance cost by reducing swap latency by a couple orders of magnitude.  But SSDs still have delays tens of thousands of times worse than memory.

VMware engineers have been working on a technology that we unofficially call on-demand memory compression (ODMC) or compression cache, depending on who you talk to.  The idea of ODMC is to avoid swapping by compressing a set of target pages to a special region.  We have measured the additional latency–the compression time–to be over a hundred times better than rotating disk latencies.  And this will decrease at CPU performance increases.

SSD devices will continue to play an important part in solving the performance problems of extreme memory over-commit because of the great volume of data they can serve at a speed much faster than rotating disks.  But it will be a secondary solution to the faster ODMC, which uses a small dedicated area of system memory.  VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
2. Use ODMC to a predefined cache to decrease memory utilization.*
3. Swap to persistent memory (SSD) installed locally in the server.**
4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.

(**) Part of our vision and not yet demonstrated.

The end goal of this prioritized use of different technologies is the reduction of the performance penalty of swap.  If VMware can reduce the penalty of swap, the virtual machines in extremely over-committed environments will not slow much when they need more memory than the host has available.  When this performance cost of swapping is reduced, you can safely drive your consolidation ratios even higher.

I cannot wait for the general availability of ODMC.  It is another technological accomplishment in a long line of innovation out of our phenomenal engineering organization.  At times I wonder when the last time another operating system software company introduced something really awesome to the industry.  But of course, I am biased.

Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.

I believe we know the text that is misrepresented.  It comes from our performance best practices document and could be misunderstood by someone to whom the terms “working set” and “active memory” are unfamiliar.  Here is the sentence, quoted from the oldest available version of our best practices document:

Swapping is used to forcibly reclaim memory from a virtual machine when both page sharing and ballooning fail to reclaim sufficient memory from an overcommitted system. If the working set (active memory) of the virtual machine resides in physical memory, using the swapping mechanism and having inactive pages swapped out does not affect performance. However, if the working set is so large that active pages are continuously being swapped in and out (that is, the swap I/O rate is high), then performance may degrade significantly.

This excerpt describes the condition under which a host will swap.  That condition is best summarized as “the sum of the working sets of all virtual machines exceeds the amount of memory on the host”.  This definition presupposes that the reader understands the definition of a working set, which I think some readers may not.  For this discussion I will simplify the definition of working set as “recently active memory” and refer readers to their helpful search engine for a more complete description.

When a system’s working set exceeds available memory, the system will swap.  This is not unique to virtual, consolidated workloads.  As long as operating systems have implemented virtual memory, there existed a possibility that a working set could exceed available physical memory.  The only thing that has changed in a virtual environment is that the working set is calculated by summing the working sets from multiple virtual machines as opposed to a single application or operating system instance.

But the key here–and the reason why memory over-commitment remains so powerful–is that allocated memory (the virtual machine’s size) exceeds active memory (the working set) nearly 100% of the time.  Memory management in consolidated environments is about pushing a host’s active memory to as close to 100% as possible.  This is something that was not possible in physical environments, and cannot be done in virtual environments without over-commitment.

To sum this up, not only does VMware recommend some over-commitment, but we know that it is impossible to fully use your available memory without the flexibility provided by over-commitment.

But a recent article by my engineering colleague Chethan Kumar shows an avenue that allows VI admins to aggressively over-commit memory and avoid the catastrophic performance penalty of swapping: use solid state disks to host ESX swap files.

The fundamental problem with host swapping comes from the high latency of traditional disks compared to memory.  Data can be retrieved from memory in nanoseconds but takes milliseconds to fetch from a hard drive.  That means a single 4K memory page takes 100,000 times longer to retrieve if the operating system swapped it out.

The value that solid state disks offer to this problem is exceptional latency, as compared to traditional drives.  The SSD that Chethan used showed microsecond latencies, about 1,000 times lower than physical disks.  This means that  time spent waiting for swap activity* has been decreased to 0.1% of the time spent swapping to physical disks.

The importance of fast swap files is that it enables administrators to more aggressively over-commit memory.  Today our admins rightfully fear the VMs’ aggregate active memory exceeding the available physical memory, which results in swapping.  Today SSD technology in shared storage such as EMC’s new CLARiiONs allows our admins to cleverly place swap files and drive up memory utilization to previously unheard of levels.  This may enable standard memory overcommitment of 200% or more, with extreme over-commit being much higher than this.

In future versions of ESX we want to automate the usage of SSDs to maximize the use of available memory.  But that’s a roadmap discussion that I will leave for another day.

(*) This swap wait time has conveniently been added to ESX 4′s version of esxtop under the counter %SWPWT.  See Interpreting esxtop Statistics for more information.