I started my research on Twitter and asked friends to help me find whitepapers on the subject of VMware environment network best practices.  One Tweep (Nate Raper) asked me to share everything I found.  This article, vPivot’s first aggregation, contains the VMware environment network papers that I have received and reviews.

Below are the network papers I know of and my comments on them.  I will continue to add to this list as more papers come to my attention.

If you know of any other networking documents focused on VMware deployments, please mention them in the comments. I will continue reading them and linking them here.

Calling attention to a subset of the performance features in this launch, I will augment the published documentation with my own comments.

Wide VM NUMA Support

A “wide VM” is defined by VMware as a virtual machine whose memory is too large for a single NUMA node.  In this case, some of the memory must be placed on a remote node, which has a relatively higher memory latency.  ESX 4.0 would place as much memory as possible on a single node, then arbitrarily spill the rest over to other nodes.  ESX 4.1 now recognizes memory locality of reference and places frequently accessed memory on the local node, potentially eliminating remote memory access penalties.  Expect big gains with wide virtual machines running Java or very active databases.

Memory Compression

When I wrote on Steve Herrod’s preview of memory compression at PEX, I am sure you knew this feature’s release was imminent.  VMware’s documentation is sufficient on the gains provided by this feature, so I will not repeat those gains here.  The key thing to remember about memory compression is that it greatly reduces the need for swap.  For years VMware administrators have feared the spectre of memory swapping and have left memory woefully underutilized, even in consolidated environments.  With memory compression in place, you should more confidently push active memory closer to 100%.

Storage IO Control (SIOC)

Before the new VMware documentation, you had my article on SIOC and a delightful video previewing the feature to whet your appetites.  Now that the feature is out, I want to repeat the moral of the story: SIOC will save your high priority applications’ in the event of storage contention.  But if you storage performance stinks before SIOC, it will continue to stink with SIOC.  SIOC just buys you time for your mission critical applications so you can correct that storage problem.

Faster vMotion

(I’ll take a break here and point out the change of spelling from VMotion to vMotion.  This innocuous change will surely be missed by the large numbers of people that misspell VMWare [sic].  In truth, the case of vMotion is not particularly critical, but those of you grammatical pedants like myself take note.)

Many customers, already happy with vMotion, will scratch their heads as to what is left to be improved in this feature.  But a large number of you have tried evacuating 100 virtual machines from a host.  At two virtual machines at a time, this evacuation would have taken tens of minutes.  VMware was not limiting the vMotion concurrency for no good reason; they wanted to guarantee 100% correctness.  Careful evaluation, experimentation, and critical code improvements allowed the vMotion engineering team to greatly improve the efficiency of a migration in vSphere 4.1.  The result is that virtual machines more efficiency use the vMotion network which means VMware can qualify and support more virtual machines being concurrently migrated.

Part of this efficiency change included a decrease in the virtual machine switchover time, during which the application is unresponsive.  In every production environment I have seen, this switchover time was quite small, resulting in no application downtime.  But as processor performance and memory access time improved, and with vMotion efficiency remaining flat, eventually pages would be touched faster than vMotion could migrate them. This would result in vMotion failures.

The new vMotion efficiency improvements have dropped application switchover times to minuscule levels, guaranteeing zero application downtime for many years to come.

Network IO Control

Missing from the recently published performance document is an overview on Network IO Control (NetIOC).  In truth, I may be responsible for its lack of inclusion.  Apologies.  But luckily performance engineering released a best practices document on this wonderful new feature.

NetIOC is the network version of SIOC and may be even more important than SiOC in 10 Gb network environments and infrastructure using converged network adapters.  Let us be honest: the best practice of giving dedicated network hardware to each vSphere network traffic stream is so 2007.  It’s time to consolidate network and put everything on fewer 10 Gb adapters.  But this is going to create occasional network contention that would benefit from the same resource prioritization that CPU and memory shares have provided for years.

NetIOC will help prioritize your network streams in such an environment. Converge your networks and investigate NetIOC.

Chad parenthetically wrote on VMDirectPath in a recent blog:

Normally, as people who know me can attest to, I’m not a general fan of VMdirectpath. Most people think of it as a “performance thing” when the reality is that with VMXNET3 and pvSCSI adapters in vSphere 4 you can get to within low percentage points of the same performance as you can with “hypervisor bypass” (met 3 customers in the last 2 weeks who were adamant they needed VMDirectPath for “performance” – but didn’t know what their perf goals were, and were using VI3 as the baseline – argh), and I know that this is called out in Cisco’s training on UCS. For some reason people think for “high performance IO” you can’t do it without bypass, but that is not explicitly true.

I have been touring the world telling people that VMDirectPath is not a performance feature ever since it was released.  Unfortunately my pithy marketing statements are only partial truth.  In March of this year VMware released an update to their ongoing SPECweb work that leveraged VMDirectPath.  It is unfortunate that this article encourages customers to implement VMDirectPath.  I want to convince you to ignore it.

Before it was released, VMware was internally calling VMDirectPath “pass through”.  The idea was to provide direct access to the underlying hardware to the guest operating system and its applications.  The monumental negative impact of this is that the guest operating system becomes married to the physical hardware.  This means the virtual machine cannot be moved (vMotion), load balanced (DRS), protected through fault tolerance (FT), and many other awesome vSphere feature.  Enabling VMDirectPath for a virtual machine effectively moves your virtualization technology back to 2006.

The only reason why anyone is considering VMDirectPath for production deployments is the possibility of increased performance.  But the only workload for which VMware has ever claimed substantial gains from this feature is the SPECweb work I quoted above.  That workload sustained 30 Gb/s of network traffic.  I doubt any of VMware’s customers are using even a fraction of this network throughput on a single server in their production environments.

Even in this extremely intense network environment, the VMDirectPath contribution to application throughput is not quantified.  I discussed its impact with the engineer that performed the test and he estimated the performance improvement at less than 10%.

The reason why device passthrough is not useful in vSphere is because of the incredible efficiency gains VMware has made with vmxnet and its virtual storage stack.  As Chad mentioned above, customers carry with them bad memories of VMware’s IO capabilities in VI3.  That was a different era, my friends.  I hope that I can convince you to trust VMware’s core technologies and not be looking for features that circumvent them.

So, let me summarize:

• If you enable VMDirectPath you lose most of the features you love about vSphere.
• VMDirectPath has never been shown to have any impact on tier-1 applications even with high network needs (10 Gb/s).
• The largest possible performance gain VMDirectPath has been shown to add to an application is about 10%, and that was an extreme network usage of 30 Gb/s.

Tread carefully with this feature, everyone.  Its usage should be a last resort, not a first.

VMware’s dedicated and curious customer base somehow obtained information on an undocumented parameter that limits the maximum number of concurrent VMotions.  Information on modifying this parameter is sprinkled around the internet tubes but my favorite comes from Jason Boche.  Jason and others have identified how you can change the current limit of two VMotions per host.  I want to give you an explanation of why you should not do this and show you what you can expect from future releases of vSphere.

The concurrent VMotion limit was set after careful analysis of the capabilities of existing hardware, VMotion implementation details, and a large number of enterprise applications.  It has been set at two to provide a near 100% guarantee of no downtime during the migrations.  On some workloads or on older hardware, it is possible that three or more concurrent migrations could saturate a system resource and result in downtime.  If your hardware is brand new and your application is not wildly touching memory, it is possible that you can somewhat safely increase the concurrent VMotion limit.  But, as Kit Colbert told me, “I think allowing four simultaneous VMotions is probably OK in most scenarios, but if the VMs are really large and/or have very big working sets, then I’d dissuade customers from bumping up the limits.”

And if Kit’s gentle reminder is not enough to dissuade you from making this change to your production environments, I will point out that problems that arise as a result of changing the concurrent VMotion limit are not supported by VMware.  We simply cannot promise the unfaltering quality of VMotion if end users increase this limit.

Now, back to the feature preview in my ongoing performance road show (today: Bellevue, WA!).  Using an in-house development version of ESX, we are running eight concurrent VMotions on a single host with the even better quality than that of vSphere 4.  A phenomenally dedicated group of engineers has drastically improved the throughput of VMotions and decreased the already tiny virtual machine stun time.  This means that we can maintain our zero downtime commitment while upping the number of concurrent VMotions by a factor of four.  Furthermore, total migration time has also decreased, so our development host can evacuate a large number of virtual machines almost order of magnitude faster than vSphere 4.

(*) Any time I talk about this or other unreleased features the standard VMware disclaimer applies.  We are not committing this feature to any specific product nor committing any product to a specific date.  Not my rules, guy

What follows is my tale of performance optimizations so arcane and of extremely limited relevance as to make this guidance useful to one customer in 100,000.  The parameters I reference here should rarely–if ever–be touched by a VI admin.  I am documenting them for here for that one mischievous soul that is allowed to do as much damage as good in the pursuit of performance excellence. The rest of you should not even consider these changes for your production environment.

Seriously.  Don’t do it.  Don’t do it.

About two weeks ago I got a call that a large customer was in need of an immediate performance assistance.  Ten hours after that I was on a plane to Boston.  Nine hours later I was sitting in the customer’s lab looking at the problem.  Five hours later the problem was fixed.  Three hours more and I was reporting out to the customer’s management.  Two hours after that I was at Fenway with a beer in one hand and chowder in the other.  What follows is a very brief synopsis of the problem and its solutions.

The customer was using VMware on a very strange (non-HCL) server.  Their stringent requirements stated that one virtual machine, configured as a network bridge using RHEL 5.3′s bridge-utils, must drop zero 64-byte UDP packets at up to 80,000 frames per second.  The Ixia device the customer was using for network tests reported that the ESXi-based virtual machine was maxing out at at 20,000 FPS while meeting the 0% packet loss requirement.

I tried about a dozen configuration changes in my half-day of experimentation and learned a few interesting things.

Occasionally the e1000 Will Outperform vmxnet3

This was a surprise to me at first but our network team later explained it to me clearly.  The e1000 device will never make a monitor-to-kernel call and instead relies entirely on VMkernel polling to pass information from the guest to the host.  vmxnet does make VMM->VMk calls after queuing up packets to limit the overhead of packet processing.  This buffering behavior benefits network throughput in about 99% of cases but the additional VMM->VMk calls were not helpful with this rare workload.

The vmxnet Throughput/Latency Tradeoff Is Configurable

ESX and ESXi expose a large number of advanced network settings that can be used to configure vmxnet’s predilection for throughput or latency.  Increasing vmxnetThroughputWeight, for instance, tells vmxnet to buffer packets longer to reduce the number of VMM->VMk calls, thus improving efficiency.  CoalesceTxTimeout will similarly affect buffering by setting a maximum coalesce time for both vmxnet and e1000.

VMware has demonstrated on one workload a successful modification of the default for vmxnetThroughputWeight.  See our SPECweb work that peaked at 16 Gb/s throughput, for instance.  My customer could not benefit from this parameter as it only changes vmxnet behavior but we did see gains by increasing CoalesceTxTimeout.

Guest Drivers Require Optimization for Small Packets

The size of the ring queues used by device drivers for physical and virtual devices is configurable.  Those buffers default to 256 entries but can be increased to 4096 for both vmxnet and e1000 when using vSphere.  For very small packets that arrive incredibly rapidly a larger ring queue is needed to avoid unnecessary packet drops.  When I used ethtool to increase in ring queue size the packet drop rate decreased dramatically.

Summary

If you have a environment that consistently seeing large volumes of tiny UDP packets (less than 128 B) some of these changes may improve your environment. But this configuration is uncommon enough in the virtualized enterprise that you probably do not have such a workload. So, do not start playing with vSphere’s defaults unless you know you are running an application that is a clear outlier.