Apple makes it difficult to quote their End-user License Agreement (EULA) here.  I found the EULA using Apple’s App Store application on my Mac.  On the right sidebar of the Lion information screen is a link that says “App License Agreement”.  That brings you to the EULA, which is in a non-searchable, non-copyable textbox.  So, I’m retyping the relevant section here for discussion.  I have checked this twice and I think there are no typographical errors.

In this entry I want to share with you VMware’s top 10 list that Pete presented with Tim Hartman. I have provided a PDF version of their presentation on my blog in case you want more detail. Feel free to contact me, your local VMware SE, or a vSpecialist if you want more information. Now, on to the top 10!

Reason #10: OSX Lion Support. If you follow me on Twitter you know I think OSX Lion is Apple’s Vista.  Application load times stink, the built-in PDF viewer “Preview” will not print on my laptop, I am always warned that no preferred network is available followed by a dialog showing my preferred home network, and there are other annoyances.  But, hey, at least Apple is allowing VMware to virtualize this version of OSX.  And vSphere 5 supports it.

Reason #9: Auto-deploy. VMware has been working on stateless ESXi for some time and vSphere 5 finally got it right. You can easily build installable images and use vCenter’s Auto Deploy to fully configure a new instance during install time.  This means a small amount of configuration information in the image points the new ESX server to vCenter’s host profiles, which provides the full configuration for an ESX host.

Reason #8: vSphere Replication. Site Recovery Manager (SRM) 5 can now use software replication in the ESX server, which I informally call host-based replication (HBR).  Even though EMC sells replication, we love what VMware is doing with HBR.  Its simple, elegant, and wonderfully integrated.  Everyone should give it a try.  You will find EMC’s replication products as simple, as integrated, and even more robust.  As your SRM deployment grows consider EMC replication.  But for your first SRM deployment I highly recommend VMware’s HBR.

Reason #7: Cloud Security. Roman Tarnavski and I love VMware vShield.  I am convinced that every customer in the world should deploy vShield.  vShield 5 comes embedded with RSA’s data loss protection.  This means vShield can automatically detect and fence off virtual machines with sensitive information.

Reason #6: Monster VMs. Now you can make 32-way virtual machines with up to 1 TB of vRAM.  VMware’s goal here is to enable customers to go 100% virtualized with virtual machines that can contain your largest workloads.  But in my opinion the applications that could not fit in vSphere 4′s 8-way virtual machines represented less than 0.01% of the world’s x86 applications.  The result is these new virtual machine sizes only opened vSphere up for the final 0.01% of the world’s un-virtualized x86 applications.

Reason #5: Manage Anywhere. Finally, a vSphere Web Client! ’nuff said.

Reason #4: Failback Planned Migration. SRM 5 has been augmented to allow re-protection of a protected site after a migration or DR event.  This allows for failback, which customers have been asking for since SRM was first released.  On top of this, VMware has enabled a planned migration scenario with SRM 5.  This means SRM 5 customers can shutdown protected virtual machines, synchronize their content using the vendor’s storage replication adapter (SRA), and restart the virtual machines on the remote site.  With the click of a button an entire datacenter can be moved with little disruption.

Reason #3: vSphere Storage Appliance. Turn your unused x86 servers with direct-attached storage into vSphere storage nodes.  The VSA will protect data using a RAID 1+0 configuration.

Reason #2: Linked Clones. vCloud Director now supports linked clones.  This greatly improves virtual machine deployment times and drastically reduces the cost of storage in cloud deployments.

Reason #1: Smarter Storage. VMware finally released the storage DRS you knew was coming for years.  On top of this, the new vStorage APIs for Storage Awareness will subtly but powerfully change the face of storage management.

I asked about nine questions of the forum’s attendees and four of them generated rich discussion.  There were about 25 people in attendance, which were self-selected to have both opinions and total comfort sharing them.  Here are the questions, the results, and my commentary.

Question: What do you think is the state of performance of VMware vSphere 4.1?

Answer options:

1. I am not concerned about performance in any of my virtualized applications
2. I am concerned about performance in a very small number (<2%) of my applications
3. I do not think that vSphere can perform well enough for my big databases
4. I virtualize IT applications, but fear the virtual performance of my lines of business applications
5. I fear performance in general, and only virtualize after rigorous testing

The results:

My perspective on VMware performance is that it is sufficient to virtualize all but about 0.1% of application instances in today’s enterprise since vSphere 4, nevertheless vSphere 4.1.  This means a theoretical penetration of 99.9% for VMware’s hypervisor.  It is clear from these results that VMware’s customers do not yet believe VMware is so close to ubiquitous virtualization.  But often when I dig into performance objections the root of concern is dated results on older hardware and experiences on previous versions of VMware’s products.  My mantra to those of you that remain skeptical of what VMware can do with your big applications is “give it another try”.

Question: How do you feel VMware vSphere 4.1 performance compares against the competition?

Answer options:

1. vSphere 4.1 is generally much better than competing hypervisors
2. vSphere 4.1 is sometimes better and other times similarly performing to competing hypervisors
3. vSphere 4.1 performance is equivalent to other hypervisors, sometimes winning and sometimes losing
4. vSphere 4.1 is consistently slower than at least one other hypervisor
5. Who cares?  I am not making a purchasing decision based on performance anyway

The results:

A few times in my years at VMware I directly engaged in competitive debate on the relative merits of VMware’s products.  But my position on the need of direct competitive comparison has mellowed recently.  I much prefer to put my company’s products in their best light and let our customers decide if its merits outweigh those of our competitors.  But I am always fascinated by the public perception of our work, even when evaluated at a conference largely composed of enthusiasts.

The large portion of respondents that believe vSphere in a dominant position with respect to performance will extol the incredible pace of innovation coming from Palo Alto.  And the sizable portion of attendees that consider most hypervisors at performance parity with each other will surely site all products’ heavy reliance on hardware assist.  I believe that both of these perspectives are true.  When ESX is “dumbed down” to be feature equivalent to another product (by doing something like removing the balloon driver or turning off memory compression) it is possible for a less mature product to deliver comparable performance.  But it is because of the barrage of features included in each launch that VMware can generate more work per server than its competition.

Question: Which of the following represents the most important performance accomplishment in the past three years?

Answer options:

1. The ESX scheduler’s continual improvement
2. Improved hardware assist in the CPU
3. A better vSphere IO stack, including the paravirtualized SCSI device
4. Paravirtualization of the operating system (including Windows enlightenments)
5. Other

The results:

Intel and AMD’s dedication to improving the efficiency of virtual computing environments cannot be overstated.  But it is often lost in that discussion that VMware put tremendous engineering effort to making good use of these features, including rebuilding a scheduler to leverage Hyper-threading more aggressively than ever before.

From my perspective the most important improvement in VMware’s history was the storage stack optimizations that came with vSphere 4.  It took those improvements before one could seriously consider running intense, business-critical, IO-bound applications in a virtual environment.  But now you can.  Storage performance is not longer a problem.  Case closed on this, believe me.

Question: What is the coolest performance feature added to vSphere 4.1?

Answer options:

1. Memory compression
2. Storage IO control (SIOC)
3. Network IO control (NetIOC)
4. Faster vMotion, with more concurrent vMotions
5. Other

The results:

I was glad to see how popular the new vMotion is in vSphere 4.1.  I was surprised that many people took note of this improvement.  But very glad.

I am not surprised that Storage IO Control (SIOC) figures prominently with customers of new features.  Given that storage causes so many performance problems, VMware clearly has a lot of customers that want help correcting them.

My erstwhile colleague and VMworld co-presenter, Kaushik “K-ban” Banerjee, votes strongly for Network IO Control (NetIOC) as the more important addition to the vSphere 4.1 arsenal.  Kaushik argues that 10 Gb networking is exploding in IT and that application performance management on it is very difficult without bandwidth prioritization.  One attendee of the Copenhagen group discussion agreed with K-ban in principle but claimed that 10 Gb networks were not yet prevalent enough to demand the presence of NetIOC.  If this is so, then we can thank VMware for having the foresight to correct a problem before it is widespread.

I hope to soon receive the results from the VMworld in Copenhagen.  When I do I will certainly post a follow-up blog.  The sample size is relative small as a portion of VMware’s US and EMEA business but it could be fun to draw some generalizations from each region’s responses.

Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

On VI3, many of VMware’s customers disabled Hyper-Threading on their older, Netburst architecture Intel processors. Intel has vaguely described the new Hyper-Threading as more efficient than the previous generation and I believe this to be due to a shorter pipeline and an improved ability to context switch pipeline stage data. Long pipelines–such as the Netburst era Xeons of model numbers x1xx and x2xx–are more likely to suffer bubbles during context switches and are therefore penalized versus shorter pipeline products, such as the Core i7. Furthermore, by pushing and restoring pipeline stage data during a hardware context switch, the new HT can reduce pipeline bubbles.

But the gains vSphere users experience as a result of the new Hyper-Threading also comes from changes in ESX. ESX’s scheduler must make decisions as to when to co-locate two worlds on a physical core to take advantage of Hyper-Threading. In some conditions the scheduler will perform this co-location and in others it will allow a world to run on the core by itself. The decision to execute worlds concurrently instead of serially on a physical core can be informally called the scheduler’s trust of Hyper-Threading. The vSphere scheduler trusts Hyper-Threading more than the VI3 scheduler did. This amplifies the effect of HT.

I am now going to bore you with a disclaimer before I give you any data showing the effect of Hyper-Threading. The value of HT will vary from workload to workload and the ultimate authority of HT’s value is the end-user. The following numbers are the result of informal analysis and VMware that should only be used as a guide in your own analysis. Please do not make purchasing decisions on this information, which is devoid of the detail we would normally commit to a white paper.

In addition to the gains we informally cite here, I can say that we have not yet seen a workload where the new Hyper-Threading slows down consolidated performance. As far as we can tell, the new Hyper-Threading should be left enabled in 100% of virtualized environments.

I spend a great deal of time answering customers’ questions about the scheduler. Never have so many questions been asked about such an abstruse component for which so little user influence is possible. But CPU scheduling is central to system performance, so VMware strives to provide as much information on the subject as possible. In this blog entry, I want to point out a few nuggets of information on the CPU scheduler. These four bullets answer 95% of the questions I get asked.

Item 1: ESX 4′s Scheduler Better Uses Caches Across Sockets

On UMA systems at low load levels, virtual machine performance improves when each virtual CPU (vCPU) is placed on its own socket. This is because providing each vCPU its own socket also gives it the entire cache on that CPU. On page 18 of a recent paper on the scheduler written by Seongbeom Kim, a graph highlights the case where vCPU spreading improves performance.

The X-axis represents different combinations of VM and vCPU counts. SPECjbb is memory intensive and shows great gains with increases in CPU cache. The few cases that show dramatic benefit due to the ESX 4.0 scheduler are benefiting from the distribution of vCPUs across sockets. Very large gains are possible in this somewhat uncommon case.

Item 2: Overuse of SMP Only Slows Consolidated Environments At Saturation

For years customers have asked me how many vCPUs they should give to their VMs. The best guidance, “as few as possible”, seems too vague to satisfy. It remains the only correct answer, unfortunately. But a recent experiment performed by Bruce Herndon’s team sheds some light on this VM sizing question.

In this experiment we ran VMmark against VMs that were configured outside of VMmark specifications. In one case some of the virtual machines were given too few vCPUs and in another they were given too many. Because VMmark’s workload is fixed, increasing the VMs’ sizes does not increase the work performed by the VMs. In other words, the system’s score does not depend on the VMs’ vCPU count. Until CPU saturation, that is.

Notice that the scores are similar between the undersized, right-sized, and over-sized VMs. Up until tile 10 (60 VMs) they are nearly identical. There is a slight difference in processor utilization that begins to impact throughput (score) as the system runs out of CPU. At that point the additional vCPUs waste cycles which degrades system performance. Two points I will call out from this work:

• Sloppy VI admins that provide too many vCPUs need not worry about performance when their servers are under low load. But performance will suffer when CPU utilization spikes.
• The penalty of over-sizing VMs gets worse as VMs get larger. Using a 2-way VM is not that bad, but unneeded use of 4-way VMs when one or two processors suffice can cost up to 15% of your system throughput. I presume that unnecessarily eight vCPUs would be criminal.

Item 3: ESX Has Not Strictly Co-scheduled Since ESX 2.5

I have documented ESX’s relaxation of co-scheduling previously (Co-scheduling SMP VMs in VMware ESX Server). But this statement cannot be repeated too frequently: ESX has not strictly co-scheduled virtual machines since version 2.5. This means that ESX can place vCPUs from SMP VMs individually. It is not necessary to wait for physical cores to be available for every vCPU before starting the VM. However, as Item 3 pointed out, this does not give you free license to over-size your VMs. Be frugal with your SMP VMs and assign vCPUs only when you need them.

Item 4: The Cell Construct Has Been Eliminated in ESX 4.0

In the performance best practices deck that I give at conferences I talk about the benefits of creating small virtual machines over large ones. In versions of ESX up to ESX 3.5, the scheduler used a construct called a cell that would contain and lock CPU cores. The vCPUs from a single VM could never span a cell. With a ESX 3.x’s cell size of four this meant that VMs never spanned multiple four-core sockets. Consider this figure:

What this figure shows is that a 4-way VM on ESX 3.5 can only be placed in two locations on this hypothetical two-socket configuration. There are 12 combinations for a 2-way VM and eight for a uniprocessor VM. The scheduler has more opportunities to optimize VM placement when you provide it with smaller VMs.

In ESX 4 we have eliminated the cell lock so VMs can span multiple sockets, as item one states. Continue to think of this placement problem as a challenge to the scheduler that you can alleviate. By choosing multiple, smaller VMs you free the scheduler to pursue opportunities to optimize performance in consolidated environments

A few days after the conference a coworker of mine shared the following quote with me, courtesy of an article by David on Internet Evolution:

The fact is, most data center managers wouldn’t trust VMware to manage their Tier 1 applications because if something goes wrong performance-wise, you still need to roll in the VMware PhDs to solve it.

Let me respond to a few of the suggestions from this quote.

“Customers Do Not Trust VMware for Tier-1 Apps”

The following chart presents data collected from 676 VMware customers in July and August of 2008.

Percentage of application instances virtualized by VMware customers.

This graph shows the large rates of virtualization of the most well-known enterprise applications.  By any definition of “Tier-1 Application”, at least one tier-1 application is mostly virtualized by this customer sample.  And the survey date bears repeating: summer, 2008.  Virtualization acceptance has greatly increased in the past 12 months.

Concerns about performance management aside, VMware customers are virtualizing their tier-1 apps today.  So let’s talk about the process of performance troubleshooting.

“A PhD From VMware Is Required to Fix Performance Problems”

I think that David must have inferred from my confident and detailed talk on a great number of performance-related technical topics that I am the cream of the crop of America’s educational system.  For the record, I went to a state school in Alabama and spent far more time drinking beer than going to class.  Nonetheless, I am sure what he meant to say was…

“A Highly-skilled Performance Expert Is Required to Fix Performance Problems”

VMware now boasts over 150,000 customers, and I only interact with a relative handful a year.  If I count the experts in our small performance community I can conclude that our performance experts touch a very small percentage of our customer base each year.  That means that the great majority of our customers are solving their performance problems without engaging us.

Customers are fixing their problems using a variety of tools that I continue to document:

• The vSphere client interface to vCenter is known to everyone and its counters operate with 20s granularity and are effective at fixing about 90% of most performance problems.
• esxtop, with its finer granularity and larger counter list, can be used to fix 95% of problems, most of which could have been fixed with vCenter statistics.
• vscsiStats is extraordinarily useful for a small percentage of problems, perhaps 10-20% of those I see.

We are currently working on collecting all of these views into the client and adding a framework, vProbes, that will enable unprecedented visibility into operating systems and applications.  But even as things stand today, we have provided documentation and tools that all of our customers can use to fix any problem.  There is always room for improvement, but no PhD is required.