The discussions at the conference I attended in 2008 were incredibly broad.  From enterprise hardware to unreleased two-dimensional UPC codes to game consoles to mobile phones.  But the work I contributed two only aimed to provide one piece to the puzzle of worldwide information processing: the amount of data processed by enterprise hardware in 2008.

I joined the HMI team for a conference they hosted in the San Francisco Bay Area in 2008.  Work continued after I left, of course, and my place was taken by VMware’s Bruce Herndon, who runs the VMmark team.  With years of experience in enterprise benchmarking, Bruce was a much better choice than I to help quantify information processing.  And Jim recently restated to me his thanks to VMware and Bruce for their continued help on this project.

Jim recently shared with me the public report on enterprise server information from 2008,which was completed earlier this year.  Enterprise processing–which does not include game consoles, mobile phones, desktop computers, notebooks, tablets, etc.–obviously represents only a fraction of the world’s ongoing information processing.  Yet the amount of information that enterprise hardware alone processed in 2008 is staggering.

That number is 9.57 zettabytes.  That’s almost 10 to the 22nd power.  Or, as this entry’s title expands, 9,570,000,000,000,000,000,000 bytes.  Unbelievable.

The executive summary from HMI’s 2008 “Report on Enterprise Server Information”  includes some fun statistics.  And remember, these numbers apply only to the information processed by enterprise hardware:

• Servers process 12 GB of data for each worker in the world daily.  How much email are you sending?
• Two-thirds of the world’s information was processed by “low end” hardware, costing $25,000 or less.  It is an x86 world after all, isn’t it?
• About half of the information processing is attributed to transaction processing (invoicing, paying bills, checking stocks, etc.) and the other half is attributed to web content and office applications.
• Web services and business applications doubled their performance-per-cost every 1.5 years.  But enterprise servers only doubled their information processing every two years.  Enterprises are getting more value out of their processed information every year.
• High-end computers are doubling performance-per-cost and raw performance every four years, or half the pace of low-end computers.  It is certainly an x86 world.

The full HMI paper linked above is rich with information, which I will be processing on own various systems in the coming weeks.  I encourage you to check it out to understand the scale of data that pulses through our hardware every day.  And if you are really into the HMI project, read about an upcoming conference continuing this research that might benefit from your participation.

It was during discussions with advanced VMware customers in Melbourne that a solution to this problem occurred to me. I have reasoned it through and I think it holds water. I have socialized it with more customers and my colleagues and we think it stands. So I want to introduce a system for implementing virtual machines with a better assurance of a performance SLA.

The key to this process is that minimum performance can be measured using limits and that performance can be assured using reservations. You can develop and document virtual machines with performance SLAs using the following procedure:

• First, as always, define a small number of strictly-sized virtual machines to be used by all applications in your environment. Often these look something like small VMs of 1 vCPU and 4 GB RAM, medium VMs of 2 vCPUs and 8 GB of RAM, and large VMs of 4 vCPUs and 16 GB of RAM. Tune these numbers for your environment, as needed.
• For any application, benchmark its maximum performance against each of these virtual machine configurations on an unloaded system. Chose an ISV-supplied benchmark or a well-known third party tool. This sets your high water mark for throughput for each application in its virtual machine.
• For each configuration, set a CPU limit at 50% of the available CPU and a memory limit of 50% of the available memory. Retest the application against this smaller, limited configuration.
• During the applications’ deployment, change the limits to reservations. That is, remove limits and set reservations equal to the limits’ previous values, in this case 50%.
• Your application now has a maximum performance defined in bullet two, and a “guaranteed” performance measured in bullet three.  This is your application’s performance SLA.

The concept is simple: limits can be used to measure the performance of an application in the presence of that degree of contention. Reservations ensure that those resource amounts are always present. Here are some notes on this process:

• This is not a true guarantee since network and storage throughput may drop. No tool can eliminate this risk entirely but SIOC and NetIOC can reduce the risk of a network- or storage-induced performance failure.
• The memory test is going to be highly dependent on the working set created by your load generation tool. Your mileage will vary depending on your application owners’ use of the virtual machine.
• vCenter will guarantee that the reservations are always available through a process called admission control, which checks the cluster to ensure that enough CPU or memory is available to run the virtual machine immediately and in the event of a server failure.

As I said above, this is not a true guarantee of application performance. But it is as close as we can get until AppSpeed or a replacement evolves into universal application latency measurement that is fed into vCenter. And this is another in a growing list of reasons why CPU and memory reservations should be part of all VMware deployments.

VMmark is clearly the reigning king of consolidation benchmarks and anything that enters its arena must stand against its standard. VMmark pioneered a new method of benchmarking that resonates with virtualization experts. It tests system performance by adding fixed load virtual machines instead of scaling up a single application to system saturation.  Traditional benchmarks tune up their load generation against a single instance but VMmark piles on virtual machines until the system is capable of no more work.

VMmark is one of VMware’s many industry-leading initiatives and was started when VMware worked closely with server vendors that wanted to benchmark their servers’ ability to run virtual machines. VMmark was conceived many years ago, well before VMware had competition. It is because of this fact that I scratch my head at claims that VMmark is biased towards VMware. There was no commercial implementation of Xen when VMmark was specified and Microsoft was only dreaming of entering the market.

But even in an environment devoid of competition, customers want certainty that their benchmarks are not hiding flaws in a product. SPEC has for years been developing honest benchmarks that survive the crucible of debate among its large member community. SPECvirt, or more properly SPECvirt_sc2010, is the result of this vigorous debate. You can read up on SPECvirt in the FAQ released coincident with the product’s launch. But I will add a few comments and comparisons here.

1. SPECvirt costs $3000 to purchase.  VMmark is free.  But VMmark requires commercial software and versions of SPEC benchmarks that are not free.  Depending on your licensing model, you may find VMmark or SPECvirt cheaper.  But the prices of each are essentially comparable.
2. VMmark uses the most common applications in the data center (like Apache and Microsoft Exchange).  SPECvirt does not mandate application choice for the system under test.
   • This is a Good Thing, because you may now choose a configuration that models your environment by running the exact applications you run.
   • This is a Bad Thing, because five different testers may choose five different application sets in their tests resulting in incomparable results.
3. SPECvirt cannot be run against a cluster of hosts.  But VMmark cannot, either.  We will have to wait for an update to one of these benchmarks before we can properly test DRS clusters and their competitive equivalents.
4. There is only one published SPECvirt result, courtesy of IBM running KVM.  There are a boatload of VMmark results, as one would expect of a more mature product.  It will be interesting to watch the rate of submissions of these two benchmarks over the coming year or two.
5. SPECvirt runs three workloads and an idle virtual machine in its tile.  One of those workloads, tested by SPECweb, is implemented with three virtual machines.  The end product is a six-VM tile that looks very much like VMmark’s six-VM tile.

For years we have seen online and in-person griping about VMware’s misunderstood benchmark restriction in its EULA.  Both VMmark and SPECvirt can be run on any supported hypervisor.  So now its time for all the hypervisor vendors to put up or shut up.  Run one of these benchmarks on your product and compare the results against existing published results.  Then the world will know where your product stands.

Here are a few interesting stats about the published TPC-H score:

• The score of 1,316,882 QphH is the highest ever reported throughput for the 1,000 GB results.  It is 13% higher than the next greatest performance score.
• The $0.70 price per QphH is the lowest price/performance ratio ever submitted on the 1,000 GB results.  It is 40% less than the next closest price/performance score.
• ParAccels results are a mere 1/8 the price/performance of the second best performing result.
• The vSphere 4 configuration used 40 HP servers with Xeon 5560 processors running 80 4-way virtual machines.

ParAccel’s amazing work represents a large number of firsts for VMware: our first audited TPC result, our first proof point with big data applications, our first published partnered work in the rapidly growing data warehouse space.  One more amazing thing about this accomplishment: ParAccel and VMware set this record on commodity x86 hardware, which means the infrastructure can be shared with other enterprise applications.

Congrats to ParAccel.  They have shown a product that can set records not in spite of virtualization, but because of it.

The scalability limitations in web servers were investigated and documented by Sreekanth Setty, whose VMworld 2008 presentation is summarized in this graph:

A multiple virtual machine configuration can outperform native with web servers.

This graph shows four uniprocessor virtual machines outperforming a single four core native configuration. The advantage of the multiple virtual machine configuration improves as virtual machine count increases.  Sree has since shown that ESX can beat physical when tested by SPECweb using 4-way virtual machines, which means fewer OS and web server instances to maintain. Most recently Sree updated his results again using an Intel Xeon 5570 and VMDirectPath, which freed a few precious CPU cycles for more web transactions.

It is now beyond doubt that multiple web servers will outperform a single instance on the same hardware. However, some of our customers still challenge our suggestion that those web servers should be run in virtual machines.  Some resist the idea of increasing the number of web and OS instances at all.  So, let me run down the options here and their ramifications:

1. Single web server instance in an OS on a physical server. This is the worst performing configuration, scaling to only 38% of the system’s maximum possible performance.
2. Multiple web server instances in a single OS on a physical server. This would deliver the theoretical best possible performance but requires the admin to run the notoriously uncooperative web servers together in a single operating system.  I am no web server administrator, but our customers have told me that this is a poor solution for production.
3. Multiple web server instances each in their own virtual machine. Excellent performance with web server instances isolated so they cannot harm each other.  Sreekanth’s VMworld 2008 work showed this delivering about 80% of the system’s maximum theoretical throughput.

Since the performance the first configuration is so poor I exclude it from consideration in a data center.  By instantiating one 3-way virtual machine for a web server, we have nearly matched the physical configuration with no additional maintenance cost.  Then we can scale out the virtual web farm using vSphere management tools to deliver a flexible architecture.  And by isolating these instances in their own hardened virtual machines, we can guarantee that runaway utilization, zombie processes, and any other cross contamination caused by bad web servers will not harm the farm.

Among the many results presented by VRC, the configurations that most perplexed us were the two and four virtual machine configurations, each with four vCPUs per virtual machine.  The configuration with two virtual machines looked good and matched our internal numbers.  In this configuration there are a total of eight vCPUs on the host which maps each to its own physical core on the Xeon 5500 series processor.  The problem arose when the virtual machine count was increased to four, resulting in 16 total vCPUs.  In this configuration each vCPU is paired with one logical, Hyper-Threaded core.  Project VRC showed this configuration supporting no more desktops than the two-VM configuration, which suggests no value to Hyper-Threading on this configuration.

It took us some time to understand the reason for these results, but we eventually identified a very specific condition where ESX’s scheduler enforces fairness in scheduling vCPUs at at cost of throughput.  ESX’s scheduler has long be subject of the intensive scrutiny of a large number of VMware engineers to guarantee fair access to the processor for each virtual machine.  It is because of this fairness that VMware’s customers can rely on CPU resource controls.  But, when fairness goes too far, throughput may be sub-optimal.

Hyper-Threading presents particular problems to fairness because of the non-linear performance it delivers.  A thread will run at one speed when it has full access to a physical core, at another speed when it is sharing a core, and at third speed when sharing a core with a different thread.  As a result, ESX’s scheduler will sometimes pause a thread to enforce fairness.  These pauses are more common when Hyper-Threading is present to account for its lack of uniformity in thread performance.  If the host lacks vCPUs that are ready to run, the result is CPU utilization below saturation, leaving CPU cycles unused.

There are three specific conditions that can excite this condition:

1. A Xeon 5500 series processor is present with Hyper-Threading enabled,
2. CPU utilization is near saturation, and
3. A roughly one-to-one mapping between vCPUs and logical processors.

In this scenario, VMware vSphere favors fairness over throughput and sometimes pauses one vCPU to dedicate a whole core to another vCPU, eliminating gains provided by Hyper-Threading.  In cases outside of these three conditions, the performance of VMware vSphere 4 meets the high expectations of VMware’s R&D team and its customers.  Of course production environments rarely (never?) have a one-to-one ratio of vCPUs to logical processors.  This occurs when there are only four 4-way virtual machines on a Xeon 5500 system, for example.

But environments such as Project VRC’s are simplifications of production environments meant to understand the capabilities of virtual platforms.  VMware has provided a patch to Project VRC that will allow them to improve throughput in their environment.  We are going to release this patch and its documentation to the general public within a couple of weeks.  I do not expect that any of VMware’s customers will benefit from the changes is allows, but I will later document the patch and its usage for anyone that cares to experiment.

The story of Project VRC’s work goes back nearly a year.  This group of Dutch consultants created an ambitious project to quantify the performance of desktop virtualization platforms.  Their workload simulates multiple users running desktop applications in a Terminal Services environment. A few operations—including window appearances, Windows calculator load time, and others—are timed and the number of users is increased until a specified response time limit is reached.

While this may appear to be straightforward, benchmarking is deceptively difficult, as Project VRC will attest. Through engagements with VMware they have identified a performance gain by using ESX’s hardware assist monitor mode and acknowledged a timing bias in XenServer.  Furthermore, Project VRC has engaged other partners to improve the workload, including minimizing “stuck sessions” and reducing the impact of in-guest sleeps, which cause a variety of problems.

With this new paper, we can now make a few observations about the Project VRC workload and the platforms it runs on:

• The previously reported XenServer numbers were incorrect.  The old results showed a significant performance lead for XenServer but the update reports that vSphere 4.0 outperforms XenServer 5.5 by 3.5%.
• Project VRC has recognized the inaccuracy of sleep system calls in virtual environments so they have removed them from within timed operations.
• Some system sleeps remain in the benchmark, which will result in unpredictable operation density from run to run and across platforms.

Project VRC plans to continue to develop their workload until it closely matches customer environments.  This includes fixing any remaining in-guest timing issues that may continue to bias their results.  VMware will continue to work with them to help them eliminate issues and produce a benchmark we can all stand behind.

Microsoft SQL Server runs at better than 80% of native on VI3 in most benchmarked environments. In production environments, and under loads that model those conditions, SQL Server runs at 90-95% of native on ESX 3.5. I can say this with confidence despite a large amount of the industry’s skepticism because I’ve spent so much time on SQL Server in the past half year. I’d like to share some of my research on the subject and observations with you.

Two weeks ago my colleague Chethan Kumar and I presented on SQL Server in Cannes, France for VMworld Europe 2009. This presentation was the culmination of six months of investigation that was started at VMworld 2008 in Las Vegas. At that event I heard many customer concerns about SQL Server performance that I was resolved to identify the problems’ root causes. I talked with every customer I could find that claimed that SQL ran at anything less than 70% of native.  So many of these contacts claimed that they had measured SQL at 25% of native or worse that I knew that something was going wrong.

First, let me show you a slide that Chethan presented at the show in Cannes:

Chethan spent three months investigating SQL Server to find out how much he could improve virtual performance from the “out of the box” experience. As this figure details, the sum total of performance improvements was 15%. Here’s another break-down of these results:

The only option that we found in ESX to improve virtual performance was static transmit coalescing, which is documented on page four of one of our SPECweb papers. Large pages and SQL’s priority boost, which are best practices provided by Microsoft for SQL Server configuration, provide the largest gains in performance.

The key messages that we communicated to our audience were that a properly running SQL Server should run at 80% of native or better. In most production cases it can run at a performance indistinguishable from native speed. And if performance is lagging, there don’t exist many changes that can be made to ESX that can yield and performance gains at all.

This begs the question: “If ESX can’t be tuned to double SQL performance, what is causing these reports of terrible SQL Server throughput?” The great majority of the problems are coming from mis-configured storage. But a variety of other items such as poor hardware selection or use of the wrong virtualization software contribute to the confusion, as well. I’ve been documenting these issues in Best Practices for SQL Server on this community and will continue to update that document as more problems are discovered.

If you have a SQL Server running un-virtualized in your environment, I’d like you to try virtualizing it again. Follow our best practices document and pay close attention to your storage configuration during deployment. I feel confident that once you’ve setup your environment properly, you’re going to like what you see.