A perfunctory bit of guidance I include in nearly all of my performance talks (such as the enthralling, entertaining, and cancer-curing* presentations from VMworld 2010 that I will repeat in Copenhagen from 12-14 October) is “follow your application best practices”.  Audiences usually nod and immediately forget because this recommendation we all know to be correct yet somehow ignore.  In that way it is like, “stay away from fatty foods”, “do not drink wine with pain killers”, or “pay attention during the flight attendants’ presentation”.

Part of the reason why people forget this nugget is because advice is general, and not crystalized in a technological explanation that embeds deep in the minds of the audience.  In this case the application best practice that should be followed is to separate data from logs, putting the data on something good for random read performance (like RAID 5) and the logs on something good for sequential write performance (RAID 10).  Obviously I want everyone to consolidate their storage to VMFS and enjoy the technology, but if you are putting VMDKs that contain each of these files on the same volume, you are ignoring application best practices.

In this case I recommend building two VMFS volumes.  One backed by RAID 5 and the other by RAID 10.  Put the data on RAID 5, the logs on RAID 10.  While you will change the access profile at the array by putting multiple log files on the same RAID 10 backed LUN, the resultant IO will be much more sequential write than had you mixed data file reads among them.  So, consolidate multiple data files onto the same RAID 5 LUN and consolidate multiple log files on the same RAID 10 LUN.

Furthermore, if you are using solid state auto-tiering to manage your volumes, you do not need to protect your database log file with this technology.  What I am talking about here is EMC’s Fully Automated Storage Tiering (FAST), which is the most popular thing EMC has created since I have been paying attention.  Despite what some people will tell you, solid state disks are the cheapest way to serve huge amounts of random reads.  But their benefits diminish when the profile is sequential write when they become unattractive from a cost perspective.

EMC’s FAST works by creating a volume that is like a vertical stripe of multiple RAID groups.  LUNs, which become VMFS volumes, are then placed in that FAST volume.  Since FAST is a great technology for solid state disks, RAID 5 is the most cost efficient configuration for database data, and solid state is wasted on sequential IO such as redo logs, my best practice for virtual storage configuration for databases workloads when FAST may be present can be boiled down to the following rules:

• Always create RAID 5 volumes for your read-intensive database data.
• Always create RAID 10 volumes for your database logs.  If you have write-intensive data, you may consider putting them here, too.
• If you have FAST, use it to stripe across multiple RAID 5 volumes of different disk types and put your random, read-intensive data on VMFS on this volume.

The last bullet is clearly the most important here. I really love FAST, and it seems that EMC’s customers are crazy for it.  But its not the technology you need for sequential write workloads like redo logs.  Separate those data onto their own “normal” (not FAST-backed) VMFS volumes that use no SSDs.  Then you will have the best of all worlds: optimally deployed disk technologies, application best practice compliance, and righteous virtualized database consolidation.

(*) The claims made by the author of this blog do not reflect the views of his employer, the conference organizers, the government of the Kingdom of Denmark, or reality, for that matter.

All kidding aside, yesterday I wrote an article about a sneaky trick to leverage improved hardware assist performance on ESX 3.5 virtual machines that defaulted to binary translation. I learned very late in the day that the recommended guidance only works for AMD processors. The text below is from the original post but has since been updated to reflect my newly discovered information.

Begin Updated Article

Do you have any running instances of ESX 3.5 or older?  Are those instances running on AMD processors that are no more than a couple of years old?  If so, I have a tip for you: force hardware assist in those virtual machines.  In most situations application performance will improve by 10% or more.  Details follow.

ESX’s monitor presents virtual hardware to virtual machines’ guest operating systems.  VMware’s multi-mode monitor uses three technologies to do this: hardware assist, para-virtualization, and binary translation.  Hardware assist has gotten much faster over the years, as this figure demonstrates.

The latency of the VMEXIT instruction is shown on Intel VT systems. The longer this instruction takes to execute, the worse the virtual machine performs.

Johan De Gelas included his take on monitor mode performance when he reported “Virtualization Round Trip Latency” in a recent article on the new Xeon 5600.  His results reiterate the trend I have been sharing with my audiences for over a year now.

Because hardware assist was once so slow, older versions of ESX would utilize our faster-performing binary translation in many situations.  But virtualization assist in today’s processors–and here I am talking about Intel and AMD processors manufactured in the past two years–is generally faster than binary translation. This means your virtual machines running on ESX 3.5 on shiny new processors may not be reaching their full potential performance.

The fix is simple: force hardware assist for your ESX 3.0 and ESX 3.5 virtual machines running on newer AMD processors.  You can do this with the following lines in your virtual machines’ VMX files:

monitor.virtual_mmu = hardware

A reboot is then required. But this setting only works for AMD processors, where RVI (AMD’s hardware memory management unit) is available. With this setting both AMD-V and RVI are forced on. This setting is ignored on Intel processors, whose hardware MMU is not leveraged on ESX 3.5.

These changes are not needed with vSphere because its default monitor modes favor hardware assist more than ESX 3.5 and earlier. You can see vSphere’s default monitor modes in our wonderful monitor modes paper.

As one example of the magnitude of performance increase this small change can produce, look to the SQL Server performance paper we released last year.  Here is one graph I lifted from that document.

This figure shows AMD-V improving performance by 18% over binary translation on virtual machines running SQL Server 2008.  More gain is possible when the hardware memory management unit is utilized.

Because of ESX 3.5′s continued wide deployment, it may very well be running millions of virtual machines.  Many those virtual machines are running on newer AMD processors that can benefit from this change.  Go forth and reconfigure those virtual machines to claim the performance to which you are entitled!

In 2009 VPACT paper, VMware engineers showed application performance when the storage is consolidated.  This paper is a bit academic for the average virtualization nut, but does contain insights into choosing which VMDKs to consolidate into a single VMFS volume.  It does this by running applications that contain random or sequential IO and comparing performance with isolated (dedicated) storage to performance using consolidated storage.

The first experiment tested DVD store against Microsoft SQL Server and Oracle Swingbench OLTP against an Oracle database.  These OLTP workloads result in random IO on the data disks.  In the isolation experiment each virtual machine was run with its VMDK on a three-disk RAID 5 LUN (2+1).  In the consolidation experiment, both virtual machines’ VMDKs were put on a common six-disk RAID 5 LUN (5+1).  Here are the paper’s results from table 1 of the paper:

Consolidating Random Access Virtual Disks

The “application metric”, transactions per minute, is the most important indicator of the end user’s observed performance.  You can see from the results that consolidating (sharing) storage of random workloads does not harm performance at all.  In fact, SQL Server performance increased by 25%, reflecting the relative increase in data disks per stripe.

The second experiment again used DVD Store against SQL Server for a random IO workload.  But instead of a second random workload, an Oracle database was tested by the Swingbench Decision Support System, which results in highly sequential access.  Here are the results of that experiment, taken from table 2 of the paper:

Sequential Workload Consolidation

The random workload, SQL plus DVD Store, again improved as the relative percentage of data disks in the RAID volume increased.  But the Decision Support System workload, so heavily dependent on sequential storage performance, suffered greatly.  DSS performance dropped 30% when measured by IO throughput and 50% when measured by completed transactions.

Applications with a sequential storage access pattern can be heavily dependent on the array’s ability to coalesce IO requests and complete large numbers of IOs very rapidly.  But when a VI admin includes a random access VMDK on the same LUN, the aggregate, interleaved LUN access is no longer sequential.  This slows down the array’s sequential and its effects are profound at the application level.

There are two summary recommendations from this experiment:

• A smaller number of RAID 5 volumes using many disks will outperform a larger number of RAID 5 volumes that use fewer disks.  This is due to the relative decrease in parity on the configuration.
• VMDKs with random access can be consolidated to a single VMFS volume safely but sequential access pattern files should be separated to their own LUNs.

The VMware performance team has a lot more to say about storage design to maximize application performance in virtual environments.  I will have more blog articles as the weeks progress and a white paper to share in the second quarter of 2010.  I expect it to be ready no later than EMC World 2010.

Testing results from internal and customers showed SAP workloads: 85-95, Oracle OLTP: 80-92% bare metal. LAMP stack showed better than bare metal performance. Whitepapers will be published in how this was achieved. Java achieved up to 94% bare metal.

Frankly, I was surprised to hear this.  KVM is a hosted virtualization platform, equivalent to the free VMware Server, which runs on top of a host operating system.  VMware server is fine for a virtual machine or two, but you would not want it hosting your critical business applications.  The above KVM claim suggests that KVM possesses hypervisor-like performance.  So we ran a test with a few virtual machines to see what we could learn.  These tests confirmed my suspicions: KVM is a very long way from enterprise-class virtualization performance.

The thing to remember about virtualization benchmarking is that any vendor can provide virtualization software (hosted or hypervisor) that can virtualize a single application at better than 80% of native performance.  VMware has been doing this for a decade.  But it is extraordinarily difficult to build a hypervisor that can scale with many virtual machines.  Maybe this is one reason why you have never seen Microsoft or Citrix post results from a consolidated workload.  But I digress.

We decided that the easiest way to test this environment with a light/moderate enterprise workload is to use two or three VMs running SQL Server, as tested by DVD Store 2 (DS2).  We tried four configurations of these VMs:

• Case A: Two 4-way virtual machines.
• Case B: Two 3-way VMs and one 2-way.
• Case C: Three 3-way VMs.
• Case D: Three 4-way VMs.

Each virtual machine ran on an HP DL380 G5 and was given 4 GB.

Finding the right number of threads per virtual machine took some time.  Threads on the DS2 client determine the volume of transactions that are generated against the SQL Server.  We wanted to get the highest throughput for a reasonable latency, which we set at 33 ms.  Here are the best numbers I could produce for vSphere and KVM.

The very best performance that KVM could muster was only removed% of vSphere’s performance on the same configuration.  Notice that at 50% CPU over-commitment (1.5 vCPUs for each CPU), KVM’s performance removed.  It’s throughput fell to removed% of vSphere and its response time removed.  Increasing threads in this configuration actually made throughput and latency worse.

I had suspected that KVM would show hosted platform performance, as it relies on a host operating system.  It appears my suspicions were correct.  It will be tough for Red Hat to sell this product as part of an enterprise product.  To do so they will likely publish results based on single virtual machines and in environments where the CPUs are under-committed.

Lastly, this is the only workload that we have attempted.  I would expect KVM to do much worse when more virtual machines are part of the test or if network or storage throughput becomes significant.  But we have no plans to spend time on KVM benchmarking.  As I mentioned in my performance debate at Catalyst 2009, I think that each vendor should do its own benchmarking to best represent its products.  I challenge Red Hat to post a KVM number using TPC, SPECweb, VMmark, vConsolidate, or any enterprise-class workload.  Customers should expect nothing less of their virtualization vendor.

10/2/09 Update

I decided to remove the KVM results to allow Red Hat or a KVM enthusiast to show their own best results on a consolidated workload.  I recommend VMmark or vConsolidate.

Back in July I had the privilege of riding along with VMware’s Professional Services Organization as they piloted a possible performance offering. We are considering two possible services: one for performance troubleshooting and another for infrastructure optimization. During this trip we piloted the troubleshooting service, focusing on the customer’s disappointing experience with SQL Server’s performance on vSphere.

If you have read my blog entries (SQL Server Performance Problems Not Due to VMware) or heard me speak, you know that SQL performance is a major focus of my work. SQL Server is the most common source of performance discontent among our customers, yet 100% of the problems I have diagnosed were not due to vSphere. When this customer described the problem, I knew this SQL Server issue was stereotypical of my many engagements:

“We virtualized our environment nearly a year ago and and quickly determined that virtualization was not right for our SQL Servers. Performance dropped by 75% and we know this is VMware’s fault because we virtualized on much newer hardware on the exact same SAN. We have since moved the SQL instance back to native.”

Most professionals in the industry stop here, incorrectly files this problem as a deficiency of virtualization, and move on with their deployments. But I know that vSphere’s abilities with SQL Server are phenomenal, so I expect to make every user happy with their virtual SQL deployment. I start by challenging the assumptions and trust nothing that I have not seen for myself. Here are my first steps on the hunt for the source of the problem:

1. Instrument the SQL instance that has been moved back to native to profile its resource utilization. Do this by running Perfmon to collect stats on the database’s memory, CPU, and disk usage.
2. Audit the infrastructure and document the SAN configuration. Primarily I will need RAID group and LUN configuration and an itemized list of VMDKs on each VMFS volume.
3. Use esxtop and vscsiStats to measure resource utilization of important VMs under peak production load.

There are about a dozen other things that I could do here, but my experience in these issues is that I can find 90% of all performance problems with just these three steps. Let me start by showing you the two RAID groups that were most important to the environment. I have greatly simplified the process of estimating these groups’ performance, but the rough estimate will serve for this example:

We found two SQL instances in their environment that were generating significant IO: one that had been moved back to native and one that remained in a virtual machine. By using Perfmon for the native instance and vscsiStats the virtual one, we documented the following demands during a one-hour window:

In the customer’s first implementation of the virtual infrastructure, both SQL Servers, X and Y, were placed on RAID group A. But in the native configuration SQL Server X was placed on RAID group B. This meant that the storage bandwidth of the physical configuration was approximately 1850 IOPS. In the virtual configuration the two databases shared a single 800 IOPS RAID volume.

It does not take a rocket scientist to realize that users are going to complain when a critical SQL Server instances goes from 1050 IOPS to 400. And this was not news to the VI admin on-site, either. What we found as we investigated further was that virtual disks requested by the application owners were used in unexpected and undocumented ways and frequently demanded more throughput than originally estimated. In fact, through vscsiStats analysis, my contact and I were able to identify an “unused” VMDK with moderate sequential IO that we immediately recognized as log traffic. Inspection of the application’s configuration confirmed this.

Despite the explosion of VMware into the data center we remain the new kid on the block. As soon as performance suffers the first reaction is to blame the new kid. But next time you see a performance problem in your production environment, I urge you to look at the issue as a consolidation challenge, and not a virtualization problem. Follow the best practices you have been using for years and you can correct this problem without needing to call me and my colleagues to town.

Of course, if you want to fly us out for to help you correct a specific problem or optimize your design, I promise we will make it worth your while.

Newer processors are much more important to virtualized environments than the non-virtualized counterpart. Generational improvements have not just increased the raw compute power, they have also reduced virtualization overheads. This blog entry will describe three key changes that have particularly impacted virtual performance.

Hardware Assist Is Faster

In 2008, with the launch of the Opteron 1300, 2300 and 8300 parts, AMD became the first CPU vendor to produce a hardware memory management unit equipped to support virtualization. They called this technology Rapid Virtualization Indexing (RVI). This year Intel did the same with Extended Page Tables (EPT) on its Xeon 5500 line. Both vendors have been providing the ability to virtualize privileged instructions since 2006, with continually improving results. Consider the following graph showing the latency of one key instruction from Intel:

This instruction, VMEXIT, is called each time the guest exits to the kernel. The graph shows its latency (delay) in completing this instruction, which represents a wait time incurred by the guest. Clearly Intel has made great strides in reducing VMEXIT’s wait time from its Netburst parts (Prescott and Cedar Mill) to its Core architecture (Merom and Penryn) and on to its current generation, Core i7 (Nehalem). AMD processors have shown commensurate gains with AMD-V.

In a recent white paper detailing SQL Server on vSphere, the following graph showed the gains derived by using AMD-V in the Opteron 8324 (Shanghai).

Binary translation, AMD-V, and AMV-V plus RVI are measured using SQL Server.

This graph shows the practical value of the great gains that CPU manufacturers have made with virtualization assist.  Hardware assist can now be regularly relied upon for great performance.

Pipelines Are Shorter

The longest pipelines in the x86 world were in Intel’s Netburst processors. These processor’s pipelines had twice as many stages at their counterparts at AMD and twice as many as the generation of Intel CPUs that followed. The increased pipeline length would have enabled support for 8 GHz silicon, had it arrived. Instead, silicon switching speeds hit a wall at 4 GHz and Intel (and its customers) were forced to suffer the drawbacks of large pipelines.

Large pipelines are not necessarily a problem for desktop environments, where single threaded applications used to dominate the market. But in the enterprise, application thread counts were larger. Furthermore, consolidation in virtual environments drove thread counts even higher. With more contexts in the processor, the number of pipeline stalls and flushes increased, and efficiency fell.

Because of decreased efficiency of consolidated workloads on processors with long pipelines, VMware has often recommended that performance-intensive VMs be run on processors no older than 2-3 years. This excludes Intel’s Netburst parts. VI3 and vSphere will do a fine job at virtualizing your less-demanding applications on any supported processors. But you should use newer parts for applications that hold your highest performance expectations.

Caches Are Larger

A cache is highly effective when it fully contains the software’s working set. The addition from the hypervisor of even a small about of code will change the working set and reduce cache hit rate. I’ve attempted to illustrate this concept with the following simplified view of the relationship between cache hit rates, application working set, and cache sizes:

Performance drops with small cache systems for even small increases to working set size.

This graph is based on a model that greatly simplifies working sets and the hypervisor’s impact on them. Assuming that ESX increases the working set by 256 KB, this graph shows the decrease cache hit rate due to the contributions of the hypervisor. Notice that with very small caches and very small application working sets, the cache hit rate suffers greatly due to the addition of even 256 KB of virtualization code. And even up to 2 MB, a 10% decrease in cache hit rate can be seen in some applications. With a 256 KB contribution by the kernel, cache hit rates do not change significantly with cache sizes of 4 MB and beyond.

In some cases a 10% improvement in cache hit rate can double application throughput. This means that a doubling of cache size can profoundly effect the performance of virtual applications as compared to native. Given ESX’s small contribution to the working set, you can see why we at VMware recommend that customers run their performance-intensive workloads on CPUs with 4 MB caches or larger.

Microsoft SQL Server runs at better than 80% of native on VI3 in most benchmarked environments. In production environments, and under loads that model those conditions, SQL Server runs at 90-95% of native on ESX 3.5. I can say this with confidence despite a large amount of the industry’s skepticism because I’ve spent so much time on SQL Server in the past half year. I’d like to share some of my research on the subject and observations with you.

Two weeks ago my colleague Chethan Kumar and I presented on SQL Server in Cannes, France for VMworld Europe 2009. This presentation was the culmination of six months of investigation that was started at VMworld 2008 in Las Vegas. At that event I heard many customer concerns about SQL Server performance that I was resolved to identify the problems’ root causes. I talked with every customer I could find that claimed that SQL ran at anything less than 70% of native.  So many of these contacts claimed that they had measured SQL at 25% of native or worse that I knew that something was going wrong.

First, let me show you a slide that Chethan presented at the show in Cannes:

Chethan spent three months investigating SQL Server to find out how much he could improve virtual performance from the “out of the box” experience. As this figure details, the sum total of performance improvements was 15%. Here’s another break-down of these results:

The only option that we found in ESX to improve virtual performance was static transmit coalescing, which is documented on page four of one of our SPECweb papers. Large pages and SQL’s priority boost, which are best practices provided by Microsoft for SQL Server configuration, provide the largest gains in performance.

The key messages that we communicated to our audience were that a properly running SQL Server should run at 80% of native or better. In most production cases it can run at a performance indistinguishable from native speed. And if performance is lagging, there don’t exist many changes that can be made to ESX that can yield and performance gains at all.

This begs the question: “If ESX can’t be tuned to double SQL performance, what is causing these reports of terrible SQL Server throughput?” The great majority of the problems are coming from mis-configured storage. But a variety of other items such as poor hardware selection or use of the wrong virtualization software contribute to the confusion, as well. I’ve been documenting these issues in Best Practices for SQL Server on this community and will continue to update that document as more problems are discovered.

If you have a SQL Server running un-virtualized in your environment, I’d like you to try virtualizing it again. Follow our best practices document and pay close attention to your storage configuration during deployment. I feel confident that once you’ve setup your environment properly, you’re going to like what you see.