In 2009 VPACT paper, VMware engineers showed application performance when the storage is consolidated.  This paper is a bit academic for the average virtualization nut, but does contain insights into choosing which VMDKs to consolidate into a single VMFS volume.  It does this by running applications that contain random or sequential IO and comparing performance with isolated (dedicated) storage to performance using consolidated storage.

The first experiment tested DVD store against Microsoft SQL Server and Oracle Swingbench OLTP against an Oracle database.  These OLTP workloads result in random IO on the data disks.  In the isolation experiment each virtual machine was run with its VMDK on a three-disk RAID 5 LUN (2+1).  In the consolidation experiment, both virtual machines’ VMDKs were put on a common six-disk RAID 5 LUN (5+1).  Here are the paper’s results from table 1 of the paper:

Consolidating Random Access Virtual Disks

The “application metric”, transactions per minute, is the most important indicator of the end user’s observed performance.  You can see from the results that consolidating (sharing) storage of random workloads does not harm performance at all.  In fact, SQL Server performance increased by 25%, reflecting the relative increase in data disks per stripe.

The second experiment again used DVD Store against SQL Server for a random IO workload.  But instead of a second random workload, an Oracle database was tested by the Swingbench Decision Support System, which results in highly sequential access.  Here are the results of that experiment, taken from table 2 of the paper:

Sequential Workload Consolidation

The random workload, SQL plus DVD Store, again improved as the relative percentage of data disks in the RAID volume increased.  But the Decision Support System workload, so heavily dependent on sequential storage performance, suffered greatly.  DSS performance dropped 30% when measured by IO throughput and 50% when measured by completed transactions.

Applications with a sequential storage access pattern can be heavily dependent on the array’s ability to coalesce IO requests and complete large numbers of IOs very rapidly.  But when a VI admin includes a random access VMDK on the same LUN, the aggregate, interleaved LUN access is no longer sequential.  This slows down the array’s sequential and its effects are profound at the application level.

There are two summary recommendations from this experiment:

• A smaller number of RAID 5 volumes using many disks will outperform a larger number of RAID 5 volumes that use fewer disks.  This is due to the relative decrease in parity on the configuration.
• VMDKs with random access can be consolidated to a single VMFS volume safely but sequential access pattern files should be separated to their own LUNs.

The VMware performance team has a lot more to say about storage design to maximize application performance in virtual environments.  I will have more blog articles as the weeks progress and a white paper to share in the second quarter of 2010.  I expect it to be ready no later than EMC World 2010.

Number 5: Comparing Intel Dual-Core and Quad-Core Using VMmark

This article received 4,700 views in 2009.  I am not surprised that this two-year-old article is so popular, though.  Even this morning at Intel’s live chat I was asked for advice on selecting core/socket configurations.  Everyone wants to know if the cores on the largest servers can be put to good use.  The article summed up findings with this graph:

Doubling cores or sockets does not exactly double performance.  In this case the quad-core system produced 70% more throughput than its equivalent dual-core configuration.

Number 4: 100,000 I/O Operations Per Second, One ESX Host

In VMware’s early days customers were concerned that ESX lacked the storage throughput for demanding workloads.  In our first effort to dispel this rumor, we partnered with EMC to demonstrate a handful of VMs driving 100,000 IOPS on a single host.  A year later we updated those results on vSphere and showed VMs on ESX demanding 365,000 IOPS from EMC’s Enterprise Flash Devices (EFDs).  But the original article remains the more popular and has received 8,000 hits in 2009.

Number 3: Virtualizing XenApp on XenServer 5.0 and ESX 3.5

I guess that people love a good argument.  Our customers have heard enough misinformation on vSphere and XenServer performance that our performance team wanted to weigh in with a rare competitive comparison.  That article has received 13,400 hits in 2009.  It details an experiment in which we used a workload that will eventually be released as View Planner to demonstrate scalability performance of VMware’s and Citrix’s respective products.

Desktop and terminal services performance is continually a hot topic.  Unlike enterprise applications, there is no industry standard for measuring virtual desktop performance.  VMware has favored an approach that uses many operations on the most common desktop applications.  Every other public comparison either restricts is measurements to a single operation or favors easy automation over application relevancy.

Number 2: Ten Reasons Why Oracle Databases Run Best on VMware

In his first public work at VMware, our Chief Performance Architect, Richard McDougall, showed us how good he is at evangelizing technology to large audiences.  His article has received 14,300 hits in 2009 despite its 2007 publication date.  Oracle databases are incredibly popular topics among VI administrators and Richard was the first to point out that VI3′s storage throughput ability was far beyond the needs of the average 4-way Oracle database:

Back in 2007 we were trying to convince everyone to virtualize their demanding Oracle databases.  We knew that VI3 could handle it and our early adopters knew, too.  But since vSphere launched, everyone has known.  If you think that vSphere cannot handle your most demanding Oracle databases, think again.

Number 1: Windows Vista Performance in VMware Workstation 6.0

I attribute the fact that fact that this aging article has received 15,000 hits in 2009 to the incredible size of consumer desktop products as opposed to enterprise software and hardware. But it is worth mentioning that this article was written by Zhelong Pan, the developer responsible for esxtop, who has also written the most popular article on the VMware performance communities. That guy has the magic touch!

Earlier this month we finally published one of my favorite papers from ongoing vSphere launch activities. This paper on ESX memory management, written by Fei Guo of performance engineering, has three graphs that are absolute gems. They show balloon driver memory savings next to throughput numbers for three common benchmarks. The conclusion is inescapable: the balloon driver reclaims memory from over-provisioned VMs with virtually no impact to performance. This is true on every workload save one: Java.

Example 1: Kernel Compile

Linux kernel compilation models a common developer environment. This process is CPU- and IO-intensive but uses very little memory.

Results of two experiments are shown on this graph: in one configuration memory is reclaimed only through ballooning and in the other memory is reclaimed only through host swapping. The bars show the amount of memory reclaimed by ESX and the line shows the workload performance. The steadily falling green line reveals a predictable deterioration of performance due to host swapping. The red line holds steady, demonstrating that as the balloon driver inflates, kernel compile performance is unchanged.

Kernel compilation performance remains high with ballooning because this workload needs very little memory and the guest OS can easily take unused pages from the application. Performance falls with swapping because ESX randomly selects virtual machine pages for swapping whether those pages are in use by the application or not. The guest OS is better at selecting pages for reclamation than ESX is.

Example 2: Oracle/Swingbench

Oracle’s database is best tested against Swingbench, the OLTP load generation tool provided by Oracle. Database workloads utilize all system resources but show a non-linear dependence on memory. Memory can be safely reclaimed from OSes running databases until the cache becomes smaller than needed by the workload. The following figure shows this.

As before, the virtual machine using only ballooning maintains higher performance under memory pressure than the virtual machine whose memory is being swapped away by the host. Performance remains high as the balloon driver inflates until it encroaches into the 2G SGA. Again, ESX’s host swapping randomly selects pages to send to disk which degrades performance at all swap amounts.

As with kernel compile, the balloon driver safely reclaims memory from over-provisioned VMs with little impact to application performance.

Example 3: Java/SPECjbb

Java provides a special challenge in virtual environments due to the JVM’s introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages as arbitrarily and inefficiently as ESX’s swap routine.

Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed inside the JVM and efforts by the host or guest to remove pages will both degrade Java applications’ performance. In these environments it is wise to manually set the JVM’s heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.

Conclusions and Scott’s Special Recommendation

Love your balloon driver. Your application owners are always asking for more memory than they need. With great comfort you can over-provision memory some and rely on ESX and the balloon driver to reclaim what is not in use. Without the balloon driver, ESX will be forced to use its last technology for managing memory over-commit: host swapping. And host swapping always hurts performance.

So here is my special recommendation: never, ever disable the balloon driver. This eliminates efficient ballooning as an option to ESX.  Should any of that virtual machine’s memory be needed, ESX can only swap it out. Where ballooning usually will not hurt performance, swapping always will. If you must protect an application from memory reclamation due to memory over-commitment, use reservations. They make admission control more effective, they self-document the needs of the VM, and they are easily configured.