This morning one such partner, VMTurbo, gave me a demonstration of their offering in this field.  Their product provides an obvious improvement on vSphere’s performance visualization capabilities.  But given the state of VMware’s visualization capabilities virtually any graphical front-end provides an improvement.  But what really set off my imagination were two features I had not seen before:

• A third-party alternative to DRS.
• Cross-cluster resource optimization.

VMTurbo provides a variety of monitoring and analysis capabilities but I want to focus most on optimization, in particular load balancing.  But before describing what VMTurbo has done, I want to point out the economics of competing with VMware’s DRS.

VMware provides four vSphere editions for its customers.  The cheapest edition that offers DRS is Enterprise at a list price of $2,875USD per socket.  The cheapest edition with vMotion is Standard at $995USD per socket.  There are plenty of cool features that come with upgrading from Standard to Enterprise: DRS, VAAI, Fault Tolerance, Storage vMotion, vShield Zones, and others.  But certainly DRS is one of the most valuable of that list.

By leaving such a big price gap between the cheapest vMotion edition and the cheapest DRS edition, VMware has provided its partners an economic incentive to innovate and provide DRS value to customers at a discount.  VMTurbo may capitalize on this incentive and it would not surprise me if numerous other ISVs are already doing so or soon will.  Once a vendor has built a robust monitoring environment, it is only a clever algorithm away from implementing DRS.  And then a trivial API call away from extending DRS to DPM.

The VMTurbo guys explained that their algorithm uses more resources than just CPU and memory and could therefore be better than DRS.  But I know how much work has gone into VMware’s memory-and-CPU DRS that I will only believe VMTurbo’s claims when I see the data.

Another area in which VMTurbo is tinkering is with inter-cluster load balancing.  The demo I received this morning showed a pre-cursor step to datacenter-wide load balancing by modeling the merge of two DRS clusters.  As the discussion in my maximum cluster size entry showed, choosing and changing cluster sizes is not easy.  And fluidly moving virtual machines between different clusters is not often possible for a variety of reasons.  But modeling cluster merging is the first step in considering cross-cluster operations.  And I think that there is a huge opportunity in the industry for someone to innovate in datacenter-wide optimization.

I would be curious to see what other vendors are doing with DRS, DPM, or datacenter-wide load balancing.  Can anyone refer me to any ISVs that are trying to crack these difficult problems?

The first tool in the VI administrator’s toolbox should always be vCenter. Through the vSphere client you can use vCenter’s performance counters to confirm a problem with any of the four resources (storage, CPU, memory, network). vCenter’s 20 second sample window impedes its ability to eliminate a resource as a problem. This is because a three second spike in any resource will be smoothed and missed over the 20 second window. But when vCenter confirms a sustained resource bottleneck, it is sure to be the performance problem’s cause.

If vCenter fails to confirm an obvious performance problem, the administrator must next go to more precise, more time-intensive, and more knowledge-intensive tools such as esxtop and vscsiStats. esxtop takes more skill and time than vCenter but provides better resolution and more visibility into the system. vscsiStats is the most time-intensive tool and has limits with ESXi hosts but can uncover a world of detail invisible to esxtop and vCenter.

I estimate each tool’s chance of identifying a random performance problem as follows:

• vCenter: used in 90% of performance problems
• esxtop: used in 9% of problems
• vscsiStats: used 0.9% of the time

The remaining 0.1% of the time is when you engage your account team or your local VMware performance expert.

Even within each tool’s usage there is an hierarchy of investigation: storage, CPU, memory and network. My experience with troubleshooting has informed this decision.  Storage causes the most problems, then CPU, then memory, and lastly (and rarely) network. After each resource level is inspected in vCenter, a repeat of the inspection should occur on esxtop. Guest tools may be a third option for memory, CPU, and network but vscsiStats should always be consulted if the performance problem persists.

VMware’s growing array of performance management tools will change this flow somewhat. AppSpeed, for instance, adds the ability to make very educated guesses about resource bottlenecks based on inside information into the application execution.  Hyperic can provide in-guest process visibility and Ionix ADM will map application interdependenies to focus the investigation.  But, I will abstain from providing best practices on these tools until I have used them more. In all cases, however, the fundamental relationship of “easy first, precise later” remains.

VMware continues to work towards integrating all of these tools into a single view within the vSphere client. I expect that integration will improve the success rate of the performance layman in troubleshooting these problems. But I am sure that even into the distant future performance people will find their jobs secure.

Before I describe the test and its results, I want to share an important point on guest defragmentation.  Most of the computer literate are aware that file fragmentation–the separation of logically contiguous pieces of a file–can hurt storage performance.  But many may not realize that free space fragmentation is as big of an issue.  When free space is fragmented, writes take longer and files are re-fragmented rapidly.  PerfectDisk defragments files and free space and the results below benefit from both of these improvements.

The following steps were used to setup the test environment:

1. A  new virtual machine was constructed with Windows Server 2008 on a single 50 GB virtual disk.  29 GB (58%) of the disk was populated with he OS and miscellaneous user files.  21 GB of free space remained.
2. They ran a tool that simulated months of user activity by reordering blocks to fragment files and free space.
3. The fragmented virtual machine was cloned.
4. PerfectDisk was run on the second virtual machine to produce a defragmented virtual disk.

The Raxco team then compared the performance of the fragmented virtual machine with the defragmented one by measuring the installation time of Microsoft SQL Server 2008. This workload was chosen because it produces a bounded, write-intensive test that can easily be monitored with vscsiStats.  Only one virtual machine was running on the host.

Let’s take a look at a few key graphs.

The number of IOs during the application install, organized by size.

This first histogram shows IO counts by size.  You can see that IO counts for all but the largest bucket have decreased because Perfect disk is reordering files to minimize small IOs and maximize large IOs.  For the storage hardware this means greater efficiency in processing IOs.  But it also means two things to ESX:

• More host throughput as the fixed HBA queue is now holding larger commands.
• Fewer virtual storage stack traversals resulting is lower CPU utilization.

However, PerfectDisk not only consolidates small IOs into large IOs, it also makes files logically contiguous as they are seen by the NTFS file system. This means less work for disk controllers when mapping logical to physical clusters on disk.

Next we have the vscsiStats seek distance histogram which shows the shift from random to sequential IO.

The seek distance histogram shows the number of logical blocks between each successive IO.

The seek distance histogram shows a clear increase in the number of IOs that were exactly one block after the previous IO.  This pattern, demonstrated in the bucket “1″ increase, reflects the increased sequential nature of accesses to the defragmented virtual disk.  In this case the fragmented disk access was 15% sequential while the defragmented disk was 30% sequential.

Let us next look at latency.

This histogram counts IOs by latency.

The latency histogram shows a decrease in all IOs across the board and a near elimination of IOs that took more than 30,000 microseconds (30 ms).  Those very slow IOs accounted for 15% of all the commands in the fragmented case.  By eliminating the 15% slowest IOs, you can imagine that the total IO performance and application execution time would greatly improve.  That is exactly what happened, as the following data show:

Let me repeat one of those amazing data points: the average IO latency dropped from about 55 ms to less than 4 ms. While this is a phenomenal number, the increase depends on characteristics of the storage system.  Since these improvements are configuration dependent, your results may vary considerably.

As Raxco continues its investigation I remain cautiously optimistic of the value of guest defragmentation.  I think the exchange of small IO for large IO is indisputably a Good Thing.  However, virtual environments are very complex and I harbor some concerns about guest defragmentation in virtual environments that must be considered.  For instance:

• Defragmentation in your virtual machines backed by linked clones may explode those VMs’ VMDKs’ consumption of their VMFS volumes.
• The value of increased sequential access in a single virtual machine will decrease some in consolidated environments. This is because multiple virtual machines’ sequential access gets interleaved at the array, increasing the randomness of the IO from the array’s point of view.
• Guest block reordering may have negative consequences to your array, as Vaughn Stewart argued in a comment to my first entry on the subject.
• The value of file defragmentation may be limited when applications produce small random block access to data files, as some databases tend to do.

Raxco is continuing to investigate guest defragmentation to respond to some of these concerns and provide best practices for PerfectDisk’s usage.  I will update you as the research continues.

Test Details

ESX Server Configuration

• ESX Version: 3.5.0 Update 1
• Motherboard: Intel S5000PSL
• CPU Type: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
• Number of CPUs: 2
• Cores per CPU: 4
• Logical Processors: 8
• Memory: 4 GB

Storage Configuration

• RAID controller: Adaptec RAID 3805
• Number of Drives: 4
• Drive Type: WD1001FALS 1TB 7200 RPM 32MB Cache
• Total Capacity: 4.0 TB
• Number of LUNS: 2
• LUN 1 RAID level: 5
• LUN 1 Capacity: 2.00 TB
• LUN 1 Partitions: 1
• LUN 1 Name: IOTesting
• LUN 2 RAID level: 5
• LUN 2 Capacity: 744.75 GB
• LUN 2 Partitions: 1

Datastore Configuration

• Number of Datastores: 2
• Datastore 1 Name: IOTesting
• Number of VMs: 2
• Capacity: 2.00 TB
• Target LUN: LUN 1 (from Storage configuration)
• Datastore 2 Name: ISO
• Number of VMs: 0
• Capacity: 744.75 GB
• Target LUN: LUN 2 (from Storage configuration above)

VM Configuration

• Number of VMs: 2
• Operating System: Windows Server 2008 R2 (64-bit)
• Memory: 2GB
• Number of CPUs: 2
• SCSI Controller: LSI Logic (no SCSI bus sharing)
• Number of Disks: 1
• Size of Disk: 50 GB
• Provisioning Type: Thick
• Backing Datastore: IOTesting
• Virtual Memory: none (pagefile disabled)
• Network: disabled

1. Boot ESXi in tech support mode.  See KB article 1003677 for instructions.
2. Download the right version of vscsiStats using the table below.
3. Install the right vscsiStats binary using a secure copy tool such as WinSCP or scp.

I will provide vscsiStats binaries as requested.  If you do not see the version of vscsiStats that matches your ESXi build, please provide in this article’s comments your build number (not release number!) and I will add it version of vscsiStats to the repository.

Currently Posted vscsiStats Binaries

Again, comment if yours is not listed!

• Build 110271, ESX 3.5 U2 (Someone please confirm that this works.  This was a strange build.)
• Build 153875, ESX 3.5 U4
• Build 163429, ESX 3.5, patch 14
• Build 164009, ESX 4.0
• Build 171294, ESX 4.0 (same bits at 164009)
• Build 176894, ESX 3.5, patch 16
• Build 184236, ESX 3.5, patch 17
• Build 199239, ESX 3.5, patch 18
• Build 207095, ESX 3.5 U5
• Build 208167, ESX 4.0 U1
• Build 219382, ESX 4.0, patch 3
• Build 236512, ESX 4.0, patch 4
• Build 238493, ESX 3.5, patch 21
• Build 244038, ESX 4.0, patch 5
• Build 256928
• Build 261974, ESX 4.0, U2
• Build 398348, ESX 4.0, U3

Update, 15 May 2010: Change of SLA

Everyone may notice that I used to be able to be able to respond to requests for vscsiStats versions in 24 hours. Now that I no longer work for VMware and am living half a world away, I have to request the binaries myself from my contacts at VMware. So, it will take me a few days to honor requests.

Update, July 2010: ESXi in vSphere 4.1 Comes with vscsiStats

vscsiStats is now distributed with ESXi. There will be no vscsiStats for ESXi 4.1 and newer placed on this page. This page will remain up for older versions of ESXi, however.

The article describes a phenomena common to consolidated workloads called micro-bursting.  Micro-bursting occurs in such short periods as to go unnoticed in the sampling window of monitoring tools.  As Chad put it:

Remember that every metric has a timescale.   IOps is in seconds.   Disk service time is in ms (5-20ms for traditional disk, about 1ms for EFD).  If an I/O is served from cache, it’s in microseconds.   Switch latencies are in microseconds.    Here, the I/O periods were so short that they filled up the ESX LUN queues instantly, causing a “back-off” effect for the guest.   These were happily serviced by the SAN and the storage array, which had no idea anything bad was going on.

When these bursts happen queues overflow, messages backup, and service times briefly sky rocket.  These rapid overflows happen in a fraction of esxtop‘s multi-second window and vCenter‘s 20 second window.

So, what buffers are we talking about?  Take a look at Chad’s hand-drawn picture of the storage path, which is only slightly less complicated than the Republican view of Obamacare:

Chad Sakac's image showing the numerous locations of storage queues in all locations from the VM to the platter.

If you are at VI admin, you care about the LUN queue in ESX.  ESX creates one of these queues for each HBA+LUN pair.  So, multipathing to a LUN increases the effective LUN queue and using a single HBA to multiple LUNs will guarantee a queue to each LUN.  Instances of this queue will overflow if many VMs on a single server issue commands to a single LUN.  As Chad says:

In VMware land – this is usually the fact that the default LUN queue (and corresponding Disk.SchedNumReqOutstanding value) are 32 – which for most use cases is just fine, but when you have a datastore with many small VMs sitting on a single LUN, the possibility of microbursting patterns becomes more likely.

So, when will the queues overflow?  Not often:

In the example [Vaughn] used, [multi-pathing] would not help materially if there were more than 3 ESX hosts, as it would be a likely case of “underconfigured array” – not host-side queuing.

The message here is that there is only a small window of configurations will result in LUN queue overflow: many VMs on very few hosts talking to a common LUN.  This is a perfect use case for vscsiStats, which I have talked about in various forums now.  vscsiStats avoid sampling windows by recording precise information on every IO.  This means that microburst statistics will not be averaged–and lost–across a time period.

Consider the following data I pulled from a sample session on my office system:

This table shows the number of outstanding IOs as each new IO arrives in the VMkernel.  The first row means that during the collection period only two IOs arrived to a queue with one outstanding IO.  Row two says that two IOs entered when there was were two outstanding IOs.  The third row states that 50 IOs arrived while the queue had 3-4 IOs.  And so on.

This table represents a fairly healthy access pattern, showing that only 433 out of 357,402 IOs arrived while the queue had 33-64 outstanding IOs (shown on the last row).  With ESX’s default LUN queue depth at 32, vscsiStats shows that a very small number of IOs arrived to an overflowing queue.

In summary, some storage performance issues appear and disappear so rapidly as to not be visible with sampling based tools, even as fine-grained as esxtop.  As a VI admin you should consider this in your most challenging troubleshooting cases.  And remember to use vscsiStats if all else has failed.

A few days after the conference a coworker of mine shared the following quote with me, courtesy of an article by David on Internet Evolution:

The fact is, most data center managers wouldn’t trust VMware to manage their Tier 1 applications because if something goes wrong performance-wise, you still need to roll in the VMware PhDs to solve it.

Let me respond to a few of the suggestions from this quote.

“Customers Do Not Trust VMware for Tier-1 Apps”

The following chart presents data collected from 676 VMware customers in July and August of 2008.

Percentage of application instances virtualized by VMware customers.

This graph shows the large rates of virtualization of the most well-known enterprise applications.  By any definition of “Tier-1 Application”, at least one tier-1 application is mostly virtualized by this customer sample.  And the survey date bears repeating: summer, 2008.  Virtualization acceptance has greatly increased in the past 12 months.

Concerns about performance management aside, VMware customers are virtualizing their tier-1 apps today.  So let’s talk about the process of performance troubleshooting.

“A PhD From VMware Is Required to Fix Performance Problems”

I think that David must have inferred from my confident and detailed talk on a great number of performance-related technical topics that I am the cream of the crop of America’s educational system.  For the record, I went to a state school in Alabama and spent far more time drinking beer than going to class.  Nonetheless, I am sure what he meant to say was…

“A Highly-skilled Performance Expert Is Required to Fix Performance Problems”

VMware now boasts over 150,000 customers, and I only interact with a relative handful a year.  If I count the experts in our small performance community I can conclude that our performance experts touch a very small percentage of our customer base each year.  That means that the great majority of our customers are solving their performance problems without engaging us.

Customers are fixing their problems using a variety of tools that I continue to document:

• The vSphere client interface to vCenter is known to everyone and its counters operate with 20s granularity and are effective at fixing about 90% of most performance problems.
• esxtop, with its finer granularity and larger counter list, can be used to fix 95% of problems, most of which could have been fixed with vCenter statistics.
• vscsiStats is extraordinarily useful for a small percentage of problems, perhaps 10-20% of those I see.

We are currently working on collecting all of these views into the client and adding a framework, vProbes, that will enable unprecedented visibility into operating systems and applications.  But even as things stand today, we have provided documentation and tools that all of our customers can use to fix any problem.  There is always room for improvement, but no PhD is required.

Back in July I had the privilege of riding along with VMware’s Professional Services Organization as they piloted a possible performance offering. We are considering two possible services: one for performance troubleshooting and another for infrastructure optimization. During this trip we piloted the troubleshooting service, focusing on the customer’s disappointing experience with SQL Server’s performance on vSphere.

If you have read my blog entries (SQL Server Performance Problems Not Due to VMware) or heard me speak, you know that SQL performance is a major focus of my work. SQL Server is the most common source of performance discontent among our customers, yet 100% of the problems I have diagnosed were not due to vSphere. When this customer described the problem, I knew this SQL Server issue was stereotypical of my many engagements:

“We virtualized our environment nearly a year ago and and quickly determined that virtualization was not right for our SQL Servers. Performance dropped by 75% and we know this is VMware’s fault because we virtualized on much newer hardware on the exact same SAN. We have since moved the SQL instance back to native.”

Most professionals in the industry stop here, incorrectly files this problem as a deficiency of virtualization, and move on with their deployments. But I know that vSphere’s abilities with SQL Server are phenomenal, so I expect to make every user happy with their virtual SQL deployment. I start by challenging the assumptions and trust nothing that I have not seen for myself. Here are my first steps on the hunt for the source of the problem:

1. Instrument the SQL instance that has been moved back to native to profile its resource utilization. Do this by running Perfmon to collect stats on the database’s memory, CPU, and disk usage.
2. Audit the infrastructure and document the SAN configuration. Primarily I will need RAID group and LUN configuration and an itemized list of VMDKs on each VMFS volume.
3. Use esxtop and vscsiStats to measure resource utilization of important VMs under peak production load.

There are about a dozen other things that I could do here, but my experience in these issues is that I can find 90% of all performance problems with just these three steps. Let me start by showing you the two RAID groups that were most important to the environment. I have greatly simplified the process of estimating these groups’ performance, but the rough estimate will serve for this example:

We found two SQL instances in their environment that were generating significant IO: one that had been moved back to native and one that remained in a virtual machine. By using Perfmon for the native instance and vscsiStats the virtual one, we documented the following demands during a one-hour window:

In the customer’s first implementation of the virtual infrastructure, both SQL Servers, X and Y, were placed on RAID group A. But in the native configuration SQL Server X was placed on RAID group B. This meant that the storage bandwidth of the physical configuration was approximately 1850 IOPS. In the virtual configuration the two databases shared a single 800 IOPS RAID volume.

It does not take a rocket scientist to realize that users are going to complain when a critical SQL Server instances goes from 1050 IOPS to 400. And this was not news to the VI admin on-site, either. What we found as we investigated further was that virtual disks requested by the application owners were used in unexpected and undocumented ways and frequently demanded more throughput than originally estimated. In fact, through vscsiStats analysis, my contact and I were able to identify an “unused” VMDK with moderate sequential IO that we immediately recognized as log traffic. Inspection of the application’s configuration confirmed this.

Despite the explosion of VMware into the data center we remain the new kid on the block. As soon as performance suffers the first reaction is to blame the new kid. But next time you see a performance problem in your production environment, I urge you to look at the issue as a consolidation challenge, and not a virtualization problem. Follow the best practices you have been using for years and you can correct this problem without needing to call me and my colleagues to town.

Of course, if you want to fly us out for to help you correct a specific problem or optimize your design, I promise we will make it worth your while.