This morning one such partner, VMTurbo, gave me a demonstration of their offering in this field.  Their product provides an obvious improvement on vSphere’s performance visualization capabilities.  But given the state of VMware’s visualization capabilities virtually any graphical front-end provides an improvement.  But what really set off my imagination were two features I had not seen before:

• A third-party alternative to DRS.
• Cross-cluster resource optimization.

VMTurbo provides a variety of monitoring and analysis capabilities but I want to focus most on optimization, in particular load balancing.  But before describing what VMTurbo has done, I want to point out the economics of competing with VMware’s DRS.

VMware provides four vSphere editions for its customers.  The cheapest edition that offers DRS is Enterprise at a list price of $2,875USD per socket.  The cheapest edition with vMotion is Standard at $995USD per socket.  There are plenty of cool features that come with upgrading from Standard to Enterprise: DRS, VAAI, Fault Tolerance, Storage vMotion, vShield Zones, and others.  But certainly DRS is one of the most valuable of that list.

By leaving such a big price gap between the cheapest vMotion edition and the cheapest DRS edition, VMware has provided its partners an economic incentive to innovate and provide DRS value to customers at a discount.  VMTurbo may capitalize on this incentive and it would not surprise me if numerous other ISVs are already doing so or soon will.  Once a vendor has built a robust monitoring environment, it is only a clever algorithm away from implementing DRS.  And then a trivial API call away from extending DRS to DPM.

The VMTurbo guys explained that their algorithm uses more resources than just CPU and memory and could therefore be better than DRS.  But I know how much work has gone into VMware’s memory-and-CPU DRS that I will only believe VMTurbo’s claims when I see the data.

Another area in which VMTurbo is tinkering is with inter-cluster load balancing.  The demo I received this morning showed a pre-cursor step to datacenter-wide load balancing by modeling the merge of two DRS clusters.  As the discussion in my maximum cluster size entry showed, choosing and changing cluster sizes is not easy.  And fluidly moving virtual machines between different clusters is not often possible for a variety of reasons.  But modeling cluster merging is the first step in considering cross-cluster operations.  And I think that there is a huge opportunity in the industry for someone to innovate in datacenter-wide optimization.

I would be curious to see what other vendors are doing with DRS, DPM, or datacenter-wide load balancing.  Can anyone refer me to any ISVs that are trying to crack these difficult problems?

First, a little background on ESX’s storage queues.  There are two relevant queues in ESX.  First is the device queue, which has one instantiation at each HBA for each LUN.  Second is the kernel queue, which handles “overflowed” IOs that are waiting to be placed in a full device queue.

For Fibre Channel HBAs, the device queue’s default length is 32 commands.  It is much larger for iSCSI. No HBA, and thus no device queue, exists for NFS.  A 32 command queue is capable of opening 32 commands at a time.  Obviously, if you double this queue length then the queue will drive twice as many IOs to the volume.  For the rest of this article I will discuss queues in terms of the 32 element Fibre Channel queue.

Because one device queue is instantiated at each HBA for each LUN, a storage reconfiguration at an array can change the number of queues at an ESX host.  Increasing the number of queues increases the total number of IOs that the host can open against the array.  I demonstrated this in my VMworld presentation with the following figure.

Putting two VMs on two volumes results in up to 64 commands being opened from the pair of them at one time.

This figure shows the simple difference between two virtual machines sharing a single VMFS volume and two that each get their own.  In the first configuration, only 32 commands can be opened from the host and that single queue is shared between the virtual machines.  In the second configuration, the host can open up 64 total commands and each virtual machine can open up to 32.

Your first reaction to this might be, “Wow! I should put every VMDK on a VMFS volume of its own!  Then imagine the total throughput that the host could drive!!”  My first response to this is stop using so many exclamation points.  Nobody likes an overenthusiastic writer.  But second, you should consider that more is not always better.  In fact, I can think of several reasons why you should not reconfigure storage to multiply the number of queues:

1. Allowing a host to open many commands simultaneously may be good for the individual virtual machines but is likely to be dangerous for the shared infrastructure.  This could result in short but extremely intense microbursts of IO that could present challenges to your fabric or storage processors.
2. The device driver (and the HBA) can only open a fixed number of commands depending on the device’s implementation.  You have to use these sparingly.
3. The configuration that results in more queues necessarily requires more VMFS volumes which results in a greater administration cost.

In addition to reconfiguring storage to increase the number of device queues, you always have the option of increasing the length of ESX’s device queues.  This is documented on page 71 of the Fibre Channel SAN Configuration Guide.  But I will caution you from reconfiguring storage queues, too.  This requires manual changes at every host, produces longer queues that more quickly eat into the fixed number of commands each HBA can support, and increases the possible IO intensity every virtual machine on the host.

And if these detailed explanations are insufficient at explaining why storage queue manipulation is unproductive or even counterproductive towards your goal of optimizing your infrastructure, let me point out that VMware has years of experience at consolidating storage and they chose 32 commands per queue as the right number for most environments.  Trust their experience on this one.

Of course I would be remiss if I did not mention that there are rare times that a storage reconfiguration may help performance.  Redistributing virtual machines across different VMFS volumes or increasing queue depths can correct some issues.  And you can identify occasions where this change may help by a large kernel latency.

As I mentioned above, commands that are waiting for access to a full device queue reside in the kernel queue until a device queue slot becomes available.  On the whole, commands should only spend a fraction of a millisecond in the kernel queue on their way to the device queue.  A kernel queuing time of over one millisecond and certainly over two milliseconds suggests the virtual machines are not having their IO needs served fast enough.

You can see kernel queueing times in the kernel latency statistic reported in esxtop (counter: KAVG) and vCenter (counter: Kernel Latency).  When these latencies consistently average any whole number in milliseconds its time to investigate storage.  But know that slow storage can result in high kernel queuing times.  So, before you go manipulating queues, or reconfiguring your storage layout, make sure your storage is serving IOs in periods deemed acceptible by the storage teams (usually 5-10 ms).

This is kind of a long article by vPivot standards, I know.  But cut me some slack.  Chad Sakac bangs out footnotes and parenthetical digressions that are longer than this entry.  This content has already been covered in my VMworld presentations so if you have access to those recordings go listen to Kaushik and I present it there.  But for those of you that were unable to attend I wanted to present this important guidance for your consideration.

The first tool in the VI administrator’s toolbox should always be vCenter. Through the vSphere client you can use vCenter’s performance counters to confirm a problem with any of the four resources (storage, CPU, memory, network). vCenter’s 20 second sample window impedes its ability to eliminate a resource as a problem. This is because a three second spike in any resource will be smoothed and missed over the 20 second window. But when vCenter confirms a sustained resource bottleneck, it is sure to be the performance problem’s cause.

If vCenter fails to confirm an obvious performance problem, the administrator must next go to more precise, more time-intensive, and more knowledge-intensive tools such as esxtop and vscsiStats. esxtop takes more skill and time than vCenter but provides better resolution and more visibility into the system. vscsiStats is the most time-intensive tool and has limits with ESXi hosts but can uncover a world of detail invisible to esxtop and vCenter.

I estimate each tool’s chance of identifying a random performance problem as follows:

• vCenter: used in 90% of performance problems
• esxtop: used in 9% of problems
• vscsiStats: used 0.9% of the time

The remaining 0.1% of the time is when you engage your account team or your local VMware performance expert.

Even within each tool’s usage there is an hierarchy of investigation: storage, CPU, memory and network. My experience with troubleshooting has informed this decision.  Storage causes the most problems, then CPU, then memory, and lastly (and rarely) network. After each resource level is inspected in vCenter, a repeat of the inspection should occur on esxtop. Guest tools may be a third option for memory, CPU, and network but vscsiStats should always be consulted if the performance problem persists.

VMware’s growing array of performance management tools will change this flow somewhat. AppSpeed, for instance, adds the ability to make very educated guesses about resource bottlenecks based on inside information into the application execution.  Hyperic can provide in-guest process visibility and Ionix ADM will map application interdependenies to focus the investigation.  But, I will abstain from providing best practices on these tools until I have used them more. In all cases, however, the fundamental relationship of “easy first, precise later” remains.

VMware continues to work towards integrating all of these tools into a single view within the vSphere client. I expect that integration will improve the success rate of the performance layman in troubleshooting these problems. But I am sure that even into the distant future performance people will find their jobs secure.

ESX must expend CPU cycles to maintain running virtual CPUs whether they are being used by an application or not. This means that host efficiency drops as more vCPUs are put on the server. But applications that scale well with CPUs will deliver greater performance when their virtual machines have been given more CPUs. The administrator must therefore balance the desires of an individual application’s owner with the needs of the entire cluster’s of applications.

There are several resources that VI administrators can use to inform their decisions in virtual machine sizing. I have listed some of them below.

Bruce Herndon’s Cost-of-SMP Article

Last summer the VMmark team’s Bruce Herndon published an article on the cost of SMP. I summarized his findings in a vPivot article I wrote on the ESX 4 scheduler. There are two key messages that you can take away from these posts to inform your decisions on virtual machine sizing:

• Over-sized virtual machines only hurt system performance when the server’s CPUs are saturated.  When utilization is low, unneeded vCPUs only penalize the system’s CPU utilization, not the applications’ performance.
• Unneeded 2-way virtual machines are not very harmful to the environment.  But administrators should be very careful with 4-way virtual machines and larger.

Co-stop and Ready Time

Ready time indicates a vCPU waiting for an available core when it has work to perform. Co-scheduling stop time (or co-stop time) indicates a vCPU being paused by the scheduler to allow its sibling vCPUs to catch up. These two counters can help administrators recognize a certain kind of stress due to limited CPU resources.

Ready time is generally a sign of the unavailability of CPU. Correction usually requires the administrator reducing work on the host (migrating virtual machines, decreasing vCPU count, etc.) or increasing CPU capacity (more hosts or faster CPUs). Co-stop time is a sign that the scheduler is allowing vCPUs to develop skew while it runs portions of virtual machines on available cores. Considerable numbers for these counters are 10% ready time and 3% co-stop time. There is no guarantee that application performance is suffering if these thresholds are crossed, but a problem may be present.

The important thing about ready time and co-stop time is that they are signs that you are using all of the CPU you have available to you. This could be a Good Thing. But it could also be a surprise to you. When these counters get high it is a good time to start asking yourself if you capacity usage meets your expectations. If not, you should inspect your virtual machines to be sure that the applications are using the vCPUs you have given them. If your guest tools show poor in-guest utilization then decrease those VM sizes. That will free up resources in the cluster for more virtual machines.

Application Scalability Information

I wish we lived in a world where every ISV published data showing their applications’ abilities to scale with cores. Unfortunately for us, many software vendors have for years allowed their customers to assume that each doubling of cores would double the performance of the application. VMware has chosen to provide some scalability information so our customers know how well or how poorly applications scale. But every customer of a software company deserves to have the vendor provide guidance on sizing the server. And those vendors deserve the right to put these results out on their own products. Go talk to your ISV to get the information you need to size your virtual machines.

CPU Usage Calculations and CapacityIQ

I am belatedly updating this post with a fourth way of identifying oversized virtual machines: mathematical calculation or Capacity IQ.

When a virtual machine consistently uses only a fraction of its vCPU resources it is possible that the virtual machine can be downsized and still deliver the same application performance. The calculation to determine this is simple: multiply the vCPU count by utilization and round up. Set the virtual machine’s vCPU count to the result of that calculation.

If you own CapacityIQ it will make this calculation for you for every virtual machine in your data center. Here is an screenshot of its recommendations based on virtual machine CPU and memory utilization. Click for a clearer picture.

CapacityIQ monitors CPU and memory utilization to recommend VM downsizing.

esxtop shows three processor utilization rows. What do they mean?

This capture shows utilization of each physical and logical processor core in the system.  The first row, PCPU USED, provides the percent of each physical core used by the logical core, multiplied by turbo mode, a processor feature that temporarily increases the core’s internal clock frequency.  This means two threads running at full, turbo speed might produce a number like 55% for each entry. It also means that one thread can drive its logical CPU to 100% only if the logical core’s sibling is unused. The second row is a straightforward calculation of the utilization of each logical core and the third row similarly shows utilization of physical cores.

Where things get really confusing is when these results are combined into three system-wide, aggregate utilization numbers, as seen by esxtop’s batch printout.  The three utilization types above generated different utilization numbers.  Unfortunately esxtop’s batch mode labels these counters slightly differently.  But this table includes both names:

The “Single Core Example” column provides an example calculation based on threads running as fast as possible on a single Hyper-Threaded physical core.  There are some interesting observations on these calculations:

• Even a great number of threads running full bore on an HT system will not produce a PCPU USED(%) number much over 50%.
• Two running threads will produce a lower PCPU USED(%) than one running thread.  This is because each thread’s utilization is calculated against the physical core.  With two running, each is averaging 50% of the core.  But a single thread that is not sharing the core can drive this to 100%.  In both cases the actual number could be a little higher if turbo mode is on.
• You must have at least two threads–one on each logical core–to drive PCPU UTIL(%) to 100%.
• CORE UTIL(%) can be driven to 100% with only one thread per physical core.

Look for this content to be rolled into our esxtop documentation soon.

esxplot is best described in the manual pages that Geoff included in the released archive:

esxplot is a GUI application that lets you explore the data collected by esxtop in batch mode. The program takes a single command line argument which is the esxtop batch mode output file. You can also simply start esxplot without any arguments, and enter a dataset file via the File attribute of the menu bar. Esxplot loads the data in this file and presents the metrics as a hierarchical tree where the values are selectable in the left panel. In the right panel, a graph is plotted (value over time) of the selected metric, in this way, you can “browse” the contents of these somewhat unwieldy files.

Version 1.0 of esxplot features the following enhancements over the beta version:

• Ability to close one data set and re-open another
• Plot 16 graphs
• More natural plot colors
• Grayed-out tree control entries for metrics that are all zero
• “Sticky directories” when exporting a chart or CSV file, future export dialogs will open at the last directory used
• Various bug fixes

esxplot is currently being hosted on my public share on ftpsite.vmware.com.  But Geoff is diligently working through the process of making this tool available publicly on vmware.com.  I will update you when this happens.

As before, Geoff and I welcome your comments and suggestions as to how to improve this tool.

Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.

But a recent article by my engineering colleague Chethan Kumar shows an avenue that allows VI admins to aggressively over-commit memory and avoid the catastrophic performance penalty of swapping: use solid state disks to host ESX swap files.

The fundamental problem with host swapping comes from the high latency of traditional disks compared to memory.  Data can be retrieved from memory in nanoseconds but takes milliseconds to fetch from a hard drive.  That means a single 4K memory page takes 100,000 times longer to retrieve if the operating system swapped it out.

The value that solid state disks offer to this problem is exceptional latency, as compared to traditional drives.  The SSD that Chethan used showed microsecond latencies, about 1,000 times lower than physical disks.  This means that  time spent waiting for swap activity* has been decreased to 0.1% of the time spent swapping to physical disks.

The importance of fast swap files is that it enables administrators to more aggressively over-commit memory.  Today our admins rightfully fear the VMs’ aggregate active memory exceeding the available physical memory, which results in swapping.  Today SSD technology in shared storage such as EMC’s new CLARiiONs allows our admins to cleverly place swap files and drive up memory utilization to previously unheard of levels.  This may enable standard memory overcommitment of 200% or more, with extreme over-commit being much higher than this.

In future versions of ESX we want to automate the usage of SSDs to maximize the use of available memory.  But that’s a roadmap discussion that I will leave for another day.

(*) This swap wait time has conveniently been added to ESX 4′s version of esxtop under the counter %SWPWT.  See Interpreting esxtop Statistics for more information.

A colleague of mine at VMware, Geoff White, wrote a Python-based tool for esxtop batch file analysis. His tool, dubbed esxplot, provides a graphical interface to the data generated by esxtop’s batch mode. esxplot provides search capabilities to find relevant counters, can plot different counters on the same graph, and does not suffer from the size limitations of Excel and Perfmon.

Geoff has graciously provided unsupported access to this tool’s beta version (version removed — see update below) for my readers. Please test esxplot much as possible and comment here with your observations. We would like to know of any bugs you discover as well as ideas for improvement.

The esxplot package comes with Windows and Ubuntu binaries as well as the Python source. The binaries will run without need of additional libraries or tools.  But if you decide to use the Python script, as I have done on MacOS, you will need to verify the following:

1. Ensure that you have Python 2.6 or greater, but not Python 3.x.
2. esxplot uses wxPython, which you can download and install from here.
3. esxplot also requires NumPy, available here.

A manual is included in the package but I think you should find the interface simple and intuitive.  Here are a few tips to get you going:

• Load an esxtop batch file by specifying it on the command line or through File->Import->Dataset.
• Using the counter navigator in the lower left corner, navigate to the counter of interest.  For instance, to view percent ready you would first expand the host  (use the triangle to the left of the host name), then expand “Group Cpu (GID:VM)”, then select “% Ready”.  The graph will be displayed on the right.
• You can quickly find counters by entering a query search in the middle of the left panel.  This creates a new entry in the counter navigator that contains only counters whose name matches the expression you entered.  For instance, entering “ready” will create a counter query that contains all ready times in the batch file.

Please let us know what you think of this wonderful tool.

Update: 1/13/09

esxplot version 1.0 was released on 1/13/09. See the updated blog article for information on that release and continued discussion on this tool.

The article describes a phenomena common to consolidated workloads called micro-bursting.  Micro-bursting occurs in such short periods as to go unnoticed in the sampling window of monitoring tools.  As Chad put it:

Remember that every metric has a timescale.   IOps is in seconds.   Disk service time is in ms (5-20ms for traditional disk, about 1ms for EFD).  If an I/O is served from cache, it’s in microseconds.   Switch latencies are in microseconds.    Here, the I/O periods were so short that they filled up the ESX LUN queues instantly, causing a “back-off” effect for the guest.   These were happily serviced by the SAN and the storage array, which had no idea anything bad was going on.

When these bursts happen queues overflow, messages backup, and service times briefly sky rocket.  These rapid overflows happen in a fraction of esxtop‘s multi-second window and vCenter‘s 20 second window.

So, what buffers are we talking about?  Take a look at Chad’s hand-drawn picture of the storage path, which is only slightly less complicated than the Republican view of Obamacare:

Chad Sakac's image showing the numerous locations of storage queues in all locations from the VM to the platter.

If you are at VI admin, you care about the LUN queue in ESX.  ESX creates one of these queues for each HBA+LUN pair.  So, multipathing to a LUN increases the effective LUN queue and using a single HBA to multiple LUNs will guarantee a queue to each LUN.  Instances of this queue will overflow if many VMs on a single server issue commands to a single LUN.  As Chad says:

In VMware land – this is usually the fact that the default LUN queue (and corresponding Disk.SchedNumReqOutstanding value) are 32 – which for most use cases is just fine, but when you have a datastore with many small VMs sitting on a single LUN, the possibility of microbursting patterns becomes more likely.

So, when will the queues overflow?  Not often:

In the example [Vaughn] used, [multi-pathing] would not help materially if there were more than 3 ESX hosts, as it would be a likely case of “underconfigured array” – not host-side queuing.

The message here is that there is only a small window of configurations will result in LUN queue overflow: many VMs on very few hosts talking to a common LUN.  This is a perfect use case for vscsiStats, which I have talked about in various forums now.  vscsiStats avoid sampling windows by recording precise information on every IO.  This means that microburst statistics will not be averaged–and lost–across a time period.

Consider the following data I pulled from a sample session on my office system:

This table shows the number of outstanding IOs as each new IO arrives in the VMkernel.  The first row means that during the collection period only two IOs arrived to a queue with one outstanding IO.  Row two says that two IOs entered when there was were two outstanding IOs.  The third row states that 50 IOs arrived while the queue had 3-4 IOs.  And so on.

This table represents a fairly healthy access pattern, showing that only 433 out of 357,402 IOs arrived while the queue had 33-64 outstanding IOs (shown on the last row).  With ESX’s default LUN queue depth at 32, vscsiStats shows that a very small number of IOs arrived to an overflowing queue.

In summary, some storage performance issues appear and disappear so rapidly as to not be visible with sampling based tools, even as fine-grained as esxtop.  As a VI admin you should consider this in your most challenging troubleshooting cases.  And remember to use vscsiStats if all else has failed.