First, a little background on ESX’s storage queues.  There are two relevant queues in ESX.  First is the device queue, which has one instantiation at each HBA for each LUN.  Second is the kernel queue, which handles “overflowed” IOs that are waiting to be placed in a full device queue.

For Fibre Channel HBAs, the device queue’s default length is 32 commands.  It is much larger for iSCSI. No HBA, and thus no device queue, exists for NFS.  A 32 command queue is capable of opening 32 commands at a time.  Obviously, if you double this queue length then the queue will drive twice as many IOs to the volume.  For the rest of this article I will discuss queues in terms of the 32 element Fibre Channel queue.

Because one device queue is instantiated at each HBA for each LUN, a storage reconfiguration at an array can change the number of queues at an ESX host.  Increasing the number of queues increases the total number of IOs that the host can open against the array.  I demonstrated this in my VMworld presentation with the following figure.

Putting two VMs on two volumes results in up to 64 commands being opened from the pair of them at one time.

This figure shows the simple difference between two virtual machines sharing a single VMFS volume and two that each get their own.  In the first configuration, only 32 commands can be opened from the host and that single queue is shared between the virtual machines.  In the second configuration, the host can open up 64 total commands and each virtual machine can open up to 32.

Your first reaction to this might be, “Wow! I should put every VMDK on a VMFS volume of its own!  Then imagine the total throughput that the host could drive!!”  My first response to this is stop using so many exclamation points.  Nobody likes an overenthusiastic writer.  But second, you should consider that more is not always better.  In fact, I can think of several reasons why you should not reconfigure storage to multiply the number of queues:

1. Allowing a host to open many commands simultaneously may be good for the individual virtual machines but is likely to be dangerous for the shared infrastructure.  This could result in short but extremely intense microbursts of IO that could present challenges to your fabric or storage processors.
2. The device driver (and the HBA) can only open a fixed number of commands depending on the device’s implementation.  You have to use these sparingly.
3. The configuration that results in more queues necessarily requires more VMFS volumes which results in a greater administration cost.

In addition to reconfiguring storage to increase the number of device queues, you always have the option of increasing the length of ESX’s device queues.  This is documented on page 71 of the Fibre Channel SAN Configuration Guide.  But I will caution you from reconfiguring storage queues, too.  This requires manual changes at every host, produces longer queues that more quickly eat into the fixed number of commands each HBA can support, and increases the possible IO intensity every virtual machine on the host.

And if these detailed explanations are insufficient at explaining why storage queue manipulation is unproductive or even counterproductive towards your goal of optimizing your infrastructure, let me point out that VMware has years of experience at consolidating storage and they chose 32 commands per queue as the right number for most environments.  Trust their experience on this one.

Of course I would be remiss if I did not mention that there are rare times that a storage reconfiguration may help performance.  Redistributing virtual machines across different VMFS volumes or increasing queue depths can correct some issues.  And you can identify occasions where this change may help by a large kernel latency.

As I mentioned above, commands that are waiting for access to a full device queue reside in the kernel queue until a device queue slot becomes available.  On the whole, commands should only spend a fraction of a millisecond in the kernel queue on their way to the device queue.  A kernel queuing time of over one millisecond and certainly over two milliseconds suggests the virtual machines are not having their IO needs served fast enough.

You can see kernel queueing times in the kernel latency statistic reported in esxtop (counter: KAVG) and vCenter (counter: Kernel Latency).  When these latencies consistently average any whole number in milliseconds its time to investigate storage.  But know that slow storage can result in high kernel queuing times.  So, before you go manipulating queues, or reconfiguring your storage layout, make sure your storage is serving IOs in periods deemed acceptible by the storage teams (usually 5-10 ms).

This is kind of a long article by vPivot standards, I know.  But cut me some slack.  Chad Sakac bangs out footnotes and parenthetical digressions that are longer than this entry.  This content has already been covered in my VMworld presentations so if you have access to those recordings go listen to Kaushik and I present it there.  But for those of you that were unable to attend I wanted to present this important guidance for your consideration.

The claim that Paul Maritz and Steve Herrod made on stage (twice) is that virtual machines now host more than 50% of the world’s x86 applications. That means that more operating systems are now communicating with hypervisors than physical hardware. The claim is backed by data from IDC, which I recently saw summarized by a VMware presenter at an event in the Philippines:

Sometime in 2009, the number of x86 virtual machines passed the number of x86 servers not running a hypervisor.

Paul hinted at the incredible implications of this change. It is no longer the traditional operating systems that are driving the future of hardware development. We should expect even more virtualization optimizations and improvements from every part of x86 servers.

VMware is obviously claiming the lion’s share of the virtualization market. While I do not have exact numbers, I take them to own 95% of virtualized x86 servers. The import of this is that VMware may now have more control over the future of hardware development than Microsoft.

For EA7726 – Virtual Machines Outperforming Physical Machines – Crossing the Performance Barrier, VMware’s Dave Korsunsky and I provide a survey of the performance history of VMware and the applications that have defied expectations.  This session highlights the characteristics of applications that inhibit scalability and the improvements in hardware and software that have allowed the circumvention of scalability limits.  We scored 4.46 out of 5 (conference average 3.93) so this was a well-received presentation.

In TA7171 – Performance Best Practices for vSphere VMware’s Kaushik Banerjee and I reprise this perennial favorite at VMworld.  I presented this topic at both previous VMworld Europe shows and Kaushik’s broader performance engineering team presented it in the US shows.  We scored a 4.47 out of 5 (conference average 3.93) for which I entirely credit VMware engineering’s fantastic research.  Unfortunately the projector was not online for the first 20 minute so we filled time with an early, extended Q&A.  I am sure the same problem will not be repeated in Copenhagen.

The 1-on-1 events allow you to schedule a 15 minute meeting with me to discuss anything you want.  I have only been scheduled to two 1-on-1 hourlong sessions which means no more than eight people will be able to get a slot.

The Group Discussion was this year’s best addition to the show.  In this forum the organizers have provided an interactive survey that drives group discussion and questions.  I have preloaded the survey with some of the hottest topics in VMware performance but you are encouraged to ask any question you want to hear my thoughts as well as those of your industry peers.

See you at VMworld Europe 2010!

A perfunctory bit of guidance I include in nearly all of my performance talks (such as the enthralling, entertaining, and cancer-curing* presentations from VMworld 2010 that I will repeat in Copenhagen from 12-14 October) is “follow your application best practices”.  Audiences usually nod and immediately forget because this recommendation we all know to be correct yet somehow ignore.  In that way it is like, “stay away from fatty foods”, “do not drink wine with pain killers”, or “pay attention during the flight attendants’ presentation”.

Part of the reason why people forget this nugget is because advice is general, and not crystalized in a technological explanation that embeds deep in the minds of the audience.  In this case the application best practice that should be followed is to separate data from logs, putting the data on something good for random read performance (like RAID 5) and the logs on something good for sequential write performance (RAID 10).  Obviously I want everyone to consolidate their storage to VMFS and enjoy the technology, but if you are putting VMDKs that contain each of these files on the same volume, you are ignoring application best practices.

In this case I recommend building two VMFS volumes.  One backed by RAID 5 and the other by RAID 10.  Put the data on RAID 5, the logs on RAID 10.  While you will change the access profile at the array by putting multiple log files on the same RAID 10 backed LUN, the resultant IO will be much more sequential write than had you mixed data file reads among them.  So, consolidate multiple data files onto the same RAID 5 LUN and consolidate multiple log files on the same RAID 10 LUN.

Furthermore, if you are using solid state auto-tiering to manage your volumes, you do not need to protect your database log file with this technology.  What I am talking about here is EMC’s Fully Automated Storage Tiering (FAST), which is the most popular thing EMC has created since I have been paying attention.  Despite what some people will tell you, solid state disks are the cheapest way to serve huge amounts of random reads.  But their benefits diminish when the profile is sequential write when they become unattractive from a cost perspective.

EMC’s FAST works by creating a volume that is like a vertical stripe of multiple RAID groups.  LUNs, which become VMFS volumes, are then placed in that FAST volume.  Since FAST is a great technology for solid state disks, RAID 5 is the most cost efficient configuration for database data, and solid state is wasted on sequential IO such as redo logs, my best practice for virtual storage configuration for databases workloads when FAST may be present can be boiled down to the following rules:

• Always create RAID 5 volumes for your read-intensive database data.
• Always create RAID 10 volumes for your database logs.  If you have write-intensive data, you may consider putting them here, too.
• If you have FAST, use it to stripe across multiple RAID 5 volumes of different disk types and put your random, read-intensive data on VMFS on this volume.

The last bullet is clearly the most important here. I really love FAST, and it seems that EMC’s customers are crazy for it.  But its not the technology you need for sequential write workloads like redo logs.  Separate those data onto their own “normal” (not FAST-backed) VMFS volumes that use no SSDs.  Then you will have the best of all worlds: optimally deployed disk technologies, application best practice compliance, and righteous virtualized database consolidation.

(*) The claims made by the author of this blog do not reflect the views of his employer, the conference organizers, the government of the Kingdom of Denmark, or reality, for that matter.

Last year was my first year captaining the performance troubleshooting lab.  I designed the content around the performance problems I see most frequently in customer deployments. On the philosophy that every attendee should get their money’s worth in the two hour slot, I packed in more experiments than anyone could finish in the allotted time.  That worked great for experienced VI admins, but some of the more novice administrators walked out with their heads spinning.

I do want to continue challenging even the strongest admins that come to the lab, but I must provide enough basics to make the lab enjoyable for everyone.  With that requirement in your mind, maybe you can offer ideas as to how the lab should change and what content you would like to see.  Here are a few primer questions:

• What comments do you have from last year’s lab?  Did I succeed in my struggle to make the lab interesting to VCPs and beginners alike?  If not, which side of that spectrum did I favor?
• What content do you want to see added?  A few ideas:
  • Swap management: managing swap through swap file placement, use of SSDs, and memory compression*
  • Storage IO Control*: using shares and limits to maintain performance of critical VMs in the presence of storage bottlenecks
  • NFS performance diagnosis*
  • Managing storage alignment
  • Guest defgragmentation impact
  • Anything else you are itching for?
• What content should we should repeat with updated examples?
• What do you want us to avoid repeating?  Have you learned all you need to know about vscsiStats, for instance?
• What would you think if we made this lab self-paced but promised performance experts in the room and on-call for the entire day?

(*) Coming soon to a hypervisor near you.

Put your comments here and they may influence the content of this lab.