As I continue working with VMware’s Professional Services Organization (PSO) as they develop a performance troubleshooting service, I see repeats of a theme I wrote about a couple of months ago: storage is causing most performance problems, not VMware. In fact, I have yet to see a correctly-configured vSphere deployment that is running any application with unsatisfactory performance with respect to native.
My most recent engagement started last week in a city near to Palo Alto. The customer had over 100 Java virtual machines on a two-node cluster of IBM x3850 M2s. CPU utilization was reasonable and memory was not over-committed. Everyone was convinced that the correction to disappointing performance lied somewhere in ESX or the JVM. But it took only a few minutes on the vSphere client to identify the storage problem: response time averaged over 50 milliseconds per operation and peaked over 500 milliseconds.
Identifying and Correcting Slow Storage
My article on VMware’s performance community offers a list of counters that the VI admin should inspect to identify slow storage. In short, you want to look for device latencies that are over 15 milliseconds. How much over this limit you are willing to tolerate is a function of your application and users’ expectations. But if you see esxtop’s DAVG at 25 milliseconds or higher, know that your phone is about to ring with an angry user on the other end.
There are a host of issues that can result in high device latencies from ESX’s perspective. You should consult your storage admin and check them all to be sure you are on the right path. But 99% of the time the cause is an insufficient number of spindles for the workload. So let me offer two ways to design your storage to avoid a bottleneck.
Preventing Storage-based Performance Problems: Bandwidth Planning
Ideally you should know the peak and average IOPS of each of your virtual machines. If you do not know this, figure it out. If you cannot figure it out, you will be forced to try my second corrective option (below). But there is nothing that warms my heart like a well-informed VI admin. So, let us take a hypothetical example of five virtual machines on a single VMFS volume:
| VM | Peak IOPS | Avg. IOPS |
|---|---|---|
| A | 200 | 50 |
| B | 100 | 50 |
| C | 400 | 100 |
| D | 200 | 100 |
| E | 300 | 50 |
I will point out a few important numbers from this list:
- The aggregate average demand of these virtual machines is 350 IOPS.
- The aggregate peak demand of these virtual machines is 1200 IOPS.
- The largest peak demand of any one virtual machine is 400 IOPS.
You will have to size your storage to peak, to average, or somewhere in between. If you size to the average, you are counting on the peaks occurring at different times. If you are wrong, when two workloads peak simultaneously, a bottleneck will form at the array. Also note that sizing to the average in this case (350 IOPS) is insufficient for VM C’s peak of 400 IOPS. You could size to the aggregate peak of 1200 IOPS but unless all of the virtual machines peaked at once the workloads would never consume the available bandwidth.
All you can do in this case is make a best guess and modify later, as needed. I often suggest that a good start is one third of the way from average to peak which equals 633 IOPS in this case. If we assume 150 IOPS per spindle, that means five spindles for this VMFS volume.
Preventing Storage-based Performance Problems: Flexible Architecture
If it is impossible to estimate your storage demands, consider tiered performance for your VMFS volume. If you have 30 disks at your disposal, an example would be three VMFS volumes backed by 18, 8, and 4 spindles. Assuming 150 IOPS per spindle these will deliver 2,700, 1,200, and 600 IOPS, respectively.
Start with your best guess for initial virtual machine placement. Monitor virtual machine storage demands over time and storage migrate VMDKs up or down tiers based on their demands or the response time of the volume.
Disclaimer
I am not a storage expert. These thoughts should provide a very basic idea on storage performance sizing in consolidated environments. There are numerous other factors not mentioned here that require consideration: capacity, caching, RAID configuration, and connectivity, to name a few. Please consult your local storage admin or your storage vendor to get this right.
The one major problem with IOP analysis with any tool (SWAT or perfmon disk transfers) is that your peaks are affected by cache hits which can mean you get a peak of 2500 IOPs on a RAID5 array capable of only 900 (just an example I had), but the average was only 85 over an 8 hour period – hard to judge what is a real physical limitation
Excellent post couldn’t agree with you more , being a Storage Admin and having to monitor and build RAID groups , for the “unknown” guest which may come along.
At the moment I am designing RAID groups for VDI , which is a really interesting.
Disclosure – I’m an EMC employee
Good article Scott – misconfigured storage is indeed often the problem. IMO, the issue is most people ask/think about storage in “GB”, when in most cases, “MBps” and “IOps” are generally the constraints and should be the design focus for datastores and virutal disks. Regardless of datastore type, thinking about bandwidth (pipe to storage) and throughput (IOps, and also a function of latency – bandwidth consumed is a function of IOps x IO size, and modulated by latency) is **VERY** important.
This isn’t hard, but when there is a “chinese wall” between the “cylinders of excellence” of the virtualization silo vs. app silo vs. storage silo, this can be tricky.
We are planning to add performance detail to the (free) EMC storage viewer vCenter plugin shortly to help (see into the perf stats from the array, and link it to the vSCSI and ESXtop results). In the meantime, I would strongly start when troubleshooting with:
1) quick glance at guest level IO counters. This is as simple as looking at top in linux, or perfmon in windows.
2) if it looks bad (latency >20ms is usually bad), take a look at the vSCSI and ESXtop results. If they are bad….
3) talk to your storage team, and if you are the storage admin, every array has tools to quickly analyze the storage load.
If you find that the storage iteself isn’t huffing and puffing, but the ESX server is, look at IO queuing (there’s a post I wrote on virtualgeek on this topic of “microbursting”), or if using NFS/iSCSI – look at bandwidth and multipathing.
[...] Vpivot.com, VMware.com, VMware.com, [...]
Hi Chad, when this new version of the EMC Storage Viewer will be out? Performance details will be available for ESX3.x/VC2.5 as well?
Thx,
Didier
[...] 30, 2009 by deinoscloud As Chad Sakac said in a response to a blog post at vpivot.com : “We are planning to add performance detail to the (free) EMC [...]
[...] some experts point out, ESX performance issues are often caused by mis-configurations with the storage arrays. There [...]