With hardest work of making vSphere 5 generally available behind him, one of VMware’s engineers, Balaji Parimi, recently sent me scripts he wrote to create SIOC alarms.  These alarms can be used to tell administrators that SIOC is throttling some virtual machines to save high priority applications from ailing datastores.

Balaji’s script creates vCenter alarms that trigger when:

1. SIOC configuration is changed (enable / disable, congestionThreshold) on any datastore. The user does not need to do anything if a new datastore is added. The current vCenter alarm infrastructure does not allow to create alarms based on specific properties (e.g. iorm configuration enabled / disabled or congestion threshold etc.). So, I created one alarm that gets triggered when DatastoreIORMReconfiguredEvent is generated. This event gets generated whenever the StorageIORMConfiguration on any datastore in the vCenter inventory is changed.
2. SIOC Normalized Datastore Latency exceeds the congestionThreshold for a given datastore on a host. This alarm is a bit tricky as it depends on the performance metric. The performance metric based alarms are per instance only. So, this creates one alarm object per datastore in the vCenter. The script allows you to create for all datastores in the vCenter inventory in one shot. If more datastores are added later, the user needs to create an alarm for each of those datastores. If the user changes the congestionThreshold for a datastore, the user needs to delete the old alarm and re-create the alarm on that datastore for the new congestionThreshold value to be recognized.

The second alarm will tell you that SIOC is taking action, slowing some virtual machines to improve the performance of others.  This alarm only works on the datastores in existence at the time of its creation.  Thus the first alarm, which will tell the administrator that SIOC or the vSphere storage configuration has changed.  When the first alarm triggers it is time to re-register the second one.

Balaji included in his email to me the following instructions on setting the alarms:

Here are the two commands:

./siocalarms.sh 10.132.98.205 Administrator password all

This covers all the Datastores in the vCenter. This creates the SIOC configuration changed Alarm as well.

I added a new Datastore with the name FCLun0. And ran this command to cover the new Datastore.

./siocalarms.sh 10.132.98.205 Administrator password FCLun0

You can use the GUI to delete any of these Alarms. You can use the GUI to set up a specific action (like sending an email) for any of these Alarms.

I am attaching a zip file containing a README, shell and a batch script. All you need is JRE 1.6 or higher to run this.

Please give it a try and let me know your comments.

You may notice that my corporate Exchange server’s malware detection software removed the batch file Balaji references above.  But the batch file only contained a single line:

java -jar SIOCAlarms.jar <VCServer> <Username> <Password> <datastoreName>

As the instructions above state, the datastoreName “all” will add the alarms for all datastores.

Last last week Beth Pariseau at SearchServerVirtualization.com wrote a piece on vShield that quoted my blog. Beth kindly asked me for comment before her article went live but we were unable to connect. After reading her article, I wanted to add my thoughts to the discussion. I mailed these to her and want to share them with you here.

First, one of her claims is incorrect. Beth wrote, and cited an anonymous systems integrator in support, that “If vShield Manager is down, everything on its network is down.” The helpful technologists at VMware confirmed my understanding that the vShield architecture will not allow this. A vShield Manager failure will not affect network traffic.

vShield manager is a central point of management, but not a nexus through which traffic flows. The vShield architecture uses traffic-controlling security appliances on each ESX host managed by vShield. In the event of a vShield manager failure, the vShield security appliances continue with policies they previously received. A vShield Manager failure temporarily makes it difficult to change security policies but in no way affects connectivity or existing policies.

Furthermore, vShield Manager is protected by HA’s VM-aware monitoring. If the appliance’s guest operating system hangs, vpxa will restart the vShield Manager appliance. If the host running vShield manager fails, then HA will restart the appliance on another host in the cluster. This means that in the event of a vShield Manager failure, not only will the network continue to work with its existing security policies, but the network will be unmanageable for only the short duration measured by the appliance’s boot time.

This brings us to the issue I blogged about: installing vShield App on an ESX host running a vCenter virtual machine can disconnect the vCenter’s virtual NIC. VMware’s straightforward guidance for its management products is to run them in a separate management cluster. Following this recommendation vShield will only secure non-management virtual machines and this vCenter VM network connectivity issue cannot occur. While my support of this best practice is lukewarm, I recognize that some of VMware’s customers do follow it. And I am told that VMware is working on ensuring that all management products can be run as virtual machines on any host.

At the risk of belaboring the central point of this post, I will again say that no single appliance failure can bring down the network. I have already named vShield Manager in this regard and I reiterate with vCenter.

Beth solicited my comments and I explained my understanding of the article’s mistake. I also recommended she engage our friends at VMware, who I know from experience are always thrilled to share in limitless detail the workings of their fine products. I trust that Beth and VMware will get the right word out to customers. But as a constant commentator on VMware’s continued development on vShield, I wanted to append my thoughts to the ongoing vShield discussion.

VMware’s continued streak of incredible innovation of a constant source of marvel. When I expose warts on their products it is to help their faithful enthusiasts get the most out of VMware’s offerings. I believe Beth’s article was along that same vein: raise awareness and inspire continued improvement. Anyone that reads her article or mine should consider these tiny warts in the ongoing epic of datacenter rebirth.

Our fantastic APJ vSpecialist organization has been spending with vShield. Many of us believe vShield to be one of the most exciting editions to the VMware portfolio in a long time. It is simple, elegant, and very powerful. But there is a very big danger with vShield: improper use can disable the network connection of vCenter virtual machines. Fixing this problem is not intuitive.

We first observed this issue in our quarterly hands-on workshop in Tokyo with multiple people toying with the vShield products (Edge, App, EndPoint) on a single cluster. Some sequence of operations disconnected the vCenter’s vNIC. We opened the VM’s settings, clicked the vNIC’s “Connected” checkbox, and tried to go back to work. But the vNIC remained unconnected. No matter how many times we repeated this process, the vCenter VM remained offline. With vCenter offline, the infrastructure was unmanageable.

The general problem is that installing vShield App or Zones on an ESX host running vCenter introduces a circular dependency that can make installation, operation, or uninstallation fail. A warning against doing this is documented in a small cautionary note on the bottom of page 13 of the current vShield Administration guide. Unfortunately, it is an older version of this administration guide (which lacks this critical warning!!!) that appears first in response to the obvious Google search.

This same problem is described as an uninstallation issue in VMware KB article (1028151). That KB also includes suggestions on resolution.

While vShield App is not fundamentally incompatible with management of a vCenter VM, this problem raises an interesting point. In even moderate environments it makes sense to create a separate management cluster for VMware’s management products. This includes vCenter, vCloud Director, CapacityIQ, the vShield Manager, and a large number of helper VMs for other products. The key here is that these clusters are protected by HA and vCenter Server Heartbeat, but they are not managed by the tools in the cluster.

Tools like AppSpeed, CapacityIQ, vShield and others should be deployed to measure and manage the production and test clusters, but not the management clusters. This prevents circular dependencies that could result in difficult-to-correct network problems or inefficiencies like AppSpeed monitoring management traffic. This is obviously a very high level description of something that needs a reference architecture. More to come on that next week.

For those of you that share our interest in vShield, I want to direct you to the blog of a colleague of mine in Australia. Roman Tarnavski, a Sydney-based vSpecialist, has been digging into vShield and publishing some tutorials and hacking guides. Roman and I met with VMware engineering and we think that Roman will likely soon be showing you some amazing possibilities when hacking to improve vShield Edge.

Roman has already written a couple articles on the vShield and networking:

• Three types of network configurations for vCloud Director virtual machines.
• Enabling SSH in the vShield Edge appliance.

Check back here and read Roman’s blog regularly if you want to hear more about vShield.  Roman will have some incredible information on the vShield Edge internals that all of you hackers out there will not want to miss.

Recently David sat down to share his tips on setting up alarms in vCenter. Most of VMware’s customers understand the power of custom alarming but few harness its value. Using an example of storage path failures, David created a video that walks its audience through the process of configuring a custom alarm using a tailor made executable that generates useful logging message in the vCenter OS’s event log.

Here is David’s demo on YouTube:

(A high resolution version of the video is available here.)

David was kind enough to provide me a write-up of his work with details on the why and how of this process. From here on, this entry will be in his words.

vCenter Based Storage Port Monitoring

The above video demonstrates the creation of custom alarms in vCenter to monitor host level storage port failures.  To add a little bit more functionality to the alarms I wrote a little console application. When executed as a custom action assigned to an alarm, this application writes alarm details to the vCenter OS’s application event log. By adding entries to the local event log, third party monitoring solutions such as Microsoft’s System Center Operations Manager can be utilised to effectively monitor a VMware virtual environment. This is done by monitoring for specific events in the event log.  This is core function of most windows monitoring solutions.

When a command is executed as a custom action, it is run in a Windows shell.  That shell has several environment variables which contain details related to the triggered alarm. These details can then be used to provide a rich level of information to assist in problem identification and resolution. The table below shows the environment variables I have utilised to form the description field of the events raised.

In addition to the custom description, I have added additional flexibility to further customize the event.  This includes:

• Event Source – A custom source can be assigned by the executable to events it raises.  This will help easily identifying events. The source name is specified within the applications configuration file ‘vCenter-AlarmLog.exe.config’, which comes with both the binary and source distributions (below).
• Event ID – Each alarm condition can be assigned a unique ID to help direct of problem tickets resulting from the event. The unique ID is provided via the command line.
• Event Level – provides the ability to class an event as an Error, Warning or Information. The event level is specified from the command line.

These can assist in the filtering of any events raised for problem ticket routing and classification.

An example of an event created by the tool is shown below. Note the source, ID and level details in addition to the general description. The event entry below was created when a storage port was disabled resulting in a lost storage port.

Each event description is formatted identically with the field identifier and value (if present) separated by a TAB to help with parsing the details.

Command Line Arguments

Both Event ID and message type can utilise application defaults or can be specified through the command line. By default the application will raise an ‘Error’ entry and event id of ‘9999’, either of which can be changed as shown below:

Application Requirements

Custom Event Source

The custom event source is defined in the application configuration file ‘vCenter-AlarmLog.exe.config’.  See the ‘appSettings’ entry ‘source’ in this file.  The file is included in both packages but if it does not exist it will be created upon first execution of the application.  For this to work properly, the application needs to be executed once with ‘Administrative’ level access. Failure to do this will prevent the application from writing any events.

Runtime Environment

Requires Microsoft.Net 3.0. This is also a pre-requisite for vCenter, which should already exist on the vCenter server.

Source Code

Not a requirement but available should you wish to tinker with the script. The tool was written with Visual Studio 2010 in C# and its source is available as a download here.

Executable

The executable is available here.

This morning one such partner, VMTurbo, gave me a demonstration of their offering in this field.  Their product provides an obvious improvement on vSphere’s performance visualization capabilities.  But given the state of VMware’s visualization capabilities virtually any graphical front-end provides an improvement.  But what really set off my imagination were two features I had not seen before:

• A third-party alternative to DRS.
• Cross-cluster resource optimization.

VMTurbo provides a variety of monitoring and analysis capabilities but I want to focus most on optimization, in particular load balancing.  But before describing what VMTurbo has done, I want to point out the economics of competing with VMware’s DRS.

VMware provides four vSphere editions for its customers.  The cheapest edition that offers DRS is Enterprise at a list price of $2,875USD per socket.  The cheapest edition with vMotion is Standard at $995USD per socket.  There are plenty of cool features that come with upgrading from Standard to Enterprise: DRS, VAAI, Fault Tolerance, Storage vMotion, vShield Zones, and others.  But certainly DRS is one of the most valuable of that list.

By leaving such a big price gap between the cheapest vMotion edition and the cheapest DRS edition, VMware has provided its partners an economic incentive to innovate and provide DRS value to customers at a discount.  VMTurbo may capitalize on this incentive and it would not surprise me if numerous other ISVs are already doing so or soon will.  Once a vendor has built a robust monitoring environment, it is only a clever algorithm away from implementing DRS.  And then a trivial API call away from extending DRS to DPM.

The VMTurbo guys explained that their algorithm uses more resources than just CPU and memory and could therefore be better than DRS.  But I know how much work has gone into VMware’s memory-and-CPU DRS that I will only believe VMTurbo’s claims when I see the data.

Another area in which VMTurbo is tinkering is with inter-cluster load balancing.  The demo I received this morning showed a pre-cursor step to datacenter-wide load balancing by modeling the merge of two DRS clusters.  As the discussion in my maximum cluster size entry showed, choosing and changing cluster sizes is not easy.  And fluidly moving virtual machines between different clusters is not often possible for a variety of reasons.  But modeling cluster merging is the first step in considering cross-cluster operations.  And I think that there is a huge opportunity in the industry for someone to innovate in datacenter-wide optimization.

I would be curious to see what other vendors are doing with DRS, DPM, or datacenter-wide load balancing.  Can anyone refer me to any ISVs that are trying to crack these difficult problems?

I started thinking about this topic at Sydney’s vForum a month ago.  VMware’s Dan Anderson suggested that designs that implemented maximum cluster sizes (32 hosts per cluster) were the result of misguided reasoning.  Dan insisted that clusters need never be larger than eight hosts per cluster.  And on this subject we bantered for a few minutes.  Dan convinced me that there are few compelling reasons to implement large clusters.  And we could think of many reasons to avoid them.  I do not think it easy to assign one number as the “right” cluster size.  But there are many principles that suggest small to medium sized clusters being choices.

First, the argument for the largest clusters: DRS efficiency.  This was my primary claim in favor of 32-host clusters.  My reasoning is simple: with more hosts in the cluster there are more CPU and memory resource holes into which DRS can place running virtual machines to optimize the cluster’s performance.  The more hosts, the more options to the scheduler.

But on retrospect I think this is a weak argument.  Its not backed by data and in practice I cannot imagine a 16 host cluster being much more efficient than an eight host cluster.  Once vCenter is managing hundreds or more virtual machines per cluster, it has an astronomical number of combinations for VM placement.  So, doubling the host (and the virtual machine count) should have little impact to cluster efficiency.

More importantly, with respect to the efficiency argument, maximum CPU and memory utilization will be bound either by the failover capacity or the target utilization, which is usually about 80%.  With 20% reserved for resource spikes, the failover capacity is equal to the reserved resources at a 4+1 HA cluster.  Any any cluster larger than this, the failover capacity is less than 20%.  This means that only target utilization bounds resource efficiency.

The efficiency calculation is a little more tricky if you want to size your cluster for target resource utilization after a host failure.  In this case each additional host provides some incremental value to the cluster’s utilization.  To size a 4+1 cluster to 80% utilization after host failure, you will want to restrict CPU usage in the five hosts to 64%.  Going to a 5+1 cluster results in a pre-failure CPU utilization target of 66%.  The increases slowly approach 80% as the clusters get larger and larger.  But, you can see that the incremental resource utilization improvement is never more than 2%.  So, growing a cluster slightly provides very little value in terms of resource utilization.

Now why might you want to keep a cluster small?  I can think of a few reasons.

It is generally wise to avoid mixing different classes of servers in a single pool.  DRS does not make scheduling decisions based on the performance characteristics of the server so a new, powerful server in a cluster is just as likely to receive a mission-critical virtual machine as older, slower host.  This would be unfortunate if a cluster contained servers with radically different–although EVC compatible–CPUs like the Intel Xeon 5400 and Xeon 5500 series.  In the former case ESX would be using its software memory management unit which could perform as much as 40% worse than the hardware MMU in the Xeon 5500.

(I will momentarily digress to answer a question I often get in my performance talks: what is the impact of Enhanced vMotion Compatibility (EVC) on virtual machine performance?  Briefly: very little to none.  The instructions that are disabled on newer processors only benefit applications that were compiled to use those new instructions.  Those applications are rare in the enterprise space.)

Given my recommendation that servers in a cluster should be of a similar class of performance, you will soon find that your purchasing patterns will influence your cluster size.  If you are one of the few people lucky enough to work at a company that is buying servers by the truckload, you can size your clusters however you want.  But the vast majority of VMware’s customers make smaller purchases of anywhere from four to 16 servers at a time.  These will make nice, homogenous clusters of moderate size.

One more argument Dave offered for keeping clusters small is to use clusters for logical separation of applications of different class.  By putting your mission-critical applications in a cluster of their own your “server huggers” will sleep better at night.  They will be able to keep one eye on the iron that can make or break their job.  In my opinion, using physical separation in a virtual world is resisting the complete cloud and hardware independent virtualization that we are all striving for.  But I cannot begrudge an administrator that wants to hold onto some semblance of physical hardware best practices while traveling the multi-year journey to the private cloud.

Another of Dan’s arguments against large customers is the cumbersome nature of their change control.  Clusters have to be managed to a consistent state and the complexity of this process is dependent on the number of items being managed.  A very large cluster will present unique challenges when managing change.

So, have I given a recommendation?  I am not sure.  If anything I feel that Dave, Dan and I believe that a minimum cluster size needs should be set to guarantee that the CPU utilization target, and not the HA failover capacity, is the defining the number of wasted resources.  This means a minimum cluster of something like four or five hosts.  While neither of us claims a specific problem that will occur with very large clusters, we cannot imagine the value of a 32-host cluster.  So, we think the right cluster size is somewhere shy of 10.

I am quite interested to hear your thoughts on this.  Perhaps the best guidance will grow out of the crucible of debate.

First, a little background on ESX’s storage queues.  There are two relevant queues in ESX.  First is the device queue, which has one instantiation at each HBA for each LUN.  Second is the kernel queue, which handles “overflowed” IOs that are waiting to be placed in a full device queue.

For Fibre Channel HBAs, the device queue’s default length is 32 commands.  It is much larger for iSCSI. No HBA, and thus no device queue, exists for NFS.  A 32 command queue is capable of opening 32 commands at a time.  Obviously, if you double this queue length then the queue will drive twice as many IOs to the volume.  For the rest of this article I will discuss queues in terms of the 32 element Fibre Channel queue.

Because one device queue is instantiated at each HBA for each LUN, a storage reconfiguration at an array can change the number of queues at an ESX host.  Increasing the number of queues increases the total number of IOs that the host can open against the array.  I demonstrated this in my VMworld presentation with the following figure.

Putting two VMs on two volumes results in up to 64 commands being opened from the pair of them at one time.

This figure shows the simple difference between two virtual machines sharing a single VMFS volume and two that each get their own.  In the first configuration, only 32 commands can be opened from the host and that single queue is shared between the virtual machines.  In the second configuration, the host can open up 64 total commands and each virtual machine can open up to 32.

Your first reaction to this might be, “Wow! I should put every VMDK on a VMFS volume of its own!  Then imagine the total throughput that the host could drive!!”  My first response to this is stop using so many exclamation points.  Nobody likes an overenthusiastic writer.  But second, you should consider that more is not always better.  In fact, I can think of several reasons why you should not reconfigure storage to multiply the number of queues:

1. Allowing a host to open many commands simultaneously may be good for the individual virtual machines but is likely to be dangerous for the shared infrastructure.  This could result in short but extremely intense microbursts of IO that could present challenges to your fabric or storage processors.
2. The device driver (and the HBA) can only open a fixed number of commands depending on the device’s implementation.  You have to use these sparingly.
3. The configuration that results in more queues necessarily requires more VMFS volumes which results in a greater administration cost.

In addition to reconfiguring storage to increase the number of device queues, you always have the option of increasing the length of ESX’s device queues.  This is documented on page 71 of the Fibre Channel SAN Configuration Guide.  But I will caution you from reconfiguring storage queues, too.  This requires manual changes at every host, produces longer queues that more quickly eat into the fixed number of commands each HBA can support, and increases the possible IO intensity every virtual machine on the host.

And if these detailed explanations are insufficient at explaining why storage queue manipulation is unproductive or even counterproductive towards your goal of optimizing your infrastructure, let me point out that VMware has years of experience at consolidating storage and they chose 32 commands per queue as the right number for most environments.  Trust their experience on this one.

Of course I would be remiss if I did not mention that there are rare times that a storage reconfiguration may help performance.  Redistributing virtual machines across different VMFS volumes or increasing queue depths can correct some issues.  And you can identify occasions where this change may help by a large kernel latency.

As I mentioned above, commands that are waiting for access to a full device queue reside in the kernel queue until a device queue slot becomes available.  On the whole, commands should only spend a fraction of a millisecond in the kernel queue on their way to the device queue.  A kernel queuing time of over one millisecond and certainly over two milliseconds suggests the virtual machines are not having their IO needs served fast enough.

You can see kernel queueing times in the kernel latency statistic reported in esxtop (counter: KAVG) and vCenter (counter: Kernel Latency).  When these latencies consistently average any whole number in milliseconds its time to investigate storage.  But know that slow storage can result in high kernel queuing times.  So, before you go manipulating queues, or reconfiguring your storage layout, make sure your storage is serving IOs in periods deemed acceptible by the storage teams (usually 5-10 ms).

This is kind of a long article by vPivot standards, I know.  But cut me some slack.  Chad Sakac bangs out footnotes and parenthetical digressions that are longer than this entry.  This content has already been covered in my VMworld presentations so if you have access to those recordings go listen to Kaushik and I present it there.  But for those of you that were unable to attend I wanted to present this important guidance for your consideration.

The first tool in the VI administrator’s toolbox should always be vCenter. Through the vSphere client you can use vCenter’s performance counters to confirm a problem with any of the four resources (storage, CPU, memory, network). vCenter’s 20 second sample window impedes its ability to eliminate a resource as a problem. This is because a three second spike in any resource will be smoothed and missed over the 20 second window. But when vCenter confirms a sustained resource bottleneck, it is sure to be the performance problem’s cause.

If vCenter fails to confirm an obvious performance problem, the administrator must next go to more precise, more time-intensive, and more knowledge-intensive tools such as esxtop and vscsiStats. esxtop takes more skill and time than vCenter but provides better resolution and more visibility into the system. vscsiStats is the most time-intensive tool and has limits with ESXi hosts but can uncover a world of detail invisible to esxtop and vCenter.

I estimate each tool’s chance of identifying a random performance problem as follows:

• vCenter: used in 90% of performance problems
• esxtop: used in 9% of problems
• vscsiStats: used 0.9% of the time

The remaining 0.1% of the time is when you engage your account team or your local VMware performance expert.

Even within each tool’s usage there is an hierarchy of investigation: storage, CPU, memory and network. My experience with troubleshooting has informed this decision.  Storage causes the most problems, then CPU, then memory, and lastly (and rarely) network. After each resource level is inspected in vCenter, a repeat of the inspection should occur on esxtop. Guest tools may be a third option for memory, CPU, and network but vscsiStats should always be consulted if the performance problem persists.

VMware’s growing array of performance management tools will change this flow somewhat. AppSpeed, for instance, adds the ability to make very educated guesses about resource bottlenecks based on inside information into the application execution.  Hyperic can provide in-guest process visibility and Ionix ADM will map application interdependenies to focus the investigation.  But, I will abstain from providing best practices on these tools until I have used them more. In all cases, however, the fundamental relationship of “easy first, precise later” remains.

VMware continues to work towards integrating all of these tools into a single view within the vSphere client. I expect that integration will improve the success rate of the performance layman in troubleshooting these problems. But I am sure that even into the distant future performance people will find their jobs secure.

ESX must expend CPU cycles to maintain running virtual CPUs whether they are being used by an application or not. This means that host efficiency drops as more vCPUs are put on the server. But applications that scale well with CPUs will deliver greater performance when their virtual machines have been given more CPUs. The administrator must therefore balance the desires of an individual application’s owner with the needs of the entire cluster’s of applications.

There are several resources that VI administrators can use to inform their decisions in virtual machine sizing. I have listed some of them below.

Bruce Herndon’s Cost-of-SMP Article

Last summer the VMmark team’s Bruce Herndon published an article on the cost of SMP. I summarized his findings in a vPivot article I wrote on the ESX 4 scheduler. There are two key messages that you can take away from these posts to inform your decisions on virtual machine sizing:

• Over-sized virtual machines only hurt system performance when the server’s CPUs are saturated.  When utilization is low, unneeded vCPUs only penalize the system’s CPU utilization, not the applications’ performance.
• Unneeded 2-way virtual machines are not very harmful to the environment.  But administrators should be very careful with 4-way virtual machines and larger.

Co-stop and Ready Time

Ready time indicates a vCPU waiting for an available core when it has work to perform. Co-scheduling stop time (or co-stop time) indicates a vCPU being paused by the scheduler to allow its sibling vCPUs to catch up. These two counters can help administrators recognize a certain kind of stress due to limited CPU resources.

Ready time is generally a sign of the unavailability of CPU. Correction usually requires the administrator reducing work on the host (migrating virtual machines, decreasing vCPU count, etc.) or increasing CPU capacity (more hosts or faster CPUs). Co-stop time is a sign that the scheduler is allowing vCPUs to develop skew while it runs portions of virtual machines on available cores. Considerable numbers for these counters are 10% ready time and 3% co-stop time. There is no guarantee that application performance is suffering if these thresholds are crossed, but a problem may be present.

The important thing about ready time and co-stop time is that they are signs that you are using all of the CPU you have available to you. This could be a Good Thing. But it could also be a surprise to you. When these counters get high it is a good time to start asking yourself if you capacity usage meets your expectations. If not, you should inspect your virtual machines to be sure that the applications are using the vCPUs you have given them. If your guest tools show poor in-guest utilization then decrease those VM sizes. That will free up resources in the cluster for more virtual machines.

Application Scalability Information

I wish we lived in a world where every ISV published data showing their applications’ abilities to scale with cores. Unfortunately for us, many software vendors have for years allowed their customers to assume that each doubling of cores would double the performance of the application. VMware has chosen to provide some scalability information so our customers know how well or how poorly applications scale. But every customer of a software company deserves to have the vendor provide guidance on sizing the server. And those vendors deserve the right to put these results out on their own products. Go talk to your ISV to get the information you need to size your virtual machines.

CPU Usage Calculations and CapacityIQ

I am belatedly updating this post with a fourth way of identifying oversized virtual machines: mathematical calculation or Capacity IQ.

When a virtual machine consistently uses only a fraction of its vCPU resources it is possible that the virtual machine can be downsized and still deliver the same application performance. The calculation to determine this is simple: multiply the vCPU count by utilization and round up. Set the virtual machine’s vCPU count to the result of that calculation.

If you own CapacityIQ it will make this calculation for you for every virtual machine in your data center. Here is an screenshot of its recommendations based on virtual machine CPU and memory utilization. Click for a clearer picture.

CapacityIQ monitors CPU and memory utilization to recommend VM downsizing.

Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.