Usually this issue arises inside a new VMware customer or an established customer that has added new staff to the virtualization team. In both cases the administrators are familiar with existing tools and require good reason to retool their thinking and environment around a new measurement system.

I was discussing the response to these concerns with my friend and colleague Kaushik Banerjee, the head of VMware’s outbound engineering group.  Kaushik and I spend a lot of time thinking about communicating technical details to our customers and in this case we chose different approaches to answer the question.  Both responses are complementary, so choose the one that suits your needs.

Kaushik’s Approach: The Killer Examples

Kaushik suggested that if we show cases where the guest OS’s counters were obviously wrong that a naturally suspicious VI admin would never trust the guest counters again. To that end, I offer the following screen shots to make our point.

Perfmon's counters show utilization higher in the guest than the host reports.

This screen shot shows two counters available in Perfmon inside a Windows guest with the vmStatsProvider installed (available by default since vSphere). The darker, red line is the CPU utilization as reported by the guest. The lighter, greenish (?) line is CPU utilization of the virtual machine, from the host’s perspective. This is the real CPU utilization passed up to the host by vmStatsProvider.  Notice how the host is always reporting higher utilization than the guest.  This is due to one of the reasons why guest counters cannot be trusted: they are unaware of hypervisor overheads.

This second screen shot shows a different case where the host utilization is lower than that reported by the guest.  Again, the dark red line represents the guest OS’s report of CPU utilization and the lighter line shows the real CPU utilization as reported by ESX.

Perfmon's counters report a higher CPU utilization than ESX's.

The reason the host shows lower utilization than the guest is because the guest is unaware that it is only getting a fraction of the host’s CPU, time-sliced by ESX’s scheduler.  In this case the virtual machine was contending for CPU with other active virtual machines but this just the same principle would apply had a CPU limit been set.

Scott’s Approach: A Detailed Explanation

My approach to convincing VI admins to avoid guest tools is based on bottomless thirst for information that is common to technophiles. If I can provide an explanation for the underlying system of resource scheduling and manipulation, then our admins will be able to deal with the guest counter issue and maybe solve other issues with their newfound knowledge.

There are four reasons why guest counters cannot be trusted:

1. The guest is unaware of virtualization overheads.  As screen shot one showed above, the hypervisor will increase the CPU load as it virtualizes the hardware for the guest operating system.  That additional CPU work is not seeing by guest tools.
2. The guest is unaware that it is only seeing the portion of CPU that ESX’s scheduler is allowing it to see.  Because of contention or resource restrictions, virtual machines only get a slice of the CPU’s time.  When a guest thinks it is getting 100% of the CPU it may not know that the processor is being shared by eight other virtual machines. See the second screen shot above.
3. Time skew in virtual machines can change the sample window for time-based counters.  This means that the guest may have measured 10 milliseconds of time passage during a read command when 12 milliseconds have elapsed.  This is more common on older versions of ESX and when the host CPU is saturated.  More on this below.
4. The virtual machines are unaware that they are being de-scheduled when idle, which means that they appear to be working more of the time than they are.  Consider a case where a virtual machine is idle 90% of the time.  If ESX does not schedule the VM during its idle time then the guest will think that its processor queues are full 100% of the time that it is being executed.

The time drift explanation (item three) was historically the most problematic for VMware.  On older versions of our products time drift was common.  As ESX has matured we have reduced the amount of drift which has improved the accuracy of guest counters.  But the timer hardware is still being virtualized in software running on the host CPU.  This means that if host processor is fully utilized, the timer may not be scheduled on time, resulting in a delay in some ticks and a resultant skew in guest time.

References

Performance Best Practices and Benchmarking Guidelines.  This white paper was the last version that we printed that included benchmarking best practices, which contains some discussion on the need to measure performance from outside the host-under-test.

Timekeeping in Virtual Machines.  This document–not updated since VI3 but still accurate in its theory–will give the background on VMware-based time keeping and provide an explanation as to how skew can occur.

VMware vSphere™ 4: The CPU Scheduler in VMware® ESX™ 4.  This white paper provides great detail how the scheduler works which will fully explains the notion of time slicing.

mofcomp is a tool provided by Microsoft and integrated with Windows that is used to register WMI classes such as VMware’s Perfmon counters with Windows.  For reasons that we do not entirely understand, but are tracking with a PR, mofcomp is failing during the tools upgrade on some systems.  Ideally a reinstall/upgrade/repair of VMware Tools will force a rerun of mofcomp and register the classes.  But the tools installer is not configured to do this today as a mofcomp failure was not anticipated by the developers that wrote the VMware Tools installer.

The key to the fix is to force mofcomp to run a second time.  This can be done in two ways:

• Uninstall VMware Tools and then reinstall them.  Unlike a reinstall (also called a “repair” or “upgrade”) this will force mofcomp to be run during the second install.  It should work the second time.  If it does not, I would be interested to know.
• Install the counters using the vmStatsProvider executable, which I documented previously.

Sorry for this problem, everyone.  We are tracking this issue and plan to improve the robustness of the installer as soon as possible.

A colleague of mine at VMware, Geoff White, wrote a Python-based tool for esxtop batch file analysis. His tool, dubbed esxplot, provides a graphical interface to the data generated by esxtop’s batch mode. esxplot provides search capabilities to find relevant counters, can plot different counters on the same graph, and does not suffer from the size limitations of Excel and Perfmon.

Geoff has graciously provided unsupported access to this tool’s beta version (version removed — see update below) for my readers. Please test esxplot much as possible and comment here with your observations. We would like to know of any bugs you discover as well as ideas for improvement.

The esxplot package comes with Windows and Ubuntu binaries as well as the Python source. The binaries will run without need of additional libraries or tools.  But if you decide to use the Python script, as I have done on MacOS, you will need to verify the following:

1. Ensure that you have Python 2.6 or greater, but not Python 3.x.
2. esxplot uses wxPython, which you can download and install from here.
3. esxplot also requires NumPy, available here.

A manual is included in the package but I think you should find the interface simple and intuitive.  Here are a few tips to get you going:

• Load an esxtop batch file by specifying it on the command line or through File->Import->Dataset.
• Using the counter navigator in the lower left corner, navigate to the counter of interest.  For instance, to view percent ready you would first expand the host  (use the triangle to the left of the host name), then expand “Group Cpu (GID:VM)”, then select “% Ready”.  The graph will be displayed on the right.
• You can quickly find counters by entering a query search in the middle of the left panel.  This creates a new entry in the counter navigator that contains only counters whose name matches the expression you entered.  For instance, entering “ready” will create a counter query that contains all ready times in the batch file.

Please let us know what you think of this wonderful tool.

Update: 1/13/09

esxplot version 1.0 was released on 1/13/09. See the updated blog article for information on that release and continued discussion on this tool.

My colleague in product management, Praveen Kannan, has been working to extend Perfmon to show some ESX performance counters. This capability is automatically installed with VMware Tools on vSphere 4. But Praveen and I have made a stand-alone version available to those of you that are still on VI3. Download it here to give it a try.  [See note below if that link is dead.]

To install, place the file in an appropriately-named directory on any Windows VM on VI3. Double-click the executable, which will self-extract the files into the same directory. Run “install.bat” and you’re done.

Once you bring up Perfmon you’ll see two new performance objects on your computer: “VM Memory” and “VM Processor”. These objects contain counters exposed by ESX that accurately reflect the VM’s memory and CPU usage. Here’s Perfmon on my test VM after I’ve installed the tool.

This makes collection of host stats a breeze. Windows Management Instrumentation (WMI) programs can now easily get access to reliable host statistics. And anyone with access to Perfmon can get see their VM’s resource usage. Unlike guest-based statistics, the host-statistics shown through these counters accurately reflect resource usage in the presence of virtualization overheads and time slicing of VMs.

Disclaimer:

This is a pre-release “sneak peak” version. Eventually this tool will be available for download on vmware.com and supported by VMware. But today there is no support for this tool and you’re using it “as-is”. Use at your own risk and do not contact VMware support for help with this release.

That’s VMware’s official position on this tool. But feel free to comment here with any ideas about this great new feature.

If the Download Link Is Dead

Until the DLL is made available as a standalone product, we are hosting it on the VMware FTP server, which is swept regularly.  If the link is dead that means that the binary has been deleted to save space.  Leave me a comment here if you see this and I will again upload the binary for download.