Before I describe the test and its results, I want to share an important point on guest defragmentation.  Most of the computer literate are aware that file fragmentation–the separation of logically contiguous pieces of a file–can hurt storage performance.  But many may not realize that free space fragmentation is as big of an issue.  When free space is fragmented, writes take longer and files are re-fragmented rapidly.  PerfectDisk defragments files and free space and the results below benefit from both of these improvements.

The following steps were used to setup the test environment:

1. A  new virtual machine was constructed with Windows Server 2008 on a single 50 GB virtual disk.  29 GB (58%) of the disk was populated with he OS and miscellaneous user files.  21 GB of free space remained.
2. They ran a tool that simulated months of user activity by reordering blocks to fragment files and free space.
3. The fragmented virtual machine was cloned.
4. PerfectDisk was run on the second virtual machine to produce a defragmented virtual disk.

The Raxco team then compared the performance of the fragmented virtual machine with the defragmented one by measuring the installation time of Microsoft SQL Server 2008. This workload was chosen because it produces a bounded, write-intensive test that can easily be monitored with vscsiStats.  Only one virtual machine was running on the host.

Let’s take a look at a few key graphs.

The number of IOs during the application install, organized by size.

This first histogram shows IO counts by size.  You can see that IO counts for all but the largest bucket have decreased because Perfect disk is reordering files to minimize small IOs and maximize large IOs.  For the storage hardware this means greater efficiency in processing IOs.  But it also means two things to ESX:

• More host throughput as the fixed HBA queue is now holding larger commands.
• Fewer virtual storage stack traversals resulting is lower CPU utilization.

However, PerfectDisk not only consolidates small IOs into large IOs, it also makes files logically contiguous as they are seen by the NTFS file system. This means less work for disk controllers when mapping logical to physical clusters on disk.

Next we have the vscsiStats seek distance histogram which shows the shift from random to sequential IO.

The seek distance histogram shows the number of logical blocks between each successive IO.

The seek distance histogram shows a clear increase in the number of IOs that were exactly one block after the previous IO.  This pattern, demonstrated in the bucket “1″ increase, reflects the increased sequential nature of accesses to the defragmented virtual disk.  In this case the fragmented disk access was 15% sequential while the defragmented disk was 30% sequential.

Let us next look at latency.

This histogram counts IOs by latency.

The latency histogram shows a decrease in all IOs across the board and a near elimination of IOs that took more than 30,000 microseconds (30 ms).  Those very slow IOs accounted for 15% of all the commands in the fragmented case.  By eliminating the 15% slowest IOs, you can imagine that the total IO performance and application execution time would greatly improve.  That is exactly what happened, as the following data show:

Let me repeat one of those amazing data points: the average IO latency dropped from about 55 ms to less than 4 ms. While this is a phenomenal number, the increase depends on characteristics of the storage system.  Since these improvements are configuration dependent, your results may vary considerably.

As Raxco continues its investigation I remain cautiously optimistic of the value of guest defragmentation.  I think the exchange of small IO for large IO is indisputably a Good Thing.  However, virtual environments are very complex and I harbor some concerns about guest defragmentation in virtual environments that must be considered.  For instance:

• Defragmentation in your virtual machines backed by linked clones may explode those VMs’ VMDKs’ consumption of their VMFS volumes.
• The value of increased sequential access in a single virtual machine will decrease some in consolidated environments. This is because multiple virtual machines’ sequential access gets interleaved at the array, increasing the randomness of the IO from the array’s point of view.
• Guest block reordering may have negative consequences to your array, as Vaughn Stewart argued in a comment to my first entry on the subject.
• The value of file defragmentation may be limited when applications produce small random block access to data files, as some databases tend to do.

Raxco is continuing to investigate guest defragmentation to respond to some of these concerns and provide best practices for PerfectDisk’s usage.  I will update you as the research continues.

Test Details

ESX Server Configuration

• ESX Version: 3.5.0 Update 1
• Motherboard: Intel S5000PSL
• CPU Type: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
• Number of CPUs: 2
• Cores per CPU: 4
• Logical Processors: 8
• Memory: 4 GB

Storage Configuration

• RAID controller: Adaptec RAID 3805
• Number of Drives: 4
• Drive Type: WD1001FALS 1TB 7200 RPM 32MB Cache
• Total Capacity: 4.0 TB
• Number of LUNS: 2
• LUN 1 RAID level: 5
• LUN 1 Capacity: 2.00 TB
• LUN 1 Partitions: 1
• LUN 1 Name: IOTesting
• LUN 2 RAID level: 5
• LUN 2 Capacity: 744.75 GB
• LUN 2 Partitions: 1

Datastore Configuration

• Number of Datastores: 2
• Datastore 1 Name: IOTesting
• Number of VMs: 2
• Capacity: 2.00 TB
• Target LUN: LUN 1 (from Storage configuration)
• Datastore 2 Name: ISO
• Number of VMs: 0
• Capacity: 744.75 GB
• Target LUN: LUN 2 (from Storage configuration above)

VM Configuration

• Number of VMs: 2
• Operating System: Windows Server 2008 R2 (64-bit)
• Memory: 2GB
• Number of CPUs: 2
• SCSI Controller: LSI Logic (no SCSI bus sharing)
• Number of Disks: 1
• Size of Disk: 50 GB
• Provisioning Type: Thick
• Backing Datastore: IOTesting
• Virtual Memory: none (pagefile disabled)
• Network: disabled

When people ask me about file system fragmentation I explain that fragmentation can come from two sources: the guest file system or VMFS.  In 2009 we included experiments in our thin provisioning white paper that showed that both internal and external fragmentation in VMFS have no significant effect on performance.  As for guest fragmentation, VMware has avoided the business of optimizing native operating systems so there is no extant, official guidance.

More precisely, the large number of mappings from the guest file to the disk make it difficult to know how changes to each can impact the system’s performance as a whole.  But in talking with this partner I realized that there are two inescapable truths that suggest guest defragmentation is critical in a virtualized environment:

1. Defragmentation can decrease the number of disk commands and the resultant IOPS.
2. The fewer IOs, the more efficient the virtualization.

Guest defrag tools will order each file’s blocks sequentially in the guest file system.  This will enable the guest to make a few number of calls to larger, contiguous data than had the blocks been separated on the guest file system.  By making fewer calls to larger blocks, the following things happen:

• The array can leverage its faster sequential access capabilities to improve storage throughput.
• The hypervisor handles fewer SCSI messages from the guest resulting in lower overhead.
• The smaller number of commands results in fewer outstanding operations in the 32-element HBA queue, which allows more virtual machines to access the storage concurrently.

I have not found out how much consolidated workloads have to gain from guest defragmentation.  Nor have I quantified the impact to shared storage of a shift from a larger number of small commands to a smaller number of large commands.  But I am going to work with this partner and see if we can publish some numbers in 2010.

mofcomp is a tool provided by Microsoft and integrated with Windows that is used to register WMI classes such as VMware’s Perfmon counters with Windows.  For reasons that we do not entirely understand, but are tracking with a PR, mofcomp is failing during the tools upgrade on some systems.  Ideally a reinstall/upgrade/repair of VMware Tools will force a rerun of mofcomp and register the classes.  But the tools installer is not configured to do this today as a mofcomp failure was not anticipated by the developers that wrote the VMware Tools installer.

The key to the fix is to force mofcomp to run a second time.  This can be done in two ways:

• Uninstall VMware Tools and then reinstall them.  Unlike a reinstall (also called a “repair” or “upgrade”) this will force mofcomp to be run during the second install.  It should work the second time.  If it does not, I would be interested to know.
• Install the counters using the vmStatsProvider executable, which I documented previously.

Sorry for this problem, everyone.  We are tracking this issue and plan to improve the robustness of the installer as soon as possible.

My colleague in product management, Praveen Kannan, has been working to extend Perfmon to show some ESX performance counters. This capability is automatically installed with VMware Tools on vSphere 4. But Praveen and I have made a stand-alone version available to those of you that are still on VI3. Download it here to give it a try.  [See note below if that link is dead.]

To install, place the file in an appropriately-named directory on any Windows VM on VI3. Double-click the executable, which will self-extract the files into the same directory. Run “install.bat” and you’re done.

Once you bring up Perfmon you’ll see two new performance objects on your computer: “VM Memory” and “VM Processor”. These objects contain counters exposed by ESX that accurately reflect the VM’s memory and CPU usage. Here’s Perfmon on my test VM after I’ve installed the tool.

This makes collection of host stats a breeze. Windows Management Instrumentation (WMI) programs can now easily get access to reliable host statistics. And anyone with access to Perfmon can get see their VM’s resource usage. Unlike guest-based statistics, the host-statistics shown through these counters accurately reflect resource usage in the presence of virtualization overheads and time slicing of VMs.

Disclaimer:

This is a pre-release “sneak peak” version. Eventually this tool will be available for download on vmware.com and supported by VMware. But today there is no support for this tool and you’re using it “as-is”. Use at your own risk and do not contact VMware support for help with this release.

That’s VMware’s official position on this tool. But feel free to comment here with any ideas about this great new feature.

If the Download Link Is Dead

Until the DLL is made available as a standalone product, we are hosting it on the VMware FTP server, which is swept regularly.  If the link is dead that means that the binary has been deleted to save space.  Leave me a comment here if you see this and I will again upload the binary for download.