To get the most out of your servers’ memory you have to over-commit it.  Go too far and the host will swap.  This fact is immutable in enterprise memory management.  Because rotating disks have seek times that are six orders of magnitude larger than memory, when your host swaps your applications’ performance suffers catastrophically.  Solid state drives (SSD) can mitigate the performance cost by reducing swap latency by a couple orders of magnitude.  But SSDs still have delays tens of thousands of times worse than memory.

VMware engineers have been working on a technology that we unofficially call on-demand memory compression (ODMC) or compression cache, depending on who you talk to.  The idea of ODMC is to avoid swapping by compressing a set of target pages to a special region.  We have measured the additional latency–the compression time–to be over a hundred times better than rotating disk latencies.  And this will decrease at CPU performance increases.

SSD devices will continue to play an important part in solving the performance problems of extreme memory over-commit because of the great volume of data they can serve at a speed much faster than rotating disks.  But it will be a secondary solution to the faster ODMC, which uses a small dedicated area of system memory.  VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
2. Use ODMC to a predefined cache to decrease memory utilization.*
3. Swap to persistent memory (SSD) installed locally in the server.**
4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.

(**) Part of our vision and not yet demonstrated.

The end goal of this prioritized use of different technologies is the reduction of the performance penalty of swap.  If VMware can reduce the penalty of swap, the virtual machines in extremely over-committed environments will not slow much when they need more memory than the host has available.  When this performance cost of swapping is reduced, you can safely drive your consolidation ratios even higher.

I cannot wait for the general availability of ODMC.  It is another technological accomplishment in a long line of innovation out of our phenomenal engineering organization.  At times I wonder when the last time another operating system software company introduced something really awesome to the industry.  But of course, I am biased.

When people ask me about file system fragmentation I explain that fragmentation can come from two sources: the guest file system or VMFS.  In 2009 we included experiments in our thin provisioning white paper that showed that both internal and external fragmentation in VMFS have no significant effect on performance.  As for guest fragmentation, VMware has avoided the business of optimizing native operating systems so there is no extant, official guidance.

More precisely, the large number of mappings from the guest file to the disk make it difficult to know how changes to each can impact the system’s performance as a whole.  But in talking with this partner I realized that there are two inescapable truths that suggest guest defragmentation is critical in a virtualized environment:

1. Defragmentation can decrease the number of disk commands and the resultant IOPS.
2. The fewer IOs, the more efficient the virtualization.

Guest defrag tools will order each file’s blocks sequentially in the guest file system.  This will enable the guest to make a few number of calls to larger, contiguous data than had the blocks been separated on the guest file system.  By making fewer calls to larger blocks, the following things happen:

• The array can leverage its faster sequential access capabilities to improve storage throughput.
• The hypervisor handles fewer SCSI messages from the guest resulting in lower overhead.
• The smaller number of commands results in fewer outstanding operations in the 32-element HBA queue, which allows more virtual machines to access the storage concurrently.

I have not found out how much consolidated workloads have to gain from guest defragmentation.  Nor have I quantified the impact to shared storage of a shift from a larger number of small commands to a smaller number of large commands.  But I am going to work with this partner and see if we can publish some numbers in 2010.

During my short cab ride from the Mandalay Bay to McCarran (LAS) the driver and I discussed routes and fares to and from the airport.  The route from the airport, favored by taxis that pick you up at LAS, follows the highway, starting on I-215 and then to I-15, at a cost of around $28.  This as opposed to the more direct, airport-bound trip on Paradise, Tropicana and the Boulevard, which takes less time and totals only $11.50.  I asked my driver, “Why do the taxis to the airport take the second, shorter, cheaper route when the hotel-bound taxis always take the highway?”  The driver answered simply: “Airport taxis are f****** crooks.”

I have spent a little time investigating these two routes on Google maps and they do not seem to be dramatically different in terms of total length.  But clearly the fares differ widely.  I am sure the taxis prefer a larger fare but might there be an innocuous reason for the highway route?  Perhaps the risk of traffic delay is decreased by taking the highway and avoiding the strip?

My driver today seemed certain that greed fuels the decision to take the longer route.  He said that no visitor should ever pay more than $12 for a ride from the airport.  He advised me to refuse payment to the next cabbie that took me for the long ride.  He also said a call to the taxi cab company, which gains from the overcharge, would have no effect.  But a call to the taxi cab authority is justified and would result in a $300 fine to the taxi company for the long haul.

Has anyone ever taken the short route from the airport?  Are there any problems with getting taxis to take that route instead of the highway?  Is there a good reason to take the highway instead of surface streets?