First, let me quote what I know Steven got right:

Flash memory is the dominant underlying chip technology for solid-state storage. But solid-state disk drives are just one packaging option for flash.

Steven then explains that flash can be added to the enterprise in a variety of ways.  Today’s most common alternative to SSD is the PCI expansion card.  Steven next extols the benefits of PCI-based flash and the drawbacks of SSD-based flash.  These include:

• Simplicity of design in PCI flash.  No SCSI or ATA controllers needed for PCI flash.
• Improved performance of PCI flash, for lack of bottleneck-inducing SCSI or ATA controllers.

Steven then concludes:

In a decade, SSD will seem a quaint throwback while flash memory will roar ahead.

I doubt this conclusion.

First, the argument that flash in PCI is faster than flash in SSD because SSD controllers will always be a bottleneck is nonsense.  Those controllers are created in the same silicon that creates microprocessors.  They can be implemented as fast as the hardware that drives the PCI-e bus.  The reason why PCI cards are faster is because the PCI-e bus can support up to 16GB/s of throughput while no storage array (today) can drive a single connection beyond 10Gb/s.  There is no need to create an SSD disk that supports 16 GB/s of throughput because no flash can serve it and no array can deliver it.  This is an example of designing to the current needs and this limitation will change.

A Standard Interface Common In the US.

But more importantly, one of the things we have learned in decades of computer science is that interfaces matter. Interfaces endure. Examples abound of good interfaces outliving their initial implementations. Instead of deciding to throw away a design and start from scratch, we improve the implementation and keep the interface, even if it is sub-optimal. One such example is the x86 architecture. It seems that the entire world has nearly agreed that this interface is how we want enterprise operating systems to communicate to processors.

(The funny thing about x86 is that years ago Intel abandoned the basic principle of their early architecture: complex instruction set computing (CISC). They designed their processors so programming the CPU would be easy but implementing it would be tough. Decades later, Intel introduced decoders on their processors that effectively translated CISC instructions to RISC microcode. They simultaneously offered a “better” pure RISC/VLIW architecture in the Itanium line. But the industry responded loudly: stay with the x86 interface that an ecosystem has come to depend on.)

I believe hard drives should be thought of as an interface.  Not just the protocol and connection by which data is read and written, but also the form factor that humans handle and that hardware vendors build around.

Why is it that the industry likes the hard drive “interface”?  Consider the following:

1. Hard drives have reasonable high-density form factors.  They are uniform size, fully enclosed, and rugged enough that an exposed component will not snag on a sweater and break.
2. Hard drive interfaces (SCSI, SATA, SAS, etc.) are created to be used frequently.  They are designed to be plugged and unplugged.  Pull and replace a SATA plug a thousand times and the connector will survive.  Try pulling and replacing a PCI-e card 1000 times.
3. We have existing means of aggregating thousands of hard drives into an array.  Because of the capacity drain of more PCI devices it is tougher to scale PCI cards to the same limits.

As it turns out, familiarity with these abstract concepts is not needed to size flash in your environment.  The same principles that Intel and AMD apply in sizing microprocessor cache can be applied to storage.  There are generalizations that will suit the majority of deployments.

First, a little background.  In building EMC’s Fully Automated Storage Tiering Virtual Pools (FAST VP), EMC studies the access patterns of over 3,500 arrays.  We measured skew and performance, capacity and footprint.  We experimented with storage layout and FAST VP block sizes.  We tried two-tier and three-tier configurations and sized each to find a best fit for the average case.  The results are summarized in the following figure.

The EMC study that preceded the launch of FAST VP identified three basic tier configurations to improve performance and footprint in 94% of environments.

Of the 3,500 arrays we analyzed, 12% of the workloads met criteria that we describe as “heavy skew”.  This means 95% of the IO occurred on 5% of the data.  In these configurations nearly all the hot blocks can be stored in flash when it is sized to 3% of the storage footprint.  In “moderate skew” environments, the addition of 15% Fibre Channel maintained performance with a footprint only slightly larger than optimal.  ”Low skew” environments still showed improvement over flash-less configurations in both performance and footprint, while at the same cost.

It was this analysis that led us to recommend the low skew configuration for unknown environments.  This has the following benefits:

• The cost of storage is the same as the flash-less configuration.
• The footprint is half the size of the flash-less configuration.
• Storage will be at least 20% faster for 94% of workloads.  Because this measurement was provided at low skew, and higher skew environments will more heavily exercise flash, and performance will exceed non-flash deployments by more than 40% under some workloads.

EMC’s Tier Advisor can help you produce a more precise guide to size your storage tiers.  But it is not strictly necessary.  Deploying a three-tier architecture will improve your existing array by reducing footprint and improving performance.  And if your environment has anything above low skew, adding rotating disks will capacity and improve efficiency.  This works because you will be approaching the more precise tier mapping for your environment’s workloads.

This ends my three-part series on flash in the enterprise.  I will conclude the series where it began.  I fell in love with flash when I installed an SSD disk in my MacBook Pro.  The impact to my own user experience was so dramatic as to revolutionize my own thinking about the nature of storage.  If your mind has not yet been similarly transformed, go get SSD for your consumer computers right away.  And know that everything we can experience for our own equipment we can deliver in the enterprise, too.

This article will describe the theory behind optimal cache sizing.  Practical guidance will follow in part three, the last entry in this series. I will again lean heavily on Denis Vilfort’s presentation that I offer for download on my blog.

Every performance discussion starts with an “it depends”.  I will kick off this discussion on the characteristic on which flash sizing depends: skew (a topic I discussed once before). Skew is the degree to which different environments will touch different amounts of data. High skew environments will access 1% of their data 80% of the time. Low skew environments will access up to 10% of their data 80% of the time. This is depicted in the following picture.

Cache, flash or otherwise, can be thought of as a FIFO for data. As your users interact with their applications, they generate new data and touch old data. This places new blocks at the head of the flash FIFO, keeping them in SSD cache for a longer period of time. These blocks remain in that cache until newer data pushes it out of this logical FIFO.

Skew is a somewhat abstruse concept. But it is more easily understood when you think of it as a data cooling rate. The rate at which applications touch data can be described as cooling. High skew environments have high cooling rates, because the applications are spending most time on a little data.  This means a great deal of data becomes lightly used, or cool. Low skew environments have low cooling rates, because they frequently touch more data, keeping it warm. The following figure shows cooling in action.

At some rate–which we are leaving purely theoretical at this point–applications slough off a certain amount of data into the rarely used, or “cool”, category. This information should fall out of the flash FIFO and be relegated to lower cost storage. The above figure shows that data centers with a low cooling rate of 1.4% will take 120 days for 80% of their data to become cool. This means slowly cooling environments need larger flash-backed cache.

Cooling rate dictates the amount of flash needed in enterprise storage. Because flash for cache purchased today will be used for the array for the until the next storage purchase, environments with rapid growth of data require a higher percentage of flash on day one. But because cooling rate is non-linear, the amount of flash needed does not scale linearly with the data growth. In non-engineering jargon, this means that an environment with 100% year-over-year growth of data does not need twice the flash of an environment with 50% year-over-year growth.

This is better shown in the following figure.

You can now see that, assuming you have efficient flash usage via some technology like cache, you can predict your flash needs with only three variables:

1. The amount of data in your environment.
2. Your yearly growth rate of data.
3. Your cooling rate.

Two of these numbers are easy to find but cooling rate is not. In fact, it is so difficult that I am not sure it is even obtainable in most environments. But without it how are we to recommend flash purchases? The answer is simpler than you think and the subject of the final article in this series.

After this revolution of my own mind, I want to dedicate a few blog entries to the issue of solid state storage in the enterprise.  First I want to frame the problem that flash both solves and causes.  In the second entry I will introduce some of the theory behind flash sizing.  My last article will give you some very simple practical advice on how to use flash in your enterprise.

These entries lean heavily on a presentation EMC’s Denis Vilfort presented within EMC a few weeks ago.  I am providing a PDF version of that presentation on this blog for your own usage.

The fundamental problem that flash is solving is summarized in the following figure.

The speed of rotating disks is dictated by the time it takes the hard drive head reach data on the platter. These mechanisms–a spinning disk and a moving arm–are basically unchanged in the past many decades. Array performance has evolved through the introduction of DRAM cache, intelligent prefetch, clever file systems, and other techniques. But the metaphorical hands of all enterprise storage vendors were bound by the physics of spinning platters and actuating arms. While latencies in hard drive technologies have stuck in the milliseconds, solid state devices like CPU, memory, and flash have improved to microseconds and nanoseconds.

This huge chasm in performance between disk and memory increasingly devalued high performance servers. When storage becomes the bottleneck for a business application, the performance of a server becomes less important. VMware introduced the importance of capacity in server sizing by showing the world consolidation. But performance of data intensive applications was still dictated by storage.

The initial challenge of solid state disks (SSD) was their perceived high cost when compared to hard disks. I remember a couple years ago hearing EMC’s confused messaging around its enterprise flash disks (EFDs): “much more expensive but much, much faster”. Customers were left with a confusing, cost-benefit calculation to make a purchasing decision.

A year and a half ago that EMC started using a more clear message for flash: in some cases, it is actually cheaper than rotating disks. The key is to identify these use cases. This is best shown in the following figure.

Because of the exceptional improvement in latency, flash drives provide dramatically higher throughput on random access patterns than hard drives. When normalized by throughput–IOPS in the case of the above figure–flash storage is much cheaper than any hard drive.

But there is another angle to storage efficiency. Even “cheap” hard drive storage was never as cheap as people thought. When arrays aggregate striped disks to sum their maximum throughputs, utilization is low. Furthermore, in some performance environments hard drives were short stroked, which meant only a small, fast section of the platter was used so performance could be maximized. With low utilization and short stroking, hard drive efficiency is very low. This means the nominal cost of rotating disks is much worse than people realized.

The low cost of your hard drives depends on their high capacity utilization. But how can you drive high capacity utilization in an environment where you have a storage performance bottleneck? Obviously flash is the answer.

SSD storage can help you solve your utilization and efficiency problems with rotating disks. But this assumes that you and your storage can determine how much flash you need and how to use it. And that will be the subject of my next entry in this three part series.

Many years ago I studied electrical engineering. I was an awful at it. Analog was never my thing. I much prefer ones and zeroes. But I challenge myself to think about electronics once every blue moon. So I decided to delve into SLC and MLC flash technologies to understand how they differ and why we should care.  The content below summarizes my online research and the little bit I remember from school. If you can add, correct, or update this article I would be happy to have your comments.

What is the Difference Between MLC and SLC Flash?

MLC flash uses many discrete voltage levels to store multiple values, or bits, per cell [1].  Single-layer Cell (SLC) technology uses fewer voltage levels to program a single bit of information to the cell.  MLC technology obviously produces greater density which means it stores more data cheaper.  But the higher density comes with a cost: MLC produces storage that is more sensitive to temperature changes, slightly slower, and more likely to fail than SLC flash.

SLC flash is ten times the endurance for write/erase operations [2].  At an average of 10,000 write/erase cycles an MLC flash cell will die.  SLC flash cells can sustain an average of 100,000 write/erase cycles.  But why do MLC flash cells fail more than SLC?

Why Do MLC Cells Fail More Than SLC Cells?

I am unable to find an answer to this anywhere on the web.  If you see one, I would love to read it.  But in the dark, dusty corners of my memory I remember enough about electronics to hazard a guess at this.  As I see it, there are two reasons why MCL flash should fail more than SLC: one reason is a statistical and the other is electronic.

The statistical argument is that MLC cells are being written two 50% more than SLC cells.  They will simply wear out sooner. An SLC cell is storing the value of zero or one.  When an application writes to the data being held by that cell there is a 50% chance that the cell’s value has changed and it requires reprogramming.  Because MLC flash stores two bits, there is a 75% chance that the new two-bit data differs from the existing value.  This means an MLC is written to 0.75 times for each 0.5 times an SLC cell is written. That’s a 50% increase.

The electronic argument is based on MLC flash programming requiring a wider range of voltages [3].  Higher voltages produce greater amperage.  This exacerbates electromagnetic migration.  And the higher voltage on the transistor’s gate will increase erosion of the polysilicon that separates the gate and the channel.  Both of these will result in circuit failure.

How Is MLC Being Made More Reliable?

Because MLC is so much more cost effective than SLC, industry innovation is improving MLC reliability.  Here are a few techniques I found online [4]:

• Hardware can level writes, which distributes writes throughout the device to avoid balance cell overuse and avoid hotspots.  This means an entire flash drive will tend to fail at once after a long time.  This as opposed to a small number overworked hotspots failing quickly.
• Hardware can include DRAM cache which can be used to coalesce writes, which decreases cell write count.
• Flash devices can be over-provisioned for error detection, correction, and dynamic bad cell replacement.
• There are also a variety of proprietary techniques from flash manufacturers.

One challenge with flash today is the lack of consistent and objective endurance measurements.  It is difficult for storage vendors to publish availability guarantees when the reliability of the underlying media is uncertain.  This means to support flash devices in its VMAX arrays–which are rated at six nines (99.9999%) availability–EMC has to do a tremendous amount of qualification of the devices. This qualification process should always mean that flash support in enterprise storage should consistently lag its support in consumer devices, where availability requirements are much lower.

Summary

No one denies that SSD storage is becoming more common in the enterprise. EMC’s support of MLC devices is only one of the items introduced by Project Lighting that will increase flash presence, producing better performing and more efficient storage. If you are interesting in learning more on the subject, follow the links below to the sources for this article. Also considering Googling “tlc flash” to see the higher density, less reliable Triple-layer Cell (TLC) that will certainly find its way to the enterprise after more innovation.

References

My information came from documents I found as a result of Google searches. Here are my recommendations for further reading.

1. http://www.smxrtos.com/articles/mlcslc.htm
2. http://www2.electronicproducts.com/Choosing_flash_memory-article-toshiba-apr2004-html.aspx
3. http://www.supertalent.com/datasheets/SLC_vs_MLC%20whitepaper.pdf
4. http://www.infostor.com/index/articles/display/1169849064/articles/infostor/disk-arrays/disk-drives/2010/july-2010/mlc-vs__slc_flash.html

Let us start this discussion with an scientific perspective.  The following graph shows trend lines of performance by disk size.  I have greatly reduced the complexity of this chart by choosing some convenient numbers for throughput (such as 100 IOPS per SATA device) and size (such as 200 GB for an EFD).

The capacity/performance trend of today's disk technologies.

This graph shows the performance and capacity provided by up to 200 disks of three different technologies: SATA, Fibre Channel, and solid state disks.  The first salient point that this figure screams at me is that the spread between the trend lines is extreme.  The capacity/performance trends of SATA and Fibre Channel are relatively similar.  EFDs are a long way from those two trend lines.

This large spread between these two technologies suggests two things to me:

1. The world will not much longer support the existence of two rotating disk technologies.  Assuming capacity and performance are the only criteria in disk selection, the differentiation between these two is not significant enough to merit both their presence in the market.
2. Since real workloads’ demands will never fall exactly on a trend line, some customers are in the unfortunate position of forcing their workloads to one of these trend lines.  This will likely result in extreme inefficiency in performance or capacity.

Let me first comment on the efficiency problem.  Any workload–or in the case of virtualization, a consolidation of multiple workloads–can be mapped to the above graph as a dot whose coordinates are defined by its capacity and performance requirements.  This dot will almost certainly fall somewhere between the SATA and EFD trend lines.

You can determine the number of either single technology (SSD or rotating disks) needed for that workload by force-fitting the dot onto a trend line.  Do this by tracing a line either up or right until it intersects the device trend line.  At that point, a homogenous collection of that device type is supporting capacity and performance with a surplus of one of these.  Since these trend lines are so far from each other, if the workload fell squarely between the two trend lines then you are guaranteed to have either a very, very large surplus of performance or a very, very large surplus of capacity.

It is because of the extreme separation of these trends that storage vendors must offer dynamic performance capabilities such as auto-tiering and large caches that come with dynamic placement algorithms.  In this world of incredible performance/capacity divergence, big caches and auto-tiering are not just nice to have.  They are essential to making efficient storage decisions.

Now back to the idea of the vast separation between rotating and solid state disks.  Because large caches and auto-tiering are so essential, those technologies will make better decisions if they have a greater spread between the two extremes.  The greater the spread, the more cheaply capacity can be provided to low-use workloads and the more cheaply IOPS can be supplied to IO-intensive workloads.  This means that technologies like EMC FAST and FAST Cache are going to drive greater spread between disk technologies over time.

Lastly, since block autoplacement benefits most from the spread of these trend lines, the future of intermediate trends is uncertain.  I therefore reason that Fibre Channel disks will eventually fall out of the market.  But know that before this happens storage arrays will need to be optimally placing blocks.  Block placement is great today, but is a long way from optimal.

EMC is placing blocks in its unified storage in 1 GB chunks and in the Symmetrix line using blocks smaller than 1 MB.  As long as the storage block placement is larger than the smallest IO size, there will be some inefficiency.  But as long as the block placement is smaller than the virtual disk size, customers are benefiting from a configuration that is much more efficient than the worst case force-fit above.

I recently sat in on a customer presentation in Singapore from EMC’s president of the Symmetrix and Virtualization Product Group, Brian Gallagher.  Brian talked about a lot of things but a slide he showed covering this same topic offered a more precise view on the mixed use of solid state and rotating disks.

Disk skew is another means of choosing disk types and counts to minimize cost and meet performance and capacity requirements.

We define skew here as the amount of IO that occurs on a subset of data. Unlike my simplistic analysis above, which assumes some flat capacity/performance requirements of the entire data set, real workloads access data in non-uniform ways. Locality of reference exists in the great majority of workloads and auto-tiering systems will produce outstanding performance/capacity fits when they can move a small number of hot blocks to the faster disk types.

The figure above contains some very rough calculations that in no way should be used as a substitute for careful calculation in your environment. But they do suggest four possible configurations where small amounts of flash, paired with very precise auto-tiering such as that provided by the VMAX, can result in storage layouts that are faster, cheaper, and smaller than configurations without these technologies.

There is a point here that bears repeating, since it has not yet penetrated the entire market. The addition of “expensive” solid state disks can produce a storage configuration that is less expensive than the configuration that lacks them.

I think I have chosen a fun time for a visit into the world of storage. I spent so much time diagnosing storage problems at VMware that I developed a lot of strong ideas about how customers should be better using stroage and how storage vendors could better support their customers. The entire storage industry is developing technologies that leverage the huge spread in performance/capacity between existing disks and those technologies are going to drive that spread even further. And as these optimizations mature, customers are going to derive even greater benefits.

A perfunctory bit of guidance I include in nearly all of my performance talks (such as the enthralling, entertaining, and cancer-curing* presentations from VMworld 2010 that I will repeat in Copenhagen from 12-14 October) is “follow your application best practices”.  Audiences usually nod and immediately forget because this recommendation we all know to be correct yet somehow ignore.  In that way it is like, “stay away from fatty foods”, “do not drink wine with pain killers”, or “pay attention during the flight attendants’ presentation”.

Part of the reason why people forget this nugget is because advice is general, and not crystalized in a technological explanation that embeds deep in the minds of the audience.  In this case the application best practice that should be followed is to separate data from logs, putting the data on something good for random read performance (like RAID 5) and the logs on something good for sequential write performance (RAID 10).  Obviously I want everyone to consolidate their storage to VMFS and enjoy the technology, but if you are putting VMDKs that contain each of these files on the same volume, you are ignoring application best practices.

In this case I recommend building two VMFS volumes.  One backed by RAID 5 and the other by RAID 10.  Put the data on RAID 5, the logs on RAID 10.  While you will change the access profile at the array by putting multiple log files on the same RAID 10 backed LUN, the resultant IO will be much more sequential write than had you mixed data file reads among them.  So, consolidate multiple data files onto the same RAID 5 LUN and consolidate multiple log files on the same RAID 10 LUN.

Furthermore, if you are using solid state auto-tiering to manage your volumes, you do not need to protect your database log file with this technology.  What I am talking about here is EMC’s Fully Automated Storage Tiering (FAST), which is the most popular thing EMC has created since I have been paying attention.  Despite what some people will tell you, solid state disks are the cheapest way to serve huge amounts of random reads.  But their benefits diminish when the profile is sequential write when they become unattractive from a cost perspective.

EMC’s FAST works by creating a volume that is like a vertical stripe of multiple RAID groups.  LUNs, which become VMFS volumes, are then placed in that FAST volume.  Since FAST is a great technology for solid state disks, RAID 5 is the most cost efficient configuration for database data, and solid state is wasted on sequential IO such as redo logs, my best practice for virtual storage configuration for databases workloads when FAST may be present can be boiled down to the following rules:

• Always create RAID 5 volumes for your read-intensive database data.
• Always create RAID 10 volumes for your database logs.  If you have write-intensive data, you may consider putting them here, too.
• If you have FAST, use it to stripe across multiple RAID 5 volumes of different disk types and put your random, read-intensive data on VMFS on this volume.

The last bullet is clearly the most important here. I really love FAST, and it seems that EMC’s customers are crazy for it.  But its not the technology you need for sequential write workloads like redo logs.  Separate those data onto their own “normal” (not FAST-backed) VMFS volumes that use no SSDs.  Then you will have the best of all worlds: optimally deployed disk technologies, application best practice compliance, and righteous virtualized database consolidation.

(*) The claims made by the author of this blog do not reflect the views of his employer, the conference organizers, the government of the Kingdom of Denmark, or reality, for that matter.

Understanding the Bottleneck

Let me show the value of over-commit and danger of swapping by way of an example. I will choose the following typical values to demonstrate my point:

• All virtual machines are on a single host which has 32 GB of RAM installed.
• Each virtual machine is sized to 8 GB of RAM.
• Each virtual machine has 25% active memory (%ACTV in esxtop and “Active” in vCenter).

A virtual machine’s active memory is dictated by the application and its usage.  But the VI admin has complete control over the number of virtual machines in the environment which means host active memory can be influenced by adding or removing virtual machines.  Because virtual machine active memory is always equal to or less than 100% the only way to drive the host active memory to 100% is to over-commit memory.  This is why hypervisors that do not support memory over-commit are simply not viable for data centers where memory optimization is a priority.

Identifying and Correcting the Bottleneck

The ongoing occurrence of swapping is identified by a non-zero swap rate in either esxtop or vCenter.  In addition to swap rate, esxtop provides a swap wait time in its CPU panel.  When swap rate exceeds hundreds of kilobytes per second or swap wait time exceeds a couple percentage points, it is time for corrective action.

There are three possible solutions to this problem:

1. Balance the virtual machines’ memory usage by moving virtual machines from hosts with higher amounts of memory usage to hosts with lower amount of memory usage.
2. Run fewer virtual machines.
3. Buy more memory.

Designing Your Infrastructure to Simplify Memory Management

Ultimately I owe you a full white paper on memory management to provide a sufficient answer. But I want to give you two ideas of the tools and techniques that I will be describing when in this future paper.  First, place host swap files on solid state disk (SSD) stores to improve their performance.  With the right SSD device it may be possible to eliminate swap penalties.  Second, even if SSDs are unavailable consider consolidating multiple swap files onto a single store.  This will make swap rate monitoring very easy but may compound the performance penalties of swapping.

Stay tuned and VMware will provide more documentation on memory management in 2010.

But a recent article by my engineering colleague Chethan Kumar shows an avenue that allows VI admins to aggressively over-commit memory and avoid the catastrophic performance penalty of swapping: use solid state disks to host ESX swap files.

The fundamental problem with host swapping comes from the high latency of traditional disks compared to memory.  Data can be retrieved from memory in nanoseconds but takes milliseconds to fetch from a hard drive.  That means a single 4K memory page takes 100,000 times longer to retrieve if the operating system swapped it out.

The value that solid state disks offer to this problem is exceptional latency, as compared to traditional drives.  The SSD that Chethan used showed microsecond latencies, about 1,000 times lower than physical disks.  This means that  time spent waiting for swap activity* has been decreased to 0.1% of the time spent swapping to physical disks.

The importance of fast swap files is that it enables administrators to more aggressively over-commit memory.  Today our admins rightfully fear the VMs’ aggregate active memory exceeding the available physical memory, which results in swapping.  Today SSD technology in shared storage such as EMC’s new CLARiiONs allows our admins to cleverly place swap files and drive up memory utilization to previously unheard of levels.  This may enable standard memory overcommitment of 200% or more, with extreme over-commit being much higher than this.

In future versions of ESX we want to automate the usage of SSDs to maximize the use of available memory.  But that’s a roadmap discussion that I will leave for another day.

(*) This swap wait time has conveniently been added to ESX 4′s version of esxtop under the counter %SWPWT.  See Interpreting esxtop Statistics for more information.