Now, as a member of the storage vendor community, I will answer for EMC. I will do so with detailed text from an outstanding TechBook I recently discovered on EMC’s Powerlink. This paper, Using Symmetrix Storage in VMware vSphere Environments (Version 7), provides incredible detail on the relationship between VMware thin disks and Symmetrix virtual provisioning. Its guidance is clear and simple.

My summary observations are as follows, with more detail below.

• The decision about thin/virtual provisioning is primarily a management one: trust your vSphere capacity management processes?  Use vSphere thin provisioning.  Trust your array capacity management processes?  Use virtual provisioning.
• VMware’s lazy zeroed thick disks reserve the space in VMFS.  But because blocks are left uninitialized until the first write, these disks are thinly provisioned in the virtual pool.  Choose this configuration when you have mature storage capacity management processes but questionable capacity management processes in vSphere.
• VMware’s thin disks do not reserve space in VMFS nor in the virtual pool.  This “thin-on-thin” configuration presents the most flexibility but requires mature capacity management in vSphere and the array.
• Even if your storage capacity management processes are mature, there are some cases where you may want eliminate the chance of an out-of-capacity problem.  This might be true with a mission critical application. In this case you have two options:
• Do not use virtual provisioning in the array, which may greatly increase your storage consumption.
• Use VMware’s eager zeroed thick disks on EMC virtual provisioning, which reserve capacity in VMFS and the virtual pool.  With VAAI, Enginuity records but does not perform the zeroing.  This means the zeroing is instantaneous and the block is reserved.

Here is a long quote from the TechBook, starting from page 128:

In general, before vSphere 4.1, EMC recommended using “zeroedthick” instead of “thin” virtual disks when using Symmetrix Virtual Provisioning. The reason that “thin on thin” was not always recommended is that using thin provisioning on two separate layers (host and array) increases the risk of out-of-space conditions for the virtual machines. vSphere 4.1 (and even more so in vSphere 5) in conjunction with the latest release of Enginuity integrate these layers better than ever. Consequently, using thin on thin is now acceptable in far more situations, but it is important to remember that risks still remain in doing so. For this reason, EMC recommends “zeroedthick” (for better reliability as only the array must be monitored) or “eagerzeroedthick” (for absolute reliability as all space is completely
reserved) for mission critical virtual machines.

It is important to remember that using “thin” rather than “zeroedthick” virtual disks does not provide increased space savings on the physical disks. As previously discussed, since “zeroedthick” only writes data written by the guest OS, it does not consume more space on the Symmetrix thin pool than a similarly sized “thin” virtual disk. The primary difference between “zeroedthick” and “thin” virtual disks is not a question of space, but a question of the quantity and capacity of the Symmetrix thin devices presented to the ESX server. Due to the architecture of Virtual Provisioning, binding more thin devices to a thin pool or increasing the size of them does not take up more physical disk space than fewer or smaller thin devices would although it does require a small amount more Symmetrix cache. So whether 10 thin devices are presented to an ESX host, or 1 thin device, the storage usage on the Symmetrix is essentially the same. Therefore, packing more virtual machines into smaller or fewer 128 thin devices is not necessarily more space efficient. Prior to ESXi 5, a single-extent VMFS volume was limited to approximately 2 TB and this led customers to use “thin on thin” to keep the number of devices presented to a host minimal as a way to ease management. “Thin” virtual disks allowed for fewer presented devices and a higher virtual machine density. Now with the larger allowable single-extent VMFS volume size in ESXi 5 (up to 64 TB) the need to add more thin devices is reduced as larger, more appropriately sized thin devices can be presented and used and expanded as necessary. With ESXi 5, customers therefore do not need to make the virtual disks “thin” in order to fit a large number of them on a single volume as it is no longer constrained by the 2 TB minus 512 byte limit.

However, as previously mentioned, VMware and EMC have recently ameliorated the practicality of using thin virtual disks with Symmetrix Virtual Provisioning. This has been achieved through refining vCenter and array reporting while also widening the breadth of direct Symmetrix integration through features such as the vStorage API for Storage Awareness (VASA—vSphere 5 only). Additionally, the slight performance overhead that existed with thin VMDKs
which was caused by zeroing-on-demand and intermittent expansion of the virtual disk as new blocks were written to by the guest OS has been significantly diminished with the advent of Block Zero and Hardware-Assisted Locking (ATS). The introduction of Storage Dynamic Resource Scheduler in vSphere 5 (SDRS) further reduces the risk of running out of space on a VMFS volume as it can be configured to migrate virtual machines from a datastore when that datastore reaches a user-specified percent-full threshold. This all but eliminates the risk of running out of space on VMFS volumes, with the only assumption being that there is available capacity elsewhere to move the virtual machines to. These improvements can make “thin on thin” a much more viable option—especially in vSphere 5. Essentially, it depends on how risk-averse an organization is and the importance/priority of an application running in a virtual machine.

If virtual machine density is valued above the added protection provided by a thick virtual disk (or simply the possible risk of an out-of-space condition is acceptable), thin on thin may be used. If “thin on thin” is used, alerts on the vCenter level and the array level should be configured.

If you want the full copy of this document, go to Powerlink and search for the paper’s title, “Using EMC Symmetrix Storage in VMware vSphere Environments”. If you change the “Filter by Content Type” drop-down to “Technical / White Papers” this document will appear first in your search.

Very few customers and partners require the full-time support of a VCDX. Most customers with even a modest VMware environment know the VCP certification does not measure enough design and administration skills. To address this, in 2011 we saw the introduction of the VMware Certified Advanced Professional (VCAP).

As a member of the VMware partner community, I can tell you EMC highly values the VCAP certification for our pre-sales roles.  As a hiring manager, I consider the VCP the minimum proficiency for anyone that will sell into today’s enterprise.  But technical sales experts and evangelists that brandish the VCAP-DCD or VCAP-DCA can engage in deep technical discussions on customers’ virtualization plans.  This is a great asset if you want to work for infrastructure vendors, resellers, or integrators.

Earlier today Alastair Cooke announced virtual, live VCAP training and study sessions at times convenient for those of us in the Asia-Pacific region.  This will be a four-session series of interactive lessons under the ProfessionalVMware BrownBag umbrella.

Alastair and the fantastic VMware’s Iwan Rahabok (VCAP-DCD!) developed this Asia-Pacific series.  The agenda is set, and some deep technical experts have already been booked to lead the sessions.  The last thing needed is you!

You can register for the series on GoToWebinar. Alastair is still looking for hosts for some of the sessions and has solicited volunteers on his blog. Please contact him using the details on his blog if you are interested.

Non-VI workload detected on the datastore

Or you may see the following warning in the vSphere client:

An external I/O activity is detected on datastore …

It is from this message that a bunch of smart questions arose from a former colleague of mine.  The bright guys at VMware–including Joey Dieckhans, a man I am very proud to have brought into VMware–provided a lot more detail about this situation.  There were so many interesting questions and answers that I will present a summary of our conversation in FAQ form.

What does SIOC do?

When VMFS volume latency passes a user-definable threshold, SIOC throttles throughput to the datastore.  Decreasing storage throughput should alleviate congestion on the datastore, hopefully resulting in a decreased latency.

What does this alarm mean?

If SIOC reduces throughput but latency does not decrease, SIOC assumes there is another workload driving storage IO to the volume and impacting latency.  This alarm informs the administrator that latency is not decreasing as virtual machine throughput is reduced.

How does behavior change once this alarm has been raised?

This alarm is effectively a sign that SIOC has given up and is no longer trying to throttle virtual machine access to the storage device.  This was deemed the proper behavior to guarantee that the virtual machines do not starve as non-virtualized workloads continue to access the shared datastore.

Has this alarm’s behavior changed in vSphere 5?

In Joey’s words, in vSphere 5 the SIOC alarm is no longer on a “hair trigger”.  It will make sure that there are several observed anomalies before raising the alarm.  This should reduce the alarm’s frequency, which was usually the result of a false positive.  Also, the alarm is no longer enabled by default in vSphere 5.  If you want to see it–and I recommend that you all do–you should enable the “Unmanaged workload detected on SIOC-enabled Datastore” alarm.

What type of configurations would raise this alarm?

In any environment where VMware is sharing storage with non-VMware workloads. Examples include two partitions on a LUN, or two LUNs in a RAID group, or two LUNs in a storage pool, or even two LUNs on different arrays but sharing common interconnect. The degree to which competing workloads will impact each other on a single array is highly specific to the workload and storage architecture. But since the alarm’s sensitivity was decreased in vSphere 5 the later possibilities of the above list are much less likely.

What does EMC recommend with respect to this alarm?

I do not want to speak for all storage vendors, but I think there should be no vendor-specific recommendation for this alarm.  Administrators should enable SIOC and they should be informed when SIOC is unable to provide its advertised value.  Administrators need not act on the alarm but they should be aware that SIOC has stopped trying. If SIOC functionality is required and this alarm is frequently raised, consider separating physical workloads to different pools or even arrays.

Where does SIOC run?

SIOC runs in the VMkernel using values configured by vCenter.  As a result, if vCenter goes offline SIOC will continue to function normally.

What exactly is done to throttle virtual machines?

Using the parameters passed to it by vCenter, SIOC can calculate the relative priority of each of the virtual machines on that ESX host.  SIOC can then parcel out queue slots based on relative priority of each virtual machine that shares the VMFS volume.

Is there a cool video that would put this feature to demonstration with clever graphics and a catchy song?

Why, yes!  See the video Joey created around SIOC’s release.

I will update this entry as questions arise.

The survey results are below with my own observations.  To the impatient I offer the following summary:

• Only 40% of respondents ran DR tests at least once a year before SRM was installed. 95% of them run DR tests at least once a year with SRM.
• 73% of respondents were not confident that their pre-SRM DR test would succeed. 95% are confident their DR test will succeed with SRM.
• 54% of customers before SRM never tested their DR plan.
• 97% of customers with SRM complete their DR tests in hours.
• 76% of respondents claimed SRM decreased production downtime during DR tests.

Many software products provide value that is directly measured by customers. With the hypervisor, customers can compare the number of servers before and after consolidation and assign a value to the software that reduced the server count. But some softwares’s value is difficult to measure, although obvious in hindsight. SRM is one example on this.

Consider the results above. How do you quantify the value of reduced production downtime during DR tests? How do you assign currency to confidence? How do you measure the benefit to business of regular DR tests? While it is difficult to assign value to each of these, most of us have a gut feel that we are talking about very big numbers.

In short, if you are not running SRM today then you are missing an opportunity to simplify DR, reduce its cost of maintenance, and improve your confidence in your DR plan.  The value of confidence to the business is difficult to measure but impossible to deny.

The great majority of customers without SRM lack confidence in their DR plan. The great majority of those customers claim confidence after installing SRM.

The frequency of DR tests increases dramatically with SRM.  DR test frequency is a direct predictor of customer confidence.  It is also a requirement in environments where DR tests are audited.

Nearly all respondents claimed that DR tests could be completed in hours with SRM present.  For anyone that has ever had to cancel weekend plans or spend an entire night in the datacenter to run a DR plan test this is welcome news.

Production downtime decreases with SRM because the test recovery site is fenced off from production so business-critical applications are unaffected by the test.  Production downtime can be measured using colossal numbers.

Because SRM is so well liked many new applications are pulled into the DR plan, improving the resiliency of the business.

There are a couple of things that later made me rue my statement.  First, any IT company with revenues measured in tens of billions is relevant everywhere in IT.  Even Apple should be considered a threat to companies selling to the enterprise.  Second, Microsoft is ubiquitous.  Like them or not, they still own the most popular enterprise operating system by far as well as the Microsoft Office cash cow.

The spirit of my poorly worded comment is that we are not seeing today from Microsoft the enterprise computing leadership they showed during the 1990s.  I have seen this depicted in a graph of their share price, whose shape is attributed to a change of leadership.

Consider the applications that drive so much of EMC and VMware’s business today.  Microsoft introduced this steady stream of good-to-great OSes and applications that are now deployed in huge volumes:

• 1989: Microsoft Office released.  I believe that Office is the best product Microsoft makes today.  I love it.
• 1993: Windows NT released.  Remember how ridiculous of an idea it was to consider replacing UNIX with Windows?  Who’s laughing now?
• 1993: SQL Server released on Windows NT.  People may view Oracle as the 800 lb gorilla in the database space. But, by instance, SQL Server is more widely deployed.  We still see many, many more SQL Server VMs than Oracle.
• 1996: Terminal Services first released with Windows NT 4.0.  Terminal Services (and XenApp, Citrix’s extension to it) was the most common application in VMware virtual machines a few years ago.
• 1996: Exchange Server released.  Exchange remains the undisputed leader in enterprise mail.
• 2002: .NET launched.  With my UNIX-focused education as a software developer, in 2002 I laughed at the idea of using a Microsoft framework.  In 2005 I was writing applications on it.

I think these few softwares show some of the innovation and staying power of Microsoft in the enterprise.  Making sure each of these worked on VMware consumed years of my professional life.  Microsoft’s problem is not that they do do not have great products out.  Its problem is that they have not introduced “the next great thing” in the software space.  Xbox seems to be a mighty product.  But it does not look like their current projects will one day obtain deployment rates of Office, for example.

So, what will Microsoft 2.0 look like? It could be based in Palo Alto. Think about the components of Microsoft software stack I listed above: operating system, database, virtual desktops, messaging, development frameworks.  Now look at what VMware has done recently:

• 2001: VMware releases the first version of ESX Server.  Measured by installations on bare metal, ESX is already the second most popular operating system in the x86 market.  Will it some day replace Windows as number one?
• 2006: VMware releases VDM, the product that will become View.  VDI (both VMware’s View and Citrix’s XenDesktop) are clearly eroding the “virtual desktop” dominance of terminal services.
• 2009: VMware acquires Spring to bring to market its own development framework.
• 2010: VMware acquires Zimbra, for its enterprise messaging capabilities.
• 2011: VMware acquires Slide Rocket, a potential SaaS replacement to part of the Microsoft Office suite I love so much.
• 2011: VMware releases vFabric Data Director.  Like Microsoft SQL Server, vFDD is derived from the code of someone else’s database.

It is clear that VMware followed since 2011 the path Microsoft laid out in the 1990s.  In some ways this makes VMware look like it is copying the Redmond giant.  But VMware would argue that each of its offerings are vast improvements on those from Microsoft: ESX improves upon Windows, View upon Terminal Services, Spring upon .NET, etc.  Whether the industry commits to VMware the way it did to Microsoft in the 90s is yet to be seen.

Anyone want to place some bets on what the next Microsoft will look like?

But the world is not ready for SaaS yet. There is a key technology that is required by some that will benefit all. I call it private data search.

Most private data is on local drives which means private data search is the same thing as a desktop search. Users that outsource services to the cloud spread private data around the world. As this happens, the world will need an instantaneous search capability that will find personal data anywhere in the world.  Not just data on the web or the local hard drive, but also files on Dropbox, presentations on SlideRocket, order details on salesforce.com, or anything any location.

A global private data search tool requires several pieces:

• A slick, cross-platform interface for user interaction.
• Smart search capabilities.
• User authentication and account management.
• SaaS modules that allow the front-end to either fetch raw data from remote services or initiate searches at those services and return a result.
• An private data API agreed to by the SaaS community that defines how data searches work, how results should be aged, and how search requests will be authenticated.
• Support user extensions of metadata for any remote content. This would enable tagging, threading, and prioritization of data that may not support the same.

The tough part of this solution lies in the SaaS module plugins and user authentication.  SaaS vendors will want to work with a single authority that manages user authentication in a secure manner.  VMware’s work with Horizon Application Manager can help with this and few companies are in VMware’s position to work with SaaS in a non-competitive way on this centralization.

It was Spotlight–Apple’s desktop search tool–that made me fall in love with my Mac.  The idea of a tool as simple and elegant as spotlight that searches every bit of information I have created in the entire world is intoxicating.  Every person I have talked to about this idea has loved it.  And some have suggested that there are projects under way today to solve this problem.

First, let me quote what I know Steven got right:

Flash memory is the dominant underlying chip technology for solid-state storage. But solid-state disk drives are just one packaging option for flash.

Steven then explains that flash can be added to the enterprise in a variety of ways.  Today’s most common alternative to SSD is the PCI expansion card.  Steven next extols the benefits of PCI-based flash and the drawbacks of SSD-based flash.  These include:

• Simplicity of design in PCI flash.  No SCSI or ATA controllers needed for PCI flash.
• Improved performance of PCI flash, for lack of bottleneck-inducing SCSI or ATA controllers.

Steven then concludes:

In a decade, SSD will seem a quaint throwback while flash memory will roar ahead.

I doubt this conclusion.

First, the argument that flash in PCI is faster than flash in SSD because SSD controllers will always be a bottleneck is nonsense.  Those controllers are created in the same silicon that creates microprocessors.  They can be implemented as fast as the hardware that drives the PCI-e bus.  The reason why PCI cards are faster is because the PCI-e bus can support up to 16GB/s of throughput while no storage array (today) can drive a single connection beyond 10Gb/s.  There is no need to create an SSD disk that supports 16 GB/s of throughput because no flash can serve it and no array can deliver it.  This is an example of designing to the current needs and this limitation will change.

A Standard Interface Common In the US.

But more importantly, one of the things we have learned in decades of computer science is that interfaces matter. Interfaces endure. Examples abound of good interfaces outliving their initial implementations. Instead of deciding to throw away a design and start from scratch, we improve the implementation and keep the interface, even if it is sub-optimal. One such example is the x86 architecture. It seems that the entire world has nearly agreed that this interface is how we want enterprise operating systems to communicate to processors.

(The funny thing about x86 is that years ago Intel abandoned the basic principle of their early architecture: complex instruction set computing (CISC). They designed their processors so programming the CPU would be easy but implementing it would be tough. Decades later, Intel introduced decoders on their processors that effectively translated CISC instructions to RISC microcode. They simultaneously offered a “better” pure RISC/VLIW architecture in the Itanium line. But the industry responded loudly: stay with the x86 interface that an ecosystem has come to depend on.)

I believe hard drives should be thought of as an interface.  Not just the protocol and connection by which data is read and written, but also the form factor that humans handle and that hardware vendors build around.

Why is it that the industry likes the hard drive “interface”?  Consider the following:

1. Hard drives have reasonable high-density form factors.  They are uniform size, fully enclosed, and rugged enough that an exposed component will not snag on a sweater and break.
2. Hard drive interfaces (SCSI, SATA, SAS, etc.) are created to be used frequently.  They are designed to be plugged and unplugged.  Pull and replace a SATA plug a thousand times and the connector will survive.  Try pulling and replacing a PCI-e card 1000 times.
3. We have existing means of aggregating thousands of hard drives into an array.  Because of the capacity drain of more PCI devices it is tougher to scale PCI cards to the same limits.

As it turns out, familiarity with these abstract concepts is not needed to size flash in your environment.  The same principles that Intel and AMD apply in sizing microprocessor cache can be applied to storage.  There are generalizations that will suit the majority of deployments.

First, a little background.  In building EMC’s Fully Automated Storage Tiering Virtual Pools (FAST VP), EMC studies the access patterns of over 3,500 arrays.  We measured skew and performance, capacity and footprint.  We experimented with storage layout and FAST VP block sizes.  We tried two-tier and three-tier configurations and sized each to find a best fit for the average case.  The results are summarized in the following figure.

The EMC study that preceded the launch of FAST VP identified three basic tier configurations to improve performance and footprint in 94% of environments.

Of the 3,500 arrays we analyzed, 12% of the workloads met criteria that we describe as “heavy skew”.  This means 95% of the IO occurred on 5% of the data.  In these configurations nearly all the hot blocks can be stored in flash when it is sized to 3% of the storage footprint.  In “moderate skew” environments, the addition of 15% Fibre Channel maintained performance with a footprint only slightly larger than optimal.  ”Low skew” environments still showed improvement over flash-less configurations in both performance and footprint, while at the same cost.

It was this analysis that led us to recommend the low skew configuration for unknown environments.  This has the following benefits:

• The cost of storage is the same as the flash-less configuration.
• The footprint is half the size of the flash-less configuration.
• Storage will be at least 20% faster for 94% of workloads.  Because this measurement was provided at low skew, and higher skew environments will more heavily exercise flash, and performance will exceed non-flash deployments by more than 40% under some workloads.

EMC’s Tier Advisor can help you produce a more precise guide to size your storage tiers.  But it is not strictly necessary.  Deploying a three-tier architecture will improve your existing array by reducing footprint and improving performance.  And if your environment has anything above low skew, adding rotating disks will capacity and improve efficiency.  This works because you will be approaching the more precise tier mapping for your environment’s workloads.

This ends my three-part series on flash in the enterprise.  I will conclude the series where it began.  I fell in love with flash when I installed an SSD disk in my MacBook Pro.  The impact to my own user experience was so dramatic as to revolutionize my own thinking about the nature of storage.  If your mind has not yet been similarly transformed, go get SSD for your consumer computers right away.  And know that everything we can experience for our own equipment we can deliver in the enterprise, too.

At this quarter’s V-week we had two honored guest speakers: EMC vSpecialist CTO, Scott Lowe and VMware’s Michael White. Scott led discussions for all the attendees on key technologies from EMC and VMware. We talked View, VPLEX, SRM, vSphere, Isilon, HA, RSA, vShield, and much more. Michael led conversations focused on SRM and View so he could bring back to VMware ideas that shape their future products.

Customers that visit V-week enjoy a full day of deep discussions with experts like Scott and Michael.  We customize the agenda for each attendee to make sure they are getting the information they need to push their virtualization projects forward.  At this quarter’s V-week, we also hosted non-EMC discussions.  These included a book signing for the Singapore VCP club and a full day dedicated to large scale design led by Scott Lowe.

If you are interested in attending a V-week in Singapore please contact me.  We recruit the world’s best speakers and technical experts to lead the discussions most relevant to you.  Depending on your need we’ll talk EMC, VMware, or EMC+VMware.  Either way, you will be our valued guest and we hope to show you some local hospitality.

In my first year and a half in Asia I have seen that this region (Australia and New Zealand excepted!) adopts technology one or two years after America.  So, today in Asia our customers are starting to consider what SRM can do for them.

Today I want to ask your help.  If you are an SRM customer–happy or not!–please take two minutes to fill out the following survey, either below or directly on Survey Monkey.  I will later share the results of this survey on this blog.  If you leave your contact information in the survey I may email you to follow-up.  If you want to share positive or negative feedback I will connect you with the right people once you have given me permission to do so.