If you have heard me talk in the past year, you have heard me relate stories of storage issues causing performance problems that are wrongly blamed on VMware. Mark Bowker of ESG recently presented a generalized version of a customer story that relates the same tale. Luckily (for me) the customer caught the problem before completing the deployment. This may have saved me a customer call to unravel a basic problem. But the customer was burned and virtualization progress slowed if not stopped.
Mark’s tale can be summarized in a few sentences:
- Customer designs virtual infrastructure, in this case with the help of a consultant.
- Post-engagement and after the implementation, someone realizes that their architecture’s storage may not support the applications’ needs.
- The customer is now stuck with a tough choice due to a design that fails to meet storage demands: buy more storage or virtualize fewer applications.
Usually I hear of these stories around step two, when I have been called to investigate a “virtualization” performance problem. I bear the bad news to the customer whose manager fumes at the idea that more money is needed for a system that is already paid for, implemented, and promised to his internal customers to deliver.
It is better for me, but just as bad for the customer, when this problem is discovered by the customer after the implementation is complete. I have been spending a lot of time talking to VMware’s field about this problem in the hope that they can fix the problem during vSphere design. I have been doing this by providing basic performance guidance for storage systems to build up our field’s storage expertise. But Mark comments on this in his article:
The bad news for these vendors is that they don’t want to ever be perceived as the bad guy in the stack, so they are finding themselves having to become SAN experts to support their customers. Crazy!
A seed of an idea has been planted in my mind by this comment. Maybe it is crazy that VMware is building storage expertise to solve customer problems. We should be able to call on our storage partners to provide the same. I am owning up to my part of this problem and making a 2010 New Year’s resolution a mere three months into the year: I will get VMware’s field to bring our storage partners into more designs and troubleshooting sessions. Our customers, our partners, and our field will all benefit from this cooperation.
I’ve always assumed SAN admins make good VM admins and should get involved in all virtulisation projects. Not because storage is always the problem, but because storage already spans multiple disciplines.
- Storage admins are used to multi-tenancy systems battling for resources
- Storage admins understand storage, networking (FC/IP), memory (and caching), OS’s (and drivers)
- Storage admins usually know quite a bit about app’s. They already deal with Exchange on a SAN – Exchange on a SAN inside a VM is no different!
Hi David
Agree with your comment 100% but then I’m a Storage guy first , then VMware
Scott,
I’m about 1 week before the POC phase of virtualizing an application. Customer, storage vendor, server vendor etc are spending a week load testing the gear before going production. This afternoon’s activities will be:
3. Review Capacity Planner Data
4. Configure MYSQL_simulator -
a. 4 GB RAM (32 bit windows OS)
b. 8 vCPU
c. PVSCSI – 65 GB virtual Disk – used for 4K 70% Read Random I/O – simulate largest table
d. PVSCSI – 20 GB virtual Disk – used for sequential I/O – redo-logs
e. PVSCSI – 100GB virtual Disk – used for 4K 70% Read Random I/O – simulate other tables
5. Use esxtop and vcenter and vscsiStats to note any anomalies