Comments on: vSphere Is Not the Performance Problem, Your Storage Is http://vpivot.com/2009/09/18/storage-is-the-problem/ Scott Drummonds on Virtualization Thu, 02 Feb 2012 12:54:27 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Todd Muirhead http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-1156 Todd Muirhead Fri, 11 Jun 2010 14:50:26 +0000 http://vpivot.com/?p=12#comment-1156 I have seen a negative scaling scenario with SQL Server in only one other case. The SQL DB was accessing an Oracle DB on another host for some of the data that it needed. Setting the SQL Server Processor Affinity manually greatly improved performance. On the Processors page of the properites settings for SQL Server, unselect Automatically Set Processor affinity and instead manually select all available processors. If you database is accessing an Oracle DB as part of the workload, I would expect this to have a fair chance of working. What was the performance scaling like up to 4vCPU? Is this an Intel Xeon 5500 series (Nehalem) based system? Todd I have seen a negative scaling scenario with SQL Server in only one other case. The SQL DB was accessing an Oracle DB on another host for some of the data that it needed. Setting the SQL Server Processor Affinity manually greatly improved performance. On the Processors page of the properites settings for SQL Server, unselect Automatically Set Processor affinity and instead manually select all available processors.

If you database is accessing an Oracle DB as part of the workload, I would expect this to have a fair chance of working.

What was the performance scaling like up to 4vCPU? Is this an Intel Xeon 5500 series (Nehalem) based system?

Todd

]]> By: Travis Foschini http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-1124 Travis Foschini Tue, 08 Jun 2010 21:46:31 +0000 http://vpivot.com/?p=12#comment-1124 Thanks for all your documentation relating to Vmware performance and SQL. I've read everything you've written or referenced relating to SQL and CPU performance or troubleshooting after running into a SQL performance/scalability problem in recent weeks which remains unresolved. We were benchmarking an important system and SQL performance get increasing worse when increasing vCPU's beyond 4. At this point, it's cheaper for my organization to change our virtual DB servers to native servers. I'm running out of time and the organization is quickly coming to the conclusion that Vmware can't handle large SQL workloads spite clear documentation to the contrary from yourself and others. I still look at this as challenge to be resolved, but my organization sees it as a problem. (actually my family sees it as problem also because I'm putting in extra time and effort till I get to the bottom of it) You're thoughts or recomendation would be very much appreciated... So here's the short of my problem as previusly summarized to VMware support: "First, our results and conclusions have not changed after additional testing during the past 24 hours. We've confirmed our results using an Active/Passive SQL cluster between a virtual node and native (physical) node. Both capacity and performance are significantly reduced when using a virtual db compared to the native db. Last night's benchmark demonstrate our previous findings that we get 20% more transactions when using 50% less virtual CPU cores. Results include: 4 vCPU cores @ 572 k/hr vs. 8 vCPU cores @ 471 k/hr. Furthermore, avg response times when using a virtual DB server compared to a native DB server are nearly twice as long in virtual. Results: 4vCPU virtual db user experience simulation @ 16.030 seconds (avg) vs. 4pCPU native user experience simulation @ 7.737 seconds (avg). Once again it was confirmed the native OS installation doesn't suffer from the same throughput problem and it yields a significantly higher quantity of transactions across the board. The native OS consistently increases in transactions with each CPU including CPU core quantities above 4, unlike the virtual DB. The throughput increases with each additional physical CPU core in the native host but throughput decreases with each virtual CPU core beyond 4-5. Results: 4 pCPU cores @ 727 k/hr, 6 pCPU cores @ 925 k /hr, 8 pCPU @ 1,065 k/hr. vs. 4 vCPU cores @ 572 k/hr, 6 vCPU cores @ 500-600 k /hr, 8 vCPU @ 471 k/hr. If all 8 CPU cores are used, the native DB achieves more than twice the transactional throughput of the virtual DB with 8 vCPU cores. Results: 8 pCPU cores @ 1,022 k/hr vs. 8 vCPU cores @ 471 k/hr." UGHHH! Thanks for all your documentation relating to Vmware performance and SQL. I’ve read everything you’ve written or referenced relating to SQL and CPU performance or troubleshooting after running into a SQL performance/scalability problem in recent weeks which remains unresolved. We were benchmarking an important system and SQL performance get increasing worse when increasing vCPU’s beyond 4.

At this point, it’s cheaper for my organization to change our virtual DB servers to native servers. I’m running out of time and the organization is quickly coming to the conclusion that Vmware can’t handle large SQL workloads spite clear documentation to the contrary from yourself and others. I still look at this as challenge to be resolved, but my organization sees it as a problem. (actually my family sees it as problem also because I’m putting in extra time and effort till I get to the bottom of it)

You’re thoughts or recomendation would be very much appreciated… So here’s the short of my problem as previusly summarized to VMware support:

“First, our results and conclusions have not changed after additional
testing during the past 24 hours. We’ve confirmed our results using an
Active/Passive SQL cluster between a virtual node and native (physical)
node. Both capacity and performance are significantly reduced when using
a virtual db compared to the native db.

Last night’s benchmark demonstrate our previous findings that we get 20%
more transactions when using 50% less virtual CPU cores. Results
include: 4 vCPU cores @ 572 k/hr vs. 8 vCPU cores @ 471 k/hr.
Furthermore, avg response times when using a virtual DB server compared
to a native DB server are nearly twice as long in virtual. Results:
4vCPU virtual db user experience simulation @ 16.030 seconds (avg) vs.
4pCPU native user experience simulation @ 7.737 seconds (avg).

Once again it was confirmed the native OS installation doesn’t suffer
from the same throughput problem and it yields a significantly higher
quantity of transactions across the board.

The native OS consistently increases in transactions with each CPU
including CPU core quantities above 4, unlike the virtual DB. The
throughput increases with each additional physical CPU core in the
native host but throughput decreases with each virtual CPU core beyond 4-5.
Results: 4 pCPU cores @ 727 k/hr, 6 pCPU cores @ 925 k /hr, 8 pCPU @
1,065 k/hr. vs. 4 vCPU cores @ 572 k/hr, 6 vCPU cores @ 500-600 k /hr, 8 vCPU @ 471 k/hr.

If all 8 CPU cores are used, the native DB achieves more than twice the transactional throughput of the virtual DB with 8 vCPU cores. Results: 8 pCPU cores @ 1,022 k/hr vs. 8 vCPU cores @ 471 k/hr.”

UGHHH!

]]> By: Why Does Cloning A VM From Template Take A Long Time? – Gestalt IT http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-398 Why Does Cloning A VM From Template Take A Long Time? – Gestalt IT Thu, 15 Apr 2010 16:00:56 +0000 http://vpivot.com/?p=12#comment-398 [...] http://vpivot.com/2009/09/18/storage-is-the-problem/ [...] [...] http://vpivot.com/2009/09/18/storage-is-the-problem/ [...]

]]> By: Why Does Cloning A VM From Template Take A Long Time? | VM /ETC http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-397 Why Does Cloning A VM From Template Take A Long Time? | VM /ETC Fri, 09 Apr 2010 11:43:54 +0000 http://vpivot.com/?p=12#comment-397 [...] http://vpivot.com/2009/09/18/storage-is-the-problem/ [...] [...] http://vpivot.com/2009/09/18/storage-is-the-problem/ [...]

]]> By: Brent Ozar http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-396 Brent Ozar Mon, 22 Mar 2010 14:14:13 +0000 http://vpivot.com/?p=12#comment-396 Great article, and I'd point out one additional detail. If your physical servers have multiple HBAs with true active-active multipathing set up (such that you can get more than one HBA's worth of bandwidth to a single LUN), then you need to understand how pathing works in virtualization. You won't get that true active/active multipathing for a single LUN unless you also install that same multipathing solution at the VMware host level, not at the guest. Great article, and I’d point out one additional detail. If your physical servers have multiple HBAs with true active-active multipathing set up (such that you can get more than one HBA’s worth of bandwidth to a single LUN), then you need to understand how pathing works in virtualization. You won’t get that true active/active multipathing for a single LUN unless you also install that same multipathing solution at the VMware host level, not at the guest.

]]> By: New: esxtop precis | vReference http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-395 New: esxtop precis | vReference Tue, 12 Jan 2010 16:07:30 +0000 http://vpivot.com/?p=12#comment-395 [...] This tools helps to identify performance issues specifically with your storage.  Most performance experts will tell you that storage issues lead to more performance problems than any other single [...] [...] This tools helps to identify performance issues specifically with your storage.  Most performance experts will tell you that storage issues lead to more performance problems than any other single [...]

]]> By: Storage Basics – Part II: IOPS | VMtoday http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-394 Storage Basics – Part II: IOPS | VMtoday Tue, 29 Dec 2009 12:51:40 +0000 http://vpivot.com/?p=12#comment-394 [...] with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance [...] [...] with a focus on VMware View/VDI and is also worth a few minutes of your time.  Also check out http://vpivot.com/2009/09/18/storage-is-the-problem/ for a rubber-meets-the-road post from Scott Drummonds on the importance of storage performance [...]

]]> By: Another Day, Another Misconfigured Storage « Pivot Point http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-393 Another Day, Another Misconfigured Storage « Pivot Point Mon, 16 Nov 2009 02:16:45 +0000 http://vpivot.com/?p=12#comment-393 [...] Organization (PSO) as they develop a performance troubleshooting service, I see repeats of a theme I wrote about a couple of months ago:  storage is causing most performance problems, not VMware.  In fact, I have yet to see a [...] [...] Organization (PSO) as they develop a performance troubleshooting service, I see repeats of a theme I wrote about a couple of months ago:  storage is causing most performance problems, not VMware.  In fact, I have yet to see a [...]

]]> By: David Francis http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-392 David Francis Mon, 21 Sep 2009 02:57:56 +0000 http://vpivot.com/?p=12#comment-392 Ohhh how true this is , I'm in the process of fixing an issue as we speak about this same problem. Itzik is right Vmware Admin's see capacity and thrown as much as they can on a data-store. Storage guy looks at his monitoring tools and says's WTF. Ohhh how true this is , I’m in the process of fixing an issue as we speak about this same problem.

Itzik is right Vmware Admin’s see capacity and thrown as much as they can on a data-store.

Storage guy looks at his monitoring tools and says’s WTF.

]]> By: Itzik http://vpivot.com/2009/09/18/storage-is-the-problem/comment-page-1/#comment-391 Itzik Fri, 18 Sep 2009 07:08:00 +0000 http://vpivot.com/?p=12#comment-391 Hi, Itzik Reich (EMC) here. it's amazing how it happens time after time..i think it's down to the fact that the VI admin understand "Capacity", and thestorage admins understand "IOPS". now, if only they could speak to each other.. Hi,
Itzik Reich (EMC) here.
it’s amazing how it happens time after time..i think it’s down to the fact that the VI admin understand “Capacity”, and thestorage admins understand “IOPS”.
now, if only they could speak to each other..

]]>