Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 

Lessons Learned from a Large Virtualization Implementation

By scott.shaw,

As background my company currently has ~3000 virtualized servers. ~400 TB is dedicated to VM. Of the total amount of virtualized servers 364 of them are virtualized SQL Servers. Our entire virtual environment runs on NetApp hardware and VMWare.

We've been virtualizing our servers for about 8 months. The project started in order to prepare our systems for a move to a new data center. This move is happening now and is supposed to be completed by end of July. Below is a list of my experiences and observations during this process. This also reflects feelings felt by other team members and other departments.

Encountered difficulties in customer and vendor buy-in

Early on there was immediate push back from customers and vendors against virtualization and especially against the aggressive timeline set to virtualize all our systems. For better or worse the corporate response to this was very heavy-handed. In many cases we were told to specifically not mention virtualization to any customer or vendor. If a vendor refused to support virtualization we were told to find a new vendor that would. In the end the vendor came into line and so did the customers.

Replacing older physical resources with fewer faster virtual resources was a difficult sell

During the P2V (physical to virtual conversion) process, resources were reduced on the systems. This was originally done without our knowledge. The physical server might have had 8 CPUs and 16 GB of RAM, but after the P2V we would see our system with 2 CPU and 4 GB of RAM. This caused a lot distrust and confusion. Some customers were very upset and worried about how this was handled. In the end the faster VM hardware proved more than sufficient to make up for the smaller amount of resources. In some cases, resources were added and the VM environment allowed this to happen seamlessly and transparently. The lack of communication was the problem in this case and not the logic behind reducing resources.

Post-VM errors are always attributed to the VM regardless of their actual cause

With any major change all subsequent problems are attributed to the change. As a database team we spent many hours trying to determine if post-VM errors were caused by the migration or something else. We became heavily reliant on the VM team to ease our mind and explain to us the new VM architecture. Again, there was some distrust but then we began to realize that there were very few post-VM problems that could be directly attributed to the migration. As we learned more about virtualizations, we increased the confidence in our ability to troubleshoot our systems.

DBA's must change the way they manage and think about SQL Server provisioning

Virtualization is excellent for helping us to not focus on the particulars of a physical system. For example; how much disk space do we need, or is there enough RAM? These can be easily provisioned later. The problem we have encountered is server sprawl. Since virtualization we have increased our SQL Server inventory by about 30% and this number is growing. Management of this sprawl is becoming a serious challenge.

Backup teams, storage teams, and database teams must work well together

Early in the process my DBA team had a deep distrust of the VM team. This was mostly because of a lack communication about the process but also the simple fact that we did not fully understand the technology and how it may affect our database systems. We were ultimately responsible for the databases and the new processes and architecture were being shoved on us, which could jeopardize our ability to manage the environment. This still poses a significant risk and we've gone through a huge learning curve in our attempt to understand new technologies, like SnapManager for SQL and vSphere. Still, we are working closely with the other teams and the lines of communication have been opened. We've learned it is critical to work together and share knowledge because each of our systems is so tightly linked.

Virtualization alters traditional backup and recovery methods

This may not be the case for other companies and also might be more of a symptom of our underlying hardware than of virtualization, but we have been forced into using NetApp's SnapManager for SQL Server as our primary backup tool going forward. This was dictated to us and implementing it has caused serious challenges, some of which have yet to be worked out. It has potential, but the product isn't quite ready for an environment as large as ours. The lesson to be learned is that virtualization puts the decision making power in the hands of the storage and server teams and in some cases they will dictate what tools you might use to monitor and backup your databases.

Overall, virtualization has been challenging but successful though long-term outlook is still unclear

Virtualization brings a lot to the table but the long-term outlook is still unclear. The system has not become more stable due to virtualization and, in fact, some of the operational costs have probably increased due to server sprawl and management overhead. Our systems may be virtualized but we migrated so quickly that we have yet to take advantage of many of the benefits. These benefits include cloning and setting up virtual test and development environments. Overall virtualization will provide us with many new opportunities but at the cost of many potential risks.

Total article views: 5985 | Views in the last 30 days: 2
 
Related Articles
FORUM

virtualization

virtualization

BLOG

Resource database

Today learned new thing about Resource database, as everybody knows resource database is the system ...

BLOG

Little info about Resouce DB (SQL Server 2008-2005)

Little info about System Resource Database in (SQL Server 2008/2005) The fifth system database in...

ARTICLE

Resource Governor

This article is about Resource Governor. A new feature introduced in SQL Server 2008. Special focus ...

ARTICLE

Virtualization Hints

More and more people are looking at virtual servers, but it's not as simple as just installing the s...

Tags
 
Contribute

Join the most active online SQL Server Community

SQL knowledge, delivered daily, free:

Email address:  

You make SSC a better place

As a member of SQLServerCentral, you get free access to loads of fresh content: thousands of articles and SQL scripts, a library of free eBooks, a weekly database news roundup, a great Q & A platform… And it’s our huge, buzzing community of SQL Server Professionals that makes it such a success.

Join us!

Steve Jones
Editor, SQLServerCentral.com

Already a member? Jump in:

Email address:   Password:   Remember me: Forgotten your password?
Steve Jones