I’ve grown up reading Tom Clancy and probably most of you have at least seen Red October, so this book caught my eye when browsing used books for a recent trip. It’s a fairly human look at what’s involved in sailing on a Trident missile submarine…
VM snapshots are the best individualization feature since sliced vBread (yeah, I’m a geek). But…have you ever had a VMware snapshot grow out of control and fill the datastore the VM resides on? You know what happens… even the best VM admins out there seem to get burned once when they create a snapshot for some routine maintenance and just forget about it afterwards. The VM crashes and goes into a suspended state until free space is added to the datastore.
Sometimes this task is harder than it sounds. If your SAN is out of space, or the SAN management tools are out of your control, you could be stuck.
But… follow a simple trick to give yourself that last little bit of wiggle room in the event that a snapshot fills a datastore.
Add a 10GB text file to the root of the datastore that you can delete if you need headroom! I know it sounds too simple… but it’s simple and effective.
These directions are for VMware environments, but the same concept applies to any hypervisor out there today.
How do you do this?
Enable the SSH server, and remote into one of the ESXi hosts. This example is from a vSphere 5.1 ESXi server from my home lab.
Change directories into the root of the datastore that you want to create the file in.
Use the Linux command ‘dd’ to write a 10GB file (or whatever size of your choosing). Basically, we’re creating a file full of zeroes with a 1KB block size and 10 million blocks. Simple, eh?
dd if=/dev/zero /of=snapshotfailsafe.txt bs=1k count=10000000
It will take a few minutes to create the file. You can see the file creation process with another terminal, and can see the activity in the hosts’ disk performance view in the vSphere Client.
Once done, you’ll see it in the terminal and in the datastore view screen.
If you have a VMware snapshot that has filled the drive, just hit the datastore browser and delete the failsafe file! You just gained time to better manage the snapshot or move data around to accommodate the growth.
I know this seems unnecessary, but every VMware admin seems to get burned with a rogue snapshot once. Occasionally, it’s more than once.
How can I prevent actually needing this?
That part is pretty simple, too. Just set up a vCenter alert to warn you if you have a snapshot that is growing out of control! The directions for how to do this are located in this VMware KB article. (Just remember to set up the email server so you actually receive the emails!)