Planning for Failure

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » SQLServerCentral.com » Editorials » Planning for Failure

Planning for Failure

Rate Topic

Display Mode

Topic Options

Author

Message

Steve Jones - SSC Editor

Posted Wednesday, September 29, 2010 8:46 PM

SSC-Dedicated

Group: Administrators
Last Login: Today @ 3:19 PM
Points: 31,526, Visits: 13,863

Comments posted to this topic are about the item Planning for Failure

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help

Post #995711

cy-dba

Posted Thursday, September 30, 2010 6:57 AM

SSC Eights!

Group: General Forum Members
Last Login: Wednesday, August 17, 2011 7:09 AM
Points: 869, Visits: 963

Well said! Don't let anyone pressure you into coming up with a fast solution, no matter who it is.

Post #995886

Eric M Russell

Posted Thursday, September 30, 2010 8:08 AM

Ten Centuries

Group: General Forum Members
Last Login: Yesterday @ 7:38 AM
Points: 1,184, Visits: 3,414

Also remember that database backups are not just for production servers, it should also go for Development or QA environments as well. Even if the data itself is replaceable, the potential for an unexpected disaster is much higher. Without a recent backup to restore from, it could take days to stand up a development environment, even if you have all your scripts in source control.

"Winter Is Coming"

Post #995962

Nadrek

Posted Thursday, September 30, 2010 12:19 PM

Say Hey Kid

Group: General Forum Members
Last Login: Monday, June 10, 2013 1:08 PM
Points: 679, Visits: 2,038

When it doubt about a system level change, make a drive image first. I've done this many times, and every once in awhile it's a major lifesaver (we applied the patch, and it won't boot anymore). This gets more true the more archaic your equipment and OS gets, such as those finicky systems in the corner left over from over a decade ago that everyone's afraid to touch... but that have to be touched for a (security) upgrade or whatnot.

Acronis True Image, Clonezilla, Norton Ghost, pick your poison; but until you know it works, try a restore to a near-identical drive/RAID.

If you're really paranoid, drive image it, pull out the drives, put them on a shelf, put in new drives, restore the image, and _then_ do your upgrade; you know your image works (you just restored it), and you know you have the original drives sitting on a shelf, ready to be popped back in, as well.

Regardless, restore your backups regularly; if you support PITR, _test it_. Maybe someone slipped in a job that truncates the transaction log and nobody's noticed yet.

Post #996155

Steve Jones - SSC Editor

Posted Thursday, September 30, 2010 12:44 PM

SSC-Dedicated

Group: Administrators
Last Login: Today @ 3:19 PM
Points: 31,526, Visits: 13,863

I haven't often imaged the system, usually because in the event of a move to a new system, I'm dealing with new hardware, or I might be.

This is one place where virtual systems have a nice advantage.

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help

Post #996189

SQLRNNR

Posted Thursday, September 30, 2010 11:43 PM

SSCoach

Group: General Forum Members
Last Login: Today @ 2:37 PM
Points: 18,853, Visits: 12,437

If I have a doubt, I move slower, not faster. I become more conservative, and if I am unsure about the outcome of an action, I've learned that time spent making another backup at that time is time well spent.

More sound advice. Move too quickly and you may end up costing yourself a lot more time.

Jason AKA CirqueDeSQLeil
I have given a name to my pain...
MCM SQL Server 2008

SQL RNNR

Posting Performance Based Questions - Gail Shaw
Posting Data Etiquette - Jeff Moden
Hidden RBAR - Jeff Moden
VLFs and the Tran Log - Kimberly Tripp

Post #996435

GSquared

Posted Friday, October 01, 2010 7:18 AM

SSCoach

Group: General Forum Members
Last Login: Yesterday @ 1:45 PM
Points: 15,442, Visits: 9,572

The best way to prepare for the unplannable is to educate yourself and practice high-stress things.

Volunteer to help out in real disasters, like Haiti. After you've had to deal with that kind of pressure, you'll be better able to remain calm, cool and collected while dealing with something as trivial as a crashed server. If the biggest emergency you've ever dealt with is a pop quiz in school, a crashed server seems big and intimidating. If the biggest emergency you've dealt with involves stopping an arterial bleed, the server issue isn't intimidating any more.

No matter how much you've planned or prepared, the ability to deal with stress and pressure will make a gigantic difference.

You also, of course, need to know what you're doing. Don't just memorize a few commands, or have a library of scripts that you don't understand but know how to plug values into the variables. Understand how the systems work. Know the theory. And then practice using it. You should do timed tests for how long it takes you to get a server back up and running after a system crash. Turn a server off, then use a stopwatch to time from pressing the start button to having it fully operational with point-in-time restores done. Do that five times. If the fifth isn't faster than the first, figure out what you could do to make it faster, and do it another five times.

Preparation for disaster isn't just about making sure the servers and databases are prepared. It's also, and perhaps most importantly, about making sure YOU are prepared.

- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread

"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon

Post #996609

« Prev Topic | Next Topic »

Permissions