I joined in late to a conference call this morning from Quest where a number of their experts were talking about various disaster stories that they'd experienced over the years. It's great to hear real DBAs talking about the problems and challenges they've actually experienced, and it should give those that haven't some confidence that they can learn to deal with one.
One of the presenters said that you'd be guaranteed to have a disaster at some point in your career. At first I thought, sure, that's probably true and I've had a few of my own, but then I started thinking about all the DBAs that get interviewed, that don't know about restoring to a point in time, that never restore lots, just fulls, and that have jobs and are successful.
SQL Server gets more stable all the time. Hardware becomes more redundant, disk drives detect errors before they occur, CPUs are manufactured better, I wonder if everyone will have a disaster. And what's a disaster?
Is it when someone deletes data? Is it when a patch doesn't work and you have to uninstall it? I guess those as disasters, but they're minor ones. Rolling back an upgrade, patch, or application change might involve some of the skills that you use in an unexpected disaster, but I'm not sure I think that's what I'd consider a disaster. More an incident to me.
I do think, however, that you should be prepared, and if you haven't practiced any of those skills involved in recovering, you should. Everyone should
- restore a full and multiple logs as a test at least once a quarter, and preferably once a month.
- Restore to a point in time once a quarter, just to practice the skill.
- Track the version/build levels for your servers on a regular basis, preferably every day or week with some automation.
- Know your vendor support numbers, product keys, and other administrative details necessary for recovering your servers.
And probably more importantly, keep your resume up to date. If a disaster strikes, hopefully you'll be able to solve it and get things fixed. However you never know when you'll take the blame for it, whether you are at fault or not. And you don't know if your company will survive.
As my son has learned in scouting, "Be Prepared" for most anything that comes your way.