And here, more than anywhere else, is where I believe DBAs really earn their money. There's no substitute for experience, and very specific experience at that. It's all very well having a backup and recovery process based on a theory that looks perfect, but until it's proven to work in practice, it's useless. And how do you know it's proven to work in practice? Test restores in "clinical" conditions? Nah. Test restores are definitely required, but it's only a DBA who's been in the hot seat before who can fully appreciate in just how many different areas problems can manifest themselves, and what effect they'll have on performing a recovery in practice.

Since we have several DBAs here where I work, we try to second guess providence by having one DBA think up the scenario and another to have to recover based on those circumstances. Quite often, we also us a routine upgrade to provide us with the raw material; for instance, decommissioning one physical server by upgrading to a new box gives us a spare system that was, until a few days before, our live production environment. That allows us to break bits and give a DBA practice in recovering a system that very closely resembles what they'd have to work with for real.

Mind you, it's still no surprise that even practice like this doesn't stop the unexpected happening when the f***-up fairy really does come for a visit.