What do you do if you come in and a database is suspect? What about you come in and someone else has already started a restore of the production database from midnight? What if any number of other failures have occurred or actions taken to restore service have occurred, what do you do?
First you figure out what has happened and what is happening. You don't take any actions, and I'd argue that you might not even want to stop what's occurring. Someone might have a great reason for starting that restore, so don't stop it for now. Besides, the database has already been whacked at this point. Stopping the restore doesn't help you.
The ability to remain calm in the face of crisis can be taught, but it's not something that all of us possess. I'd hope that anyone that deals with production systems can remain calm, but if you can't, then don't do anything rash, and try to get someone to double check your work before you go about making changes that could cause problems.
That can be hard. You will likely have management upset, and people telling you it is critical you get things working as fast as possible. That might be true, but getting them working improperly, or in an working in an unstable way doesn't necessarily help anyone. As upset as your boss might be if you move too slowly, it's likely he or she will be even more upset if you claim things are fixed and then they fail again.
Whether it's corruption because of bad hardware, or a simple fat-fingered delete, when you encounter a crisis situation, stop first, make an assessment, triage the situation, and then go to work making permanent repairs.
Steve Jones
The Voice of the DBA Podcasts
The podcast feeds are available at sqlservercentral.mevio.com. Comments are definitely appreciated and wanted, and you can get feeds from there.
You can also follow Steve Jones on Twitter:
or now on iTunes!
- Windows Media Podcast - 22.6MB WMV
- iPod Video Podcast - MB MP4
- MP3 Audio Podcast - 3.7MB MP3
Today's podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.