SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Another Disaster (Almost)


Another Disaster (Almost)

Author
Message
Andy Warren
Andy Warren
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: Moderators
Points: 24413 Visits: 2746
Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/awarren/anotherdisasteralmost.asp>http://www.sqlservercentral.com/columnists/awarren/anotherdisasteralmost.asp

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
gljjr
gljjr
UDP Broadcaster
UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)UDP Broadcaster (1.5K reputation)

Group: General Forum Members
Points: 1458 Visits: 1
All I can say is OUCH!

I'm sitting here thinking that I sure better go back over my disaster recovery plan again and make another dry run to make sure I haven't missed anything! This is the type of wake up call we all don't need to have happen but must be prepared for. Excellent job keeping your users up Andy!

Gary Johnson
DBA
Sr. DB Engineer




Gary Johnson
Microsoft Natural Language Group
DBA, Sr. DB Engineer

This posting is provided "AS IS" with no warranties, and confers no rights. The opinions expressed in this post are my own and may not reflect that of my employer.
Andy Warren
Andy Warren
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: Moderators
Points: 24413 Visits: 2746
Thanks. We actually sat down today with our vendor to figure out what clustering will cost..another OUCH! I'll probably write up some info on that as well, amazing how quickly the costs add up.

Andy
http://www.sqlservercentral.com/columnists/awarren/

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
Clive Strong
Clive Strong
SSCrazy
SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)SSCrazy (2.7K reputation)

Group: General Forum Members
Points: 2735 Visits: 6622
Andy,

Great article. I think this topic is fantastic. Probably one of the best area's on this site, even tho there are only three entries. It's a good insight to other DBA's Disaster Recovery (DR) and it's always interesting to see what extra work you have to do on the fly when the problem falls out of the DR plan scope.

Fortunately (touch wood) we've not experienced anything like that.....but who know's what the future holds...especially as were moving to a new production box....

I guess you can test and test your DR plan, but computers, as wonderfull as they are, are always full of surprises!

Don't wish to curse all the rest of you out there, but any further disasters (or near disasters) would make interesting reading....


Clive Strong
sqlsrvr_dba@hotmail.com
chance
chance
SSC Rookie
SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)SSC Rookie (38 reputation)

Group: General Forum Members
Points: 38 Visits: 8
While the content of the article was useful, I must offer some well intended criticisim. I am not sure if english is your second language, or if you intended the article to read in the style of a personal journal. The article would have been much easier to read if you had used proper puctuation, spelling or even close to the correct grammar. I would expect higher quality writting on a site such as this one.



Andy Warren
Andy Warren
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: Moderators
Points: 24413 Visits: 2746
Nope, English is my first and only language. Well, I speak geek a little. We're pretty informal here, though we do try to get the spelling correct. We (Steve, Brian, myself) find most technical content hard to read, so we try for a less intense approach. In this case I did write in journal fashion because I wanted to try to present as best I could what happened/how I felt, not really try to clean it up.

We've got a book project under way to compile info from the site and the majority of readers did want us to correct typo's and minor mistakes, so we'll be reviewing old content and putting more time into proofing new ones. In the interim, if you see something wrong, please do point it out and we'll try to fix it.

Andy
http://www.sqlservercentral.com/columnists/awarren/

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
ljones1009
ljones1009
Forum Newbie
Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)Forum Newbie (3 reputation)

Group: General Forum Members
Points: 3 Visits: 1
I had some bad memories while reading your article. Things can get ugly in a big way when dealing with SQL Server...Can you believe that SQL Server 7 was supposed to be able to run itself without needing DBA's (right)? SQL 2000 should probably fly the space shuttle or something. I have found that when SQL breaks now...it's not a small issue.

At the United Network for Organ Sharing, we had SQL 2000 running on clustered servers with the databases on a SAN. All of a sudden (I'm not kidding), a very important (1 megabyte) database file disappears off the SAN and now our 240 GB database is suspect. Now nobody in the country can match organs.

Having the cluster didn't help, since the problem was on the SAN. We pointed the app to our Hotsite, which was kept up to date via Log Shipping. It took several hours the next day to get everything back to normal, but lives were saved, thanks to Log Shipping.



Steve Jones
Steve Jones
SSC Guru
SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)SSC Guru (140K reputation)

Group: Administrators
Points: 140329 Visits: 19415
Glad I'm not Andy or the last guy

I've been pretty lucky with very few disasters, and hopefully that will continue.

Steve Jones
sjones@sqlservercentral.com
http://www.sqlservercentral.com/columnists/sjones
www.dkranch.net

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Ojamie
Ojamie
Grasshopper
Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)Grasshopper (15 reputation)

Group: General Forum Members
Points: 15 Visits: 1
Reads like a story. We know we need to restore or re-snapshot when such a thing happens. What would have added value to this article is, "what originally caused the problems" and "how to identify the root of the problem" and "how to avoid such hardware failure (Checks one should do on a regular basis)"



Andy Warren
Andy Warren
SSC-Insane
SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)SSC-Insane (24K reputation)

Group: Moderators
Points: 24413 Visits: 2746
As I mentioned above, it IS written in journal style. As for your questions, you tell me? How could I know that the container would drop? What could I do to prevent it next time? Easy to say cluster everything, keep a warm standby, etc, but sometimes companies truly cannot afford it (or you can't convince them to afford it).

What I was hoping to share was how something gets handled when regardless of whether you tried to prevent it, could have prevented it, etc, things go bad and you have to make decisions. If you can learn something, think of something, not do something from reading my sad story, then I've done some good.


Andy
http://www.sqlservercentral.com/columnists/awarren/

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search