Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

Crash and Burn Expand / Collapse
Author
Message
Posted Monday, December 28, 2009 8:22 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Yesterday @ 5:38 PM
Points: 31,355, Visits: 15,819
Comments posted to this topic are about the item Crash and Burn






Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #839662
Posted Monday, December 28, 2009 11:51 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Yesterday @ 11:11 PM
Points: 35,769, Visits: 32,432
Heh... I absolutely agree... all the backups in the world mean squat unless you can restore them. The DBA's a couple of jobs ago went through that. Took 40 people ten days to almost get all the data back. It was a huge mess.

I noticed that Jeff lost the data on the hosting provider... the more I hear about things like this, the more I'm inclined to say keep the cloud... there's too much weather there.


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #839700
Posted Tuesday, December 29, 2009 6:52 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, July 17, 2013 12:22 PM
Points: 107, Visits: 290
Excellent point on checking ability to restore.

I started as IS manager at a company some years back and asked what they had for backup and restore procedures.

The VP of Sales (also their "Techie") promptly informed that they knew they had good backups because they hadn't gotten any error messages on their backup runs for over 6 months.

This was particularly odd to me when I noticed the dust on the backup tape that was in the drive. It looked like it had been in the machine 6 months itself.

A check in the backup logs indicated that last successful run was also over 6 months in the past, due to a dead drive.

We quickly replaced the drive, did backups, and tested restores.

I still haven't figured out how they managed to be so lucky that they made it 6 months without someone wanting at least a minor file restored.
Post #839817
Posted Tuesday, December 29, 2009 8:13 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Yesterday @ 1:51 PM
Points: 2,663, Visits: 4,087
Trust no one but you with your stuff. You are utlimately responsible.
Post #839853
Posted Tuesday, December 29, 2009 8:21 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, July 17, 2013 12:22 PM
Points: 107, Visits: 290
skjoldtc (12/29/2009)
Trust no one but you with your stuff.


When it comes to my data, I find it hard to even trust myself.

Trust, but verify...
Post #839860
Posted Tuesday, December 29, 2009 12:51 PM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, October 6, 2014 11:34 AM
Points: 327, Visits: 1,014
Agreed. Also, practice makes perfect. I perform restores on a regular basis. Not just for rehearsal, but also to refine DR recovery procedures and to confirm that a copy of the latest backup is not corrupt.



Post #840021
Posted Tuesday, December 29, 2009 1:55 PM


SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Thursday, October 16, 2014 10:06 AM
Points: 176, Visits: 728
In two of the presentations I regularly give, I emphasize the importance of maintaining backups that are restorable, including the need for storing backups offsite and regularly checking to see if backups can be restored. Often, when I say this during presentations, many members of the audience roll their eyes, as if saying to me, "yeah, duh, that's obvious." The reason I include such "obvious" advice in these presentations is because some DBAs don't put this obvious advice into action, and I want to provide a little reminder to them. So, if you attend one of my presentations in the future, and you hear this advice from me, please stand up and shout, "great advice man", so that those DBAs that roll their eyes will know that I'm not the only one who believes in the importance of DBA basics.

Brad M. McGehee
DBA
Post #840051
Posted Tuesday, December 29, 2009 1:59 PM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, October 22, 2014 5:12 AM
Points: 36, Visits: 397
Last year we had a full test of our disaster recovery plan (always comforting to be part of a company that has one). Mostly went well except that the disk segment replication to our DR site ended up with unsync'd filegroups for our primary db server so we couldn't do the restore, meaning a complete failure in the end.

If it was a real disaster, we would have had to resort to tape backups taking our DR recovery from 3 hours to over 48 hours, and having some data loss.

Always good to make sure a restore works.
Post #840054
Posted Tuesday, December 29, 2009 2:29 PM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Tuesday, November 11, 2014 4:53 PM
Points: 68, Visits: 155
I've encountered the rude awakening a few times myself. Once, at a company that had no DBA, just "the SQL boy," a manager issued a DROP TABLE statement by accident on a database in bulk-logged recovery mode. The transaction logs had never been backed up, and were 300GB on 20GB of data. We soon realized that in order to restore last week's backups (because they took so long they were only done on weekends), the 100GB free was insufficient, and needed some creative space management and several days of data transfers on a severely overloaded and badly designed system. They ended up regenerating the missing data over a three week period because it was faster. They didn't lose ALL their customers...

Post #840077
Posted Wednesday, December 30, 2009 5:28 AM


SSChampion

SSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampion

Group: General Forum Members
Last Login: Today @ 4:53 AM
Points: 14,196, Visits: 28,524
Excellent, excellent, excellent post. Nothing is more important than verifying backups, except, verifying that you know how to run a restore. You're absolutely right when you say that backups are no good unless you can restore them, but it goes beyond actually validating that the backup files themselves are valid and accessible. You need to know that you, and any other DBA's in the organization, can actually run a restore, knows how to read the file header, can do a point in time recovery, etc. Practice restoring databases not only validates that the backups are good, but that you're good as well.

----------------------------------------------------
"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood..." Theodore Roosevelt
The Scary DBA
Author of:
SQL Server Query Performance Tuning
and
SQL Server Execution Plans

Product Evangelist for Red Gate Software
Post #840263
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse