Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Disaster Recovery Week


Disaster Recovery Week

Author
Message
Steve Jones
Steve Jones
SSC-Dedicated
SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)

Group: Administrators
Points: 36120 Visits: 18744
Comments posted to this topic are about the item Disaster Recovery Week

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
ALZDBA
ALZDBA
SSCertifiable
SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)SSCertifiable (7K reputation)

Group: General Forum Members
Points: 6974 Visits: 8839
just a very primitive tip: Start small !

Start building your disaster recovery project with a very small scope, not that critical to business activity.

Only read the headers of the articles / BP / books you find about Disaster Recovery, so you know the topics to keep in mind and are not overwhelmed by the tiny details behind it.

That is for later on, after you review your first, second, third cycle of building your general DRP.

Pick any non-critical database and start from there on.


Once you are comfortable with the elementary concepts, and their impact to the topic from your point of view, make sure your project gets sponsored by your superiour(s), because at certain point in time, you will have to contact business responsibles to discuss your findings and align them budget wise.


There is more to disaster recovery than SQLServer, but it makes a good starting point.

Johan


Don't drive faster than your guardian angel can fly ...
but keeping both feet on the ground won't get you anywhere w00t

- How to post Performance Problems
- How to post data/code to get the best help


- How to prevent a sore throat after hours of presenting ppt ?


"press F1 for solution", "press shift+F1 for urgent solution" :-D


Need a bit of Powershell? How about this

Who am I ? Sometimes this is me Alien but most of the time this is me Hehe
ashepard
ashepard
Grasshopper
Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)Grasshopper (13 reputation)

Group: General Forum Members
Points: 13 Visits: 18
Backups have only given me grief. Its restores that count & make me look good.

However I have gotten burned with "level of expectation"

1) A dual RAID-5 SAN died while I was on vacation. It was in the cleanest part of the city garage but too many disks failed. I came in, restored the database and maintanece jobs to another server in under six hours. I was proud - my boss was livid.
Why did it take six hours to restore and reconfigure an 18 gig database? He thought it should only take twenty min and the process be documented so anyone could do it.
Lesson learned - have a system up and ready.

2) We have several databases where we only keep six months or less of backups. I like to keep three years. Why? Problems do not always surface in six months. Year end processing has found issues with data from a year ago.

3) Would I use the cloud? Not even in an emergency - only a catastrophy. Why? (laughs) our internet connection is one of the things to go. Its not under our control - but the data center is. Those that can work while others are down make the money and keep their jobs.
crazy4sql
crazy4sql
SSC Eights!
SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)

Group: General Forum Members
Points: 929 Visits: 4422
I have a question on one of the DR scenario,
Lets say I have 14 TB database in oltp mode with no/least performance problem. (yes weired but lets accept it). Now lets say the system goes down because of any problem, and I have the valid set of latest backup. I can start restoring but the problem is how to meet the objective of RTO (recovery time objective) of 3 hrs. As restoring this ~tb size of full backup will definately take time to restore.

*************************************
I know there should be some faiover configured like replication,mirroring,cluster-failover or logshipping(least preferred). Or should have some offline site where backups getting checked and restored in time.
****************************************

Lets assume because of any reason these options was not considered and the only option now is restore. What would be my strategy to bring this database online within RTO?

----------
Ashish
Steve Jones
Steve Jones
SSC-Dedicated
SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)

Group: Administrators
Points: 36120 Visits: 18744
crazy4sql (12/10/2012)
I have a question on one of the DR scenario,
Lets say I have 14 TB database in oltp mode with no/least performance problem. (yes weired but lets accept it). Now lets say the system goes down because of any problem, and I have the valid set of latest backup. I can start restoring but the problem is how to meet the objective of RTO (recovery time objective) of 3 hrs. As restoring this ~tb size of full backup will definately take time to restore.

*************************************
I know there should be some faiover configured like replication,mirroring,cluster-failover or logshipping(least preferred). Or should have some offline site where backups getting checked and restored in time.
****************************************

Lets assume because of any reason these options was not considered and the only option now is restore. What would be my strategy to bring this database online within RTO?



No option. You can't do it. Physically it will take time to read the backup file and rewrite the disk. If you haven't pre-built the log file, that has to be zero'd out. If you don't have IFI enabled, the data file also has to be zero'd out. Both take time.

I don't know what your I/O subsystem can do, but 14TB is substantial. At the very least, I'd build a cheap, whitebox system that had consumer level drives of 16-20TB. That could probably be done for $2-3k these days and it would get you running quickly if you had some sort of log shipping enabled.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
pbarbin
pbarbin
SSC Rookie
SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)SSC Rookie (28 reputation)

Group: General Forum Members
Points: 28 Visits: 349
Good timing on the article. Our company just completed our yearly DR practice test. It's not an exhaustive test, but we take some of the more critical systems and try to bring them up in a virtual network at our mirrored site (affectionately known as "the bubble").

As part of our DR strategy, we employ Sql Server Database mirroring. All web, app, and reporting servers are virtual and are replicated to our DR site.

To start the test, the network guys create the bubble and bring up domain controllers and other infrastructure needed to support the network.
Databases mirroring is broken. Mirrored Sql servers are brought into the bubble and renamed to match production server names.
Sql Server allows you to rename the server and instance easily with the sp_addserver, sp_dropserver commands. We then recover the mirrored dbs and have to setup
linked servers and verify all logins and users have the correct passwords and permissions. All of this is scripted prior to the mirror break to minimize our virtual downtime.
We hand over the environment to a small team of testers with test scripts and they go to work.

While we didn't plan perfectly, overall, the test was a success. We learn valuable lessons each time we run the test. Interestingly, many difficulties are due to the fact that we are simulating a disaster. But experiences tells us that in a real disaster there will be many other problems that we didn't forsee. This keeps us honest with ourselves that bringing up a production environment in a real disaster won't be as easy as it looks on paper.

Paul
crazy4sql
crazy4sql
SSC Eights!
SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)SSC Eights! (929 reputation)

Group: General Forum Members
Points: 929 Visits: 4422


No option. You can't do it. Physically it will take time to read the backup file and rewrite the disk. If you haven't pre-built the log file, that has to be zero'd out. If you don't have IFI enabled, the data file also has to be zero'd out. Both take time.

I don't know what your I/O subsystem can do, but 14TB is substantial. At the very least, I'd build a cheap, whitebox system that had consumer level drives of 16-20TB. That could probably be done for $2-3k these days and it would get you running quickly if you had some sort of log shipping enabled.


thanks Steve for the reply and suggestion. Just one doubt, can we consider "restoring with partial option" ?

----------
Ashish
Steve Jones
Steve Jones
SSC-Dedicated
SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)SSC-Dedicated (36K reputation)

Group: Administrators
Points: 36120 Visits: 18744
You can restore a filegroup, which might help if you have data split by filegroups. Perhaps current data in one, archived in another(s). That might allow you to restore one group and be running.

Be aware, however, that if you put indexes on one and data in another filegroup, if you restore one, the app may not work well.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Rudy Panigas
Rudy Panigas
SSC-Addicted
SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)SSC-Addicted (440 reputation)

Group: General Forum Members
Points: 440 Visits: 1303
Hello Everyone,

Here are several observations I have notice over a period of 10+ years working on disaster recovery.

1) Most people have no idea of how their SQL production servers are setup. Like port numbers, Link Connections, sp_configure settings, etc. Documentation is very poor in 90% of the cases I've worked on. I have created a script that helps in this area and is posted on this site. Just seacsearch "SQL Server Documenter". It not perfect but a good start.

2) Backups. You won't believe how many people have never tested the restores of their databases. Backups are only as good as your restores. Lots for companies have large databases but do not replicate in any manner. Then they complain that the restores are too long.

3) Testing. You would be shocked how many companies have no DR testing. Some say "we can restore to the cloud" and yet have never tried it. If we do more DR testing then we can weed out the first two issues listed above.

4) Bare Metal builds. This seem like a lost art. With replication and HA solutions we forget that you still to know how to perform bare metal builds.

Well, that's all I have for now.

Rudy



john.w.walker
john.w.walker
SSC Journeyman
SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)SSC Journeyman (87 reputation)

Group: General Forum Members
Points: 87 Visits: 45
All of these articles this week are interesting, but they are not about disaster recovery - they are about business continuity. The difference is big, with a disaster you lose your infrastructure so you have to run off an alternate site such as SunGard or a cloud provider. Having backups and backup servers is great, but a disaster is when you have no place to restore your tape or USB drive or cloud backups, if you don't have a Disaster Recovery plan.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search