Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 123»»»

Disaster Recovery Week Expand / Collapse
Author
Message
Posted Sunday, December 9, 2012 3:43 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 4:02 PM
Points: 33,062, Visits: 15,176
Comments posted to this topic are about the item Disaster Recovery Week






Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1394390
Posted Monday, December 10, 2012 12:49 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: 2 days ago @ 12:40 AM
Points: 7,001, Visits: 8,439
just a very primitive tip: Start small !

Start building your disaster recovery project with a very small scope, not that critical to business activity.

Only read the headers of the articles / BP / books you find about Disaster Recovery, so you know the topics to keep in mind and are not overwhelmed by the tiny details behind it.

That is for later on, after you review your first, second, third cycle of building your general DRP.

Pick any non-critical database and start from there on.


Once you are comfortable with the elementary concepts, and their impact to the topic from your point of view, make sure your project gets sponsored by your superiour(s), because at certain point in time, you will have to contact business responsibles to discuss your findings and align them budget wise.


There is more to disaster recovery than SQLServer, but it makes a good starting point.


Johan


Don't drive faster than your guardian angel can fly ...
but keeping both feet on the ground won't get you anywhere

- How to post Performance Problems
- How to post data/code to get the best help


- How to prevent a sore throat after hours of presenting ppt ?


"press F1 for solution", "press shift+F1 for urgent solution"


Need a bit of Powershell? How about this

Who am I ? Sometimes this is me but most of the time this is me
Post #1394471
Posted Monday, December 10, 2012 7:50 AM


Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, January 3, 2013 8:12 AM
Points: 9, Visits: 18
Backups have only given me grief. Its restores that count & make me look good.

However I have gotten burned with "level of expectation"

1) A dual RAID-5 SAN died while I was on vacation. It was in the cleanest part of the city garage but too many disks failed. I came in, restored the database and maintanece jobs to another server in under six hours. I was proud - my boss was livid.
Why did it take six hours to restore and reconfigure an 18 gig database? He thought it should only take twenty min and the process be documented so anyone could do it.
Lesson learned - have a system up and ready.

2) We have several databases where we only keep six months or less of backups. I like to keep three years. Why? Problems do not always surface in six months. Year end processing has found issues with data from a year ago.

3) Would I use the cloud? Not even in an emergency - only a catastrophy. Why? (laughs) our internet connection is one of the things to go. Its not under our control - but the data center is. Those that can work while others are down make the money and keep their jobs.


Post #1394631
Posted Monday, December 10, 2012 8:22 AM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Sunday, July 20, 2014 10:36 PM
Points: 880, Visits: 4,081
I have a question on one of the DR scenario,
Lets say I have 14 TB database in oltp mode with no/least performance problem. (yes weired but lets accept it). Now lets say the system goes down because of any problem, and I have the valid set of latest backup. I can start restoring but the problem is how to meet the objective of RTO (recovery time objective) of 3 hrs. As restoring this ~tb size of full backup will definately take time to restore.

*************************************
I know there should be some faiover configured like replication,mirroring,cluster-failover or logshipping(least preferred). Or should have some offline site where backups getting checked and restored in time.
****************************************

Lets assume because of any reason these options was not considered and the only option now is restore. What would be my strategy to bring this database online within RTO?



----------
Ashish
Post #1394647
Posted Monday, December 10, 2012 8:31 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 4:02 PM
Points: 33,062, Visits: 15,176
crazy4sql (12/10/2012)
I have a question on one of the DR scenario,
Lets say I have 14 TB database in oltp mode with no/least performance problem. (yes weired but lets accept it). Now lets say the system goes down because of any problem, and I have the valid set of latest backup. I can start restoring but the problem is how to meet the objective of RTO (recovery time objective) of 3 hrs. As restoring this ~tb size of full backup will definately take time to restore.

*************************************
I know there should be some faiover configured like replication,mirroring,cluster-failover or logshipping(least preferred). Or should have some offline site where backups getting checked and restored in time.
****************************************

Lets assume because of any reason these options was not considered and the only option now is restore. What would be my strategy to bring this database online within RTO?



No option. You can't do it. Physically it will take time to read the backup file and rewrite the disk. If you haven't pre-built the log file, that has to be zero'd out. If you don't have IFI enabled, the data file also has to be zero'd out. Both take time.

I don't know what your I/O subsystem can do, but 14TB is substantial. At the very least, I'd build a cheap, whitebox system that had consumer level drives of 16-20TB. That could probably be done for $2-3k these days and it would get you running quickly if you had some sort of log shipping enabled.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1394654
Posted Monday, December 10, 2012 8:34 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, December 13, 2012 7:23 AM
Points: 22, Visits: 349
Good timing on the article. Our company just completed our yearly DR practice test. It's not an exhaustive test, but we take some of the more critical systems and try to bring them up in a virtual network at our mirrored site (affectionately known as "the bubble").

As part of our DR strategy, we employ Sql Server Database mirroring. All web, app, and reporting servers are virtual and are replicated to our DR site.

To start the test, the network guys create the bubble and bring up domain controllers and other infrastructure needed to support the network.
Databases mirroring is broken. Mirrored Sql servers are brought into the bubble and renamed to match production server names.
Sql Server allows you to rename the server and instance easily with the sp_addserver, sp_dropserver commands. We then recover the mirrored dbs and have to setup
linked servers and verify all logins and users have the correct passwords and permissions. All of this is scripted prior to the mirror break to minimize our virtual downtime.
We hand over the environment to a small team of testers with test scripts and they go to work.

While we didn't plan perfectly, overall, the test was a success. We learn valuable lessons each time we run the test. Interestingly, many difficulties are due to the fact that we are simulating a disaster. But experiences tells us that in a real disaster there will be many other problems that we didn't forsee. This keeps us honest with ourselves that bringing up a production environment in a real disaster won't be as easy as it looks on paper.

Paul
Post #1394655
Posted Monday, December 10, 2012 9:18 AM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Sunday, July 20, 2014 10:36 PM
Points: 880, Visits: 4,081


No option. You can't do it. Physically it will take time to read the backup file and rewrite the disk. If you haven't pre-built the log file, that has to be zero'd out. If you don't have IFI enabled, the data file also has to be zero'd out. Both take time.

I don't know what your I/O subsystem can do, but 14TB is substantial. At the very least, I'd build a cheap, whitebox system that had consumer level drives of 16-20TB. That could probably be done for $2-3k these days and it would get you running quickly if you had some sort of log shipping enabled.


thanks Steve for the reply and suggestion. Just one doubt, can we consider "restoring with partial option" ?


----------
Ashish
Post #1394673
Posted Monday, December 10, 2012 9:36 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 4:02 PM
Points: 33,062, Visits: 15,176
You can restore a filegroup, which might help if you have data split by filegroups. Perhaps current data in one, archived in another(s). That might allow you to restore one group and be running.

Be aware, however, that if you put indexes on one and data in another filegroup, if you restore one, the app may not work well.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1394680
Posted Tuesday, December 11, 2012 8:36 AM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Friday, July 18, 2014 12:37 PM
Points: 312, Visits: 1,098
Hello Everyone,

Here are several observations I have notice over a period of 10+ years working on disaster recovery.

1) Most people have no idea of how their SQL production servers are setup. Like port numbers, Link Connections, sp_configure settings, etc. Documentation is very poor in 90% of the cases I've worked on. I have created a script that helps in this area and is posted on this site. Just seacsearch "SQL Server Documenter". It not perfect but a good start.

2) Backups. You won't believe how many people have never tested the restores of their databases. Backups are only as good as your restores. Lots for companies have large databases but do not replicate in any manner. Then they complain that the restores are too long.

3) Testing. You would be shocked how many companies have no DR testing. Some say "we can restore to the cloud" and yet have never tried it. If we do more DR testing then we can weed out the first two issues listed above.

4) Bare Metal builds. This seem like a lost art. With replication and HA solutions we forget that you still to know how to perform bare metal builds.

Well, that's all I have for now.

Rudy



Post #1395156
Posted Thursday, December 13, 2012 7:39 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Friday, January 3, 2014 8:39 AM
Points: 74, Visits: 37
All of these articles this week are interesting, but they are not about disaster recovery - they are about business continuity. The difference is big, with a disaster you lose your infrastructure so you have to run off an alternate site such as SunGard or a cloud provider. Having backups and backup servers is great, but a disaster is when you have no place to restore your tape or USB drive or cloud backups, if you don't have a Disaster Recovery plan.
Post #1396192
« Prev Topic | Next Topic »

Add to briefcase 123»»»

Permissions Expand / Collapse