Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase ««123»»

Data Loss or Downtime Expand / Collapse
Author
Message
Posted Friday, January 7, 2011 7:00 AM


SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: 2 days ago @ 1:55 PM
Points: 149, Visits: 1,027
For us, any VLDB that is critical has some kind of high-availalbility option like mirroring or active-passive clustering to help stave off outages. However, no solution is perfect.

What we have started doing is warehousing data that is necessary to keep for processing, but not needed in the "day-to-day" shuffle of data.

We can conglomerate data between the OLTP and warehouse databases for reporting, when needed and we have processes that pull data out of the warehouse for day-to-day operations when needed.

One aside, how does one recover a database when it is damaged for use? How can you ensure that all of the information your application requires will be intact and has integrity?


Regards,

Irish
Post #1044381
Posted Friday, January 7, 2011 7:11 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, February 21, 2014 8:14 AM
Points: 194, Visits: 255
People working is first priority. Downtime in manufacturing is expensive and you only have to have one call from a VP or CEO to understand that insuring that people continue working is more important than your job.

I would tend to agree with Nakul. Basic data to continue functioning. As much information for the past x days as possible in as quick a time as possible. Continue working to restore historical data more than x days old as opportunity presents. In my case, x would probably be 14 days rather than 30, even though some reports would be amiss.


Post #1044393
Posted Friday, January 7, 2011 8:45 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Friday, March 4, 2011 9:15 AM
Points: 281, Visits: 57
SQLArcher (1/7/2011)
Hi,

It depends on the nature of the system. I work in a financial institute with a lot of trading; in a downtime vs. partial data loss situation business has to weigh the cost of being down until the data is recovered, risk of reputational loss and increased revenue loss against a smaller risk of reputational loss and less loss of revenue with partial data.

In most cases (in this scenario) it would be better to accept a partial data loss until it can be recovered, and get business up and running to mitigate the additional reputational and financial loss.


I also work in financial services, and I would disagree with this assessment. Having some customers log in and have missing transactions would be worse than keeping everyone offline until all the data is restored - imagine the number of support calls from customers with missing transaction data.

Also, trying to merge the missing data with a database that has had numerous changes since returning to an online state could be problematic, and could result in duplicate key issues.

Unless it was static historical data that is missing, staying offline until all is recovered would be better.
Post #1044466
Posted Friday, January 7, 2011 9:06 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Tuesday, August 19, 2014 3:29 PM
Points: 25, Visits: 199
I realized that most of the time, getting the site back up, having lookup and other types of ancillary data (like products, prices, etc), was the most important thing. Recovering other data such as older orders, was secondary.


historical data should be on separate systems and is far less critical than current transactional data such as prices and products that lead to sales systems. thereby, production transaction systems need to be very highly available. administrators should be focused on bringing these online immediately WITH good data, if that system goes down. historical data is what management uses to assist in analyzing where to go. current data is the life blood of an organization. that needs to be running all the time. management can afford to move slower than the front lines of an organization.

with this in mind, the question isn't downtime v. data loss, all systems need to be up and running. the question is what data/systems should have priority in going back online.

if it's affordable, having redundant systems and sites, eliminate this question. replicating data to warm sites makes this issue moot.
Post #1044480
Posted Friday, January 7, 2011 9:41 AM


Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, July 21, 2014 12:44 PM
Points: 389, Visits: 1,041
Another great question Steve!

I have to agree with those who've already responded with "It depends." That's not a cop-out, it's a valid question and each business needs to evaluate that question and the regularly review the decisions.

:{>


Andy Leonard
CSO, Linchpin People
Follow me on Twitter: @AndyLeonard
Post #1044516
Posted Friday, January 7, 2011 9:49 AM


SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Yesterday @ 10:24 AM
Points: 868, Visits: 1,278
I primarily support internal finance systems. Since these are not order intake or revenue generating databases I'm going to assume that management would prefer less data loss with greater downtime, but this is a large assumption. While I have my DR plan on the mechanics of how to recover the data and applications what is missing is this decision by the business of what they thing is most important. Guess I have some meetings to schedule

--Tim
Post #1044522
Posted Friday, January 7, 2011 11:21 AM


SSCommitted

SSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommittedSSCommitted

Group: General Forum Members
Last Login: Yesterday @ 1:46 PM
Points: 1,649, Visits: 4,697
It not what's important to me personally as a DBA or developer, but what's important to the business: users, clients, and management. Technically speaking, a partial restore that's still functional is not even an option unless the database is partitioned in such a way that supports it. For example, if your data is partitioned across seperate file groups, where all of the critical operational data is contained sperately from historical or non-essential reference data, then you can present the option to business with a high degree of confidence that it will actually work as expected. You don't want to end up spinning you wheels and losing valuable time that could be better spent completing a full recovery. These are good questions, and it would be worthwhile to plan ahead and think about what data would be needed to support a minimal degree of application functionality, test the theory on a QA or staging server, and then have an alternate disaster revocery plan documented.
Post #1044579
Posted Friday, January 7, 2011 11:55 AM
Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, May 19, 2014 9:26 AM
Points: 312, Visits: 1,028
It depends strongly on business needs. I work in healthcare, we need systems up 24x7 and people's lives literally depend on them, so Kimberly's statement is a valid one for me - although again the point Gail makes about what data is missing is also important. There is no point getting a db online with critical data to run the system is missing. Also to bear in mind many times is the cost of downtime vs loss of data. It costs money to maintain backups (disk space, off site storage, technology to do the backup or compression...). It even costs money if you consider the fact that i have to run a reindexing job in simple recovery mode and during that time i will not have backups. How much is all that compared to cost of loss of data? A good DR strategy is worked out with due consideration to all these factors and putting business needs in right order.
Post #1044608
Posted Friday, January 7, 2011 12:28 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Friday, May 16, 2014 9:38 AM
Points: 22, Visits: 148
bsheets 73864 (1/7/2011)
SQLArcher (1/7/2011)
Hi,

It depends on the nature of the system. I work in a financial institute with a lot of trading; in a downtime vs. partial data loss situation business has to weigh the cost of being down until the data is recovered, risk of reputational loss and increased revenue loss against a smaller risk of reputational loss and less loss of revenue with partial data.

In most cases (in this scenario) it would be better to accept a partial data loss until it can be recovered, and get business up and running to mitigate the additional reputational and financial loss.


I also work in financial services, and I would disagree with this assessment. Having some customers log in and have missing transactions would be worse than keeping everyone offline until all the data is restored - imagine the number of support calls from customers with missing transaction data.

Also, trying to merge the missing data with a database that has had numerous changes since returning to an online state could be problematic, and could result in duplicate key issues.

Unless it was static historical data that is missing, staying offline until all is recovered would be better.


I think it would depend on whether or not the system is customer facing and revenue generating. If it is, I can't imagine any business-minded executive choosing to forego future revenue so the system can come up clean. Sure it would be nice if it did, but if it didn't that's fine too b/c it can be fixed. That's what we as DBA's get paid to do. I don't think anyone will be concerned about the number of support calls either. That's what CS gets paid to do and it's not revenue impacting so no biggie. In the end, like most business decisions, it's about the almighty dollar. Maximizing revenue and minimizing loss. The amount of work it causes you or I or customer service is irrelevant. For some service oriented companies there are SLA's that state for any and all downtime the service provider will share in the revenue loss with their customers. I worked for a financial company that had such a clause. As expected, uptime was their highest priority .
Post #1044635
Posted Friday, January 7, 2011 1:02 PM


SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Yesterday @ 8:50 AM
Points: 4,425, Visits: 3,417
Oh well, I feel I am getting old.

As several times before, I need to point out that the decison what is more important (to be up ASAP or to be exactly up to date) is not up to the IT staff, DBAs included. It depends on the industry and legal reponsibility of the company or division, so ultimately (and unfortunately) it is the lawyers who have the last word, like it or not. (I don't.)

In my case, the picture is fortunately simple: I am on a BI project, and for my team data off by a day or two is no biggie, it is not accounting or CRM, so as long as we present a reliable picture of the business, we are OK.
Post #1044664
« Prev Topic | Next Topic »

Add to briefcase ««123»»

Permissions Expand / Collapse