Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Data Loss or Downtime


Data Loss or Downtime

Author
Message
Jeffrey Irish
Jeffrey Irish
SSC-Enthusiastic
SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)SSC-Enthusiastic (180 reputation)

Group: General Forum Members
Points: 180 Visits: 1132
For us, any VLDB that is critical has some kind of high-availalbility option like mirroring or active-passive clustering to help stave off outages. However, no solution is perfect.

What we have started doing is warehousing data that is necessary to keep for processing, but not needed in the "day-to-day" shuffle of data.

We can conglomerate data between the OLTP and warehouse databases for reporting, when needed and we have processes that pull data out of the warehouse for day-to-day operations when needed.

One aside, how does one recover a database when it is damaged for use? How can you ensure that all of the information your application requires will be intact and has integrity?

Regards,

Irish w00t
Joe Johnson-482549
Joe Johnson-482549
SSC Veteran
SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)SSC Veteran (206 reputation)

Group: General Forum Members
Points: 206 Visits: 262
People working is first priority. Downtime in manufacturing is expensive and you only have to have one call from a VP or CEO to understand that insuring that people continue working is more important than your job. ;-)

I would tend to agree with Nakul. Basic data to continue functioning. As much information for the past x days as possible in as quick a time as possible. Continue working to restore historical data more than x days old as opportunity presents. In my case, x would probably be 14 days rather than 30, even though some reports would be amiss.
bsheets 73864
bsheets 73864
SSC Veteran
SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)SSC Veteran (281 reputation)

Group: General Forum Members
Points: 281 Visits: 57
SQLArcher (1/7/2011)
Hi,

It depends on the nature of the system. I work in a financial institute with a lot of trading; in a downtime vs. partial data loss situation business has to weigh the cost of being down until the data is recovered, risk of reputational loss and increased revenue loss against a smaller risk of reputational loss and less loss of revenue with partial data.

In most cases (in this scenario) it would be better to accept a partial data loss until it can be recovered, and get business up and running to mitigate the additional reputational and financial loss.


I also work in financial services, and I would disagree with this assessment. Having some customers log in and have missing transactions would be worse than keeping everyone offline until all the data is restored - imagine the number of support calls from customers with missing transaction data.

Also, trying to merge the missing data with a database that has had numerous changes since returning to an online state could be problematic, and could result in duplicate key issues.

Unless it was static historical data that is missing, staying offline until all is recovered would be better.
RickObsitnik
RickObsitnik
SSC Rookie
SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)SSC Rookie (35 reputation)

Group: General Forum Members
Points: 35 Visits: 232
I realized that most of the time, getting the site back up, having lookup and other types of ancillary data (like products, prices, etc), was the most important thing. Recovering other data such as older orders, was secondary.


historical data should be on separate systems and is far less critical than current transactional data such as prices and products that lead to sales systems. thereby, production transaction systems need to be very highly available. administrators should be focused on bringing these online immediately WITH good data, if that system goes down. historical data is what management uses to assist in analyzing where to go. current data is the life blood of an organization. that needs to be running all the time. management can afford to move slower than the front lines of an organization.

with this in mind, the question isn't downtime v. data loss, all systems need to be up and running. the question is what data/systems should have priority in going back online.

if it's affordable, having redundant systems and sites, eliminate this question. replicating data to warm sites makes this issue moot.
Andy Leonard
Andy Leonard
Say Hey Kid
Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)Say Hey Kid (675 reputation)

Group: General Forum Members
Points: 675 Visits: 1093
Another great question Steve!

I have to agree with those who've already responded with "It depends." That's not a cop-out, it's a valid question and each business needs to evaluate that question and the regularly review the decisions.

:{>

Andy Leonard
Data Philosopher, Enterprise Data & Analytics
T_Peters
T_Peters
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1209 Visits: 1548
I primarily support internal finance systems. Since these are not order intake or revenue generating databases I'm going to assume that management would prefer less data loss with greater downtime, but this is a large assumption. While I have my DR plan on the mechanics of how to recover the data and applications what is missing is this decision by the business of what they thing is most important. Guess I have some meetings to schedule Sad

--Tim
Eric M Russell
Eric M Russell
SSCertifiable
SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)SSCertifiable (6.3K reputation)

Group: General Forum Members
Points: 6312 Visits: 10066
It not what's important to me personally as a DBA or developer, but what's important to the business: users, clients, and management. Technically speaking, a partial restore that's still functional is not even an option unless the database is partitioned in such a way that supports it. For example, if your data is partitioned across seperate file groups, where all of the critical operational data is contained sperately from historical or non-essential reference data, then you can present the option to business with a high degree of confidence that it will actually work as expected. You don't want to end up spinning you wheels and losing valuable time that could be better spent completing a full recovery. These are good questions, and it would be worthwhile to plan ahead and think about what data would be needed to support a minimal degree of application functionality, test the theory on a QA or staging server, and then have an alternate disaster revocery plan documented.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
dma-669038
dma-669038
Old Hand
Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)Old Hand (347 reputation)

Group: General Forum Members
Points: 347 Visits: 1035
It depends strongly on business needs. I work in healthcare, we need systems up 24x7 and people's lives literally depend on them, so Kimberly's statement is a valid one for me - although again the point Gail makes about what data is missing is also important. There is no point getting a db online with critical data to run the system is missing. Also to bear in mind many times is the cost of downtime vs loss of data. It costs money to maintain backups (disk space, off site storage, technology to do the backup or compression...). It even costs money if you consider the fact that i have to run a reindexing job in simple recovery mode and during that time i will not have backups. How much is all that compared to cost of loss of data? A good DR strategy is worked out with due consideration to all these factors and putting business needs in right order.
LSCIV
LSCIV
SSC Rookie
SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)

Group: General Forum Members
Points: 31 Visits: 154
bsheets 73864 (1/7/2011)
SQLArcher (1/7/2011)
Hi,

It depends on the nature of the system. I work in a financial institute with a lot of trading; in a downtime vs. partial data loss situation business has to weigh the cost of being down until the data is recovered, risk of reputational loss and increased revenue loss against a smaller risk of reputational loss and less loss of revenue with partial data.

In most cases (in this scenario) it would be better to accept a partial data loss until it can be recovered, and get business up and running to mitigate the additional reputational and financial loss.


I also work in financial services, and I would disagree with this assessment. Having some customers log in and have missing transactions would be worse than keeping everyone offline until all the data is restored - imagine the number of support calls from customers with missing transaction data.

Also, trying to merge the missing data with a database that has had numerous changes since returning to an online state could be problematic, and could result in duplicate key issues.

Unless it was static historical data that is missing, staying offline until all is recovered would be better.


I think it would depend on whether or not the system is customer facing and revenue generating. If it is, I can't imagine any business-minded executive choosing to forego future revenue so the system can come up clean. Sure it would be nice if it did, but if it didn't that's fine too b/c it can be fixed. That's what we as DBA's get paid to do. I don't think anyone will be concerned about the number of support calls either. That's what CS gets paid to do and it's not revenue impacting so no biggie. In the end, like most business decisions, it's about the almighty dollar. Maximizing revenue and minimizing loss. The amount of work it causes you or I or customer service is irrelevant. For some service oriented companies there are SLA's that state for any and all downtime the service provider will share in the revenue loss with their customers. I worked for a financial company that had such a clause. As expected, uptime was their highest priority :-).
Revenant
Revenant
SSCertifiable
SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)SSCertifiable (6.1K reputation)

Group: General Forum Members
Points: 6145 Visits: 4769
Oh well, I feel I am getting old.

As several times before, I need to point out that the decison what is more important (to be up ASAP or to be exactly up to date) is not up to the IT staff, DBAs included. It depends on the industry and legal reponsibility of the company or division, so ultimately (and unfortunately) it is the lawyers who have the last word, like it or not. (I don't.)

In my case, the picture is fortunately simple: I am on a BI project, and for my team data off by a day or two is no biggie, it is not accounting or CRM, so as long as we present a reliable picture of the business, we are OK.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search