SQLServerCentral / Editorials / SQLServerCentral.com / The Chance of Failure / Latest Posts

RE: The Chance of Failure

Eric M Russell — Fri, 01 Apr 2011 13:44:13 GMT

[quote][b]Steve Jones - SSC Editor (4/1/2011)[/b][hr][quote][b]H-122523 (4/1/2011)[/b][hr]Hi Steve,Interesting editorial, this, as it connects failure to probability. In the end, we all make up the statistics; I just finished 'The Black Swan' (not about the movie!) by Nassim Taleb.It is worth to check this out, as it shows a refreshing view on how to look at probability as we like to use the concept in models, versus how probability touches us in the real world.regards,Hans[/quote]Thanks and I'll check it out. My wife wants to see it, so I have an excuse to rent it.It's not that you worry about the probability of failure, but assessing that over time, or in a large environment during a relatively short time, failure is inevitable.[/quote]In addition to the size of the operation and length of time, other factors like the newness of the technology or the number of moving parts can increase the probability of failure. More generally, people or corporations who lead the pack, those who take more risks, have a higher probability of failure, although the same can compensate for it by having a talent for recovering from failure too.

RE: The Chance of Failure

Michael Valentine Jones — Fri, 01 Apr 2011 11:36:29 GMT

[quote][b]Steve Jones - SSC Editor (4/1/2011)[/b][hr][quote][b]Michael Valentine Jones (3/31/2011)[/b][hr]Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.[/quote]ROFL, that is outstanding! :-DCan I use that quote?[/quote]Yes, be my guest.

RE: The Chance of Failure

Steve Jones - SSC Editor — Fri, 01 Apr 2011 11:12:06 GMT

[quote][b]H-122523 (4/1/2011)[/b][hr]Hi Steve,Interesting editorial, this, as it connects failure to probability. In the end, we all make up the statistics; I just finished 'The Black Swan' (not about the movie!) by Nassim Taleb.It is worth to check this out, as it shows a refreshing view on how to look at probability as we like to use the concept in models, versus how probability touches us in the real world.regards,Hans[/quote]Thanks and I'll check it out. My wife wants to see it, so I have an excuse to rent it.It's not that you worry about the probability of failure, but assessing that over time, or in a large environment during a relatively short time, failure is inevitable.

RE: The Chance of Failure

webrunner — Fri, 01 Apr 2011 09:32:58 GMT

RE: The Chance of Failure

H-122523 — Fri, 01 Apr 2011 09:27:41 GMT

Hi Steve,Interesting editorial, this, as it connects failure to probability. In the end, we all make up the statistics; I just finished 'The Black Swan' (not about the movie!) by Nassim Taleb.It is worth to check this out, as it shows a refreshing view on how to look at probability as we like to use the concept in models, versus how probability touches us in the real world.regards,Hans

RE: The Chance of Failure

TravisDBA — Fri, 01 Apr 2011 09:23:06 GMT

[quote][b]Steve Jones - SSC Editor (4/1/2011)[/b][hr][quote][b]TravisDBA (3/31/2011)[/b][hr]I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-D[/quote]I completely agree with this. All those "department servers" often run without backups. What's even worse is I've seen lots of software places, building commercial software, where they don't backup the dev servers. That's their manufacturing plant, and I've seen some of them lose a decent amount of work because of it.[/quote]Exactly, also good point on the development backups as well.. I saw a case once where development lost a complete block change. Heads rolled on that one. The reason I survived it was I had an email recommending dev backups to the management, but it was considered as not needed because they could not afford to pop for the disk space for dev in their budget at the time. They sure changed their tune on that quick. It only has to happen ONCE (they never really get that), and remember, e-mail is the best CYA in the political dog-eat-dog world of IT. :-D

RE: The Chance of Failure

Steve Jones - SSC Editor — Fri, 01 Apr 2011 09:10:02 GMT

[quote][b]TravisDBA (3/31/2011)[/b][hr]I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-D[/quote]I completely agree with this. All those "department servers" often run without backups. What's even worse is I've seen lots of software places, building commercial software, where they don't backup the dev servers. That's their manufacturing plant, and I've seen some of them lose a decent amount of work because of it.

RE: The Chance of Failure

Steve Jones - SSC Editor — Fri, 01 Apr 2011 09:08:15 GMT

[quote][b]Michael Valentine Jones (3/31/2011)[/b][hr]Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.[/quote]ROFL, that is outstanding! :-DCan I use that quote?

RE: The Chance of Failure

Steve Jones - SSC Editor — Fri, 01 Apr 2011 09:07:04 GMT

[quote][b]Craig Farrell (3/31/2011)[/b][hr][quote][b]Tom Bakerman (3/31/2011)[/b][hr]This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.[/quote]I'm now picturing some poor kid wandering the halls of the data center with a giant shopping cart of new drives in front of him, just slowly meandering down the aisles looking for red lights with this zombie-fied look on his face at 3 in the morning.[/quote]It's not that bad, but I know a large company, with quite a few SANs, and replacing disk drives is a full time job. It's not 24x7, and they have lots of hot spares to bring online automatically, but someone's job every day is to order drives, unpack/pack up drives, and walk around replacing them.

RE: The Chance of Failure

webrunner — Fri, 01 Apr 2011 07:23:21 GMT

[quote][b]Keith Langmead (4/1/2011)[/b][hr]I think most IT people know they should be planning for failure, but until they have their first one it can hard to really take it seriously and act accordingly. While I obviously didn't think so at the time, looking back I'm glad I had a major failure resulting in substantial data loss fairly early in my career, since that really helped focus the mind! These days I always assume the worst and plan for it.What you do though at the end of the day is down to assessing the risk and the cost of the potential outcome. For instance, where I work we have some systems that are reasonably critical, they have decent hardware, lots of monitoring, hardware raid, and we maintain a complete set of spare parts just in case. We know the instant something is wrong and can hopefully repair it in a minimal amount of time. We have other systems that just do various non-critical tasks that run on old hardware with no spares, minimal monitoring etc. If they die then it's a pain, but if they're offline for a few days while we build a replacement there's no big issue, and the additional cost of using better kit wouldn't make economic sense.I agree with Travis though, so many of our clients don't think backups are important (until they lose something of course), and we even had one who literally told us they didn't need backups as if the data was lost they'd just re-type the documentation again! And yes, we did enjoy quoting it back to him when they did lose some data and couldn't recover it.[/quote]Yes, this is all too true. I think there are even some buzzwords for it -- normalcy bias, willful blindness, etc. There definitely seems to be a part of the human brain that does not want to think about the possibility of disaster, and it takes very hard work to overcome it. I know for me it's a daily battle. But I try to learn as much as I can about best practices and tell myself they will pay off in the end.- webrunner

RE: The Chance of Failure

Keith Langmead — Fri, 01 Apr 2011 05:27:18 GMT

I think most IT people know they should be planning for failure, but until they have their first one it can hard to really take it seriously and act accordingly. While I obviously didn't think so at the time, looking back I'm glad I had a major failure resulting in substantial data loss fairly early in my career, since that really helped focus the mind! These days I always assume the worst and plan for it.What you do though at the end of the day is down to assessing the risk and the cost of the potential outcome. For instance, where I work we have some systems that are reasonably critical, they have decent hardware, lots of monitoring, hardware raid, and we maintain a complete set of spare parts just in case. We know the instant something is wrong and can hopefully repair it in a minimal amount of time. We have other systems that just do various non-critical tasks that run on old hardware with no spares, minimal monitoring etc. If they die then it's a pain, but if they're offline for a few days while we build a replacement there's no big issue, and the additional cost of using better kit wouldn't make economic sense.I agree with Travis though, so many of our clients don't think backups are important (until they lose something of course), and we even had one who literally told us they didn't need backups as if the data was lost they'd just re-type the documentation again! And yes, we did enjoy quoting it back to him when they did lose some data and couldn't recover it.

RE: The Chance of Failure

Evil Kraig F — Thu, 31 Mar 2011 17:42:56 GMT

[quote][b]James Stover (3/31/2011)[/b][hr]Fortunately the Cloud will save us from failure. Because we know how bullet-proof telecom infrastructure is. It never fails...except when it storms, or during peak solar activity, or when a fiber-optic cable is dredged, or the provider goes bankrupt, or during DDoS attacks, or...:-D[/quote]No, the true cloud will be the 'magic mesh network' made up by the billions of cell phones that finally have enough speed to communicate everything everywhere without needing ISPs...... oh, wait, that was Shadowrun...

RE: The Chance of Failure

James Stover — Thu, 31 Mar 2011 17:35:15 GMT

Fortunately the Cloud will save us from failure. Because we know how bullet-proof telecom infrastructure is. It never fails...except when it storms, or during peak solar activity, or when a fiber-optic cable is dredged, or the provider goes bankrupt, or during DDoS attacks, or...:-D

RE: The Chance of Failure

Michael Valentine Jones — Thu, 31 Mar 2011 15:41:03 GMT

Don't think of it as a failure.Think of it as a chance to start over without the burden of all that legacy data.

RE: The Chance of Failure

Tom Bakerman — Thu, 31 Mar 2011 15:29:16 GMT

[quote][b]Craig Farrell (3/31/2011)[/b][hr][quote][b]Tom Bakerman (3/31/2011)[/b][hr]This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.[/quote]I'm now picturing some poor kid wandering the halls of the data center with a giant shopping cart of new drives in front of him, just slowly meandering down the aisles looking for red lights with this zombie-fied look on his face at 3 in the morning.[/quote]Hehe. But remember, too, that those were the days of the washing machine sized disks. :-PAnother memory I have from that discussion is that the average length of experience of employees was on the order of 10 years.

RE: The Chance of Failure

TravisDBA — Thu, 31 Mar 2011 15:28:07 GMT

[quote][b]TravisDBA (3/31/2011)[/b][hr][quote][b]Eric M Russell (3/31/2011)[/b][hr]Disaster recovery is actually easier in the modern IT world than it was in times past. One hundred years ago, if the county courthouse burnt to the ground, much of the archived documents would be lost forever with no backup copy.[/quote]It might be easier as you say, but you would be surprised at how many shops do not plan for it by not even just backing up their databases up on a regular basis. It would astound you. I have seen shops I have gone into in the past that have not backed up their system databases in over a year!!! . When I asked why? the response was "Well we didn't need to, everything works just fine and if it isn't btoke we don't fix it... " :w00t: Absolutely incredible, there are bozos out there in IT like this but there are. . Just because DR has gotten easier doesn't mean people are doing it. :-D[/quote]I was not just referring to small mom and pop shops, I was referring to large geographically distributed IT enterprises as well, and it does happens more than you might realize.:-D

RE: The Chance of Failure

Evil Kraig F — Thu, 31 Mar 2011 15:25:30 GMT

[quote][b]Tom Bakerman (3/31/2011)[/b][hr]This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.[/quote]I'm now picturing some poor kid wandering the halls of the data center with a giant shopping cart of new drives in front of him, just slowly meandering down the aisles looking for red lights with this zombie-fied look on his face at 3 in the morning.

RE: The Chance of Failure

Tom Bakerman — Thu, 31 Mar 2011 15:17:36 GMT

This seems like rehashing old news. I have vague memories from a database class I took in the 80s where we talked about an airlines data center (AA or United or somebody big like that). Like I said, my memory is vague on this, but I believe they talked about Mean Time Between Failure of the disk farm on the order of 5 minutes (translation: there will be a disk failure about every 5 minutes).The technology has changed, but the problems are still there and have to be dealt with.

RE: The Chance of Failure

Eric M Russell — Thu, 31 Mar 2011 10:09:42 GMT

If Joe's Bike Shop looses all their data without a backup, then that's a personal tragedy for their business, but it's not really a community or regional wide disaster. The chances of a government office losing all your tax records or a corporation permanently losing all your mortgage paperwork is practically unheard of. Well... the mortgage company may sit on your escrow account refund for weeks or months claiming they "misplaced the paperwork", but they can find the records at any point, if they really wanted to. That's not an information technology issue.

RE: The Chance of Failure

TravisDBA — Thu, 31 Mar 2011 09:51:39 GMT

[quote][b]Eric M Russell (3/31/2011)[/b][hr]Disaster recovery is actually easier in the modern IT world than it was in times past. One hundred years ago, if the county courthouse burnt to the ground, much of the archived documents would be lost forever with no backup copy.[/quote]It might be easier as you say, but you would be surprised at how many shops do not plan for it by not even just backing up their databases up on a regular basis. It would astound you. I have seen shops I have gone into in the past that have not backed up their system databases in over a year!!! . When I asked why? the response was "Well we didn't need to, everything works just fine and if it isn't btoke we don't fix it... " :w00t: Absolutely incredible, there are bozos out there in IT like this but there are. . Just because DR has gotten easier doesn't mean people are doing it. :-D

RE: The Chance of Failure

jay-h — Thu, 31 Mar 2011 08:34:29 GMT

[quote][b]Nadrek (3/31/2011)[/b][hr]For those with RAID; how many actually run consistency checks on a regular basis, to detect single drive corruption?...I would also note that planning for regular failure is both very expensive and very limiting; most products don't support truly transparent high availability with 0% downtime at all. Big mainframe hardware (and perhaps midrange systems) does; commodity x86 based hardware and software typically doesn't, with a few exceptions.[/quote]0% downtime is extremely expensive. One needs to be realistic. Most apps can tolerate occasional downtimes of varying degrees, and it's a lot less expensive to evaluate those needs realistically.Interestingly, we used to have a clustered RAID SQL2000 installation. Every system failure we had was in the RAID control system which meant that the clustering did us no good whatsoever in the downtime area. (Fortunately the RAIDs did not lose data during those failures). We have a mirrored system now on completely separate hardware.

RE: The Chance of Failure

GAF — Thu, 31 Mar 2011 08:17:19 GMT

Left out the the 3rd rule...Hope for the bestPlan/Prepare for the worstAlways, always, always have a plan B!!!

RE: The Chance of Failure

Eric M Russell — Thu, 31 Mar 2011 08:07:59 GMT

Disaster recovery is actually easier in the modern IT world than it was in times past. One hundred years ago, if the county courthouse burnt to the ground, much of the archived documents would be lost forever with no backup copy.

RE: The Chance of Failure

Nadrek — Thu, 31 Mar 2011 07:59:01 GMT

For those with RAID; how many actually run consistency checks on a regular basis, to detect single drive corruption?For single drives, how often is an online disk check (much less an offline disk check) done?For data, how many people actually generate checksum data (if you use Zip, raise your hand) to detect some corruption*, much less ECC data (par2 or ICE or DVDisaster, etc.) to detect and correct some corruption*?*I almost never see real corruption; the typical "I can't explain this issue [optional: but a reinstall fixed it]; some file must have gone corrupt" that I actually check out shows no corruption at all; more commonly is a configuration change of some stripe.I would also note that planning for regular failure is both very expensive and very limiting; most products don't support truly transparent high availability with 0% downtime at all. Big mainframe hardware (and perhaps midrange systems) does; commodity x86 based hardware and software typically doesn't, with a few exceptions.

RE: The Chance of Failure

webrunner — Thu, 31 Mar 2011 07:40:36 GMT

Thanks for the good reminder, Steve, and the links.This may be covered in the links (haven't read them yet), but I recall reading a similar topic where it mentioned that large companies such as Google invest in a huge amount of what they call commodity hardware -- decent but not top of the line equipment that is designed to be replaced when it fails, rather than a system where all the eggs are in one basket of a single high-end setup. Still requires a large budget, but it does make sense from a disaster mitigation point of view.Also -- and I am by no means a math expert, so please excuse any errors -- my understanding of the failure rates of systems is that the chance of a failure on any given day for modern equipment may be quite small, but the chance of failure over a longer time is almost certain. So it's only a matter of time before parts fail. I guess it's only a short step from that realization to the realization that if you don't really know when the part will fail, you always need to be ready for it to fail and have a backup plan to handle it.Just my two cents....- webrunner

RE: The Chance of Failure

jay-h — Thu, 31 Mar 2011 06:55:36 GMT

Google has the right idea. Failure WILL happen and the issue is to design around it. Their approach is resilienceUnfortunately this reality has not fully found its way into the IT world. Failure should be considered an unfortunate byproduct of system operation and handled accordingly.

RE: The Chance of Failure

blandry — Thu, 31 Mar 2011 06:15:59 GMT

"Hope for the best, plan for the worst..."Generally good advice if you are talking about sizable companies, data centers and the like, but in one's personal life (if you are a techie) this can get out of hand.I have always had a network in my home going as far back as the early 80's, and I have always gone to great lengths backing up lots of data. Now, some 30 years later, I have multiple external drives and two very old machines that I use solely for backing things up. Problem is, I am still (to this day) backing up some stuff that must be close to 30 years old and though I constantly promise myself that someday I will review what I am backing up, I never seem to get the energy to go through that task - and instead, I just buy yet another big external drive and keep backing up.Why do I still save program code from languages now dead? Why do I continue to hold onto Windows fonts that date back (I think) to Windows 3.11? I have close to 20,000 digital images and I still back them up, seemingly sure that one day I will go through them and actually throw away the pictures I inadvertently took of my own feet.Sure, hope for the best, plan for the worst... But sometimes the worst is self-inflicted, and while I have spent decades ensuring that my "main machines" are clean and up to date, I now have more space occupied in backups than I do in actively used data. And why didn't I get rid of stuff as the years passed? Well, though I am fairly sure that DOS-based Ryan/McFarland COBOL is not going to make any comeback in these days - if it does (!!!), I will be ready with the programs I wrote for it back when I didn't have grey hair and wasn't a member in good standing of AARP...

RE: The Chance of Failure

SALIM ALI — Thu, 31 Mar 2011 05:36:19 GMT

I think Dinosaur comics summed up failure best with [b] Failure: it's just success rounded down[/b]which is a healthy way to look at it.

RE: The Chance of Failure

Raju Lalvani — Thu, 31 Mar 2011 01:31:27 GMT

Large disasters are one thing, but programmers are a lot of optimists who rarely check for error conditions as trivial as the followingIf Flag = '1'do.....else /* assume flag is '2' */....end ifRather thanIf Flag = '1'do.....elseif Flag = '2' ....Else Error in Program ...end if

RE: The Chance of Failure

martin catherall — Wed, 30 Mar 2011 22:35:44 GMT

Good article Steve.I live in Christchurch, New Zealand and I have to admit that I thought the possibility of losing an entire center here was extremely remote. However recent events (two major earthquakes in 6 months) have seen this risk realized, unfortunately.I wrote two blog posts about this recentlyhttp://www.sqlservercentral.com/blogs/martin_catherall/archive/2011/03/16/disaster-recovery-exposure-part-one.aspxhttp://www.sqlservercentral.com/blogs/martin_catherall/archive/2011/03/16/disaster-recovery-exposure-part-two.aspx

The Chance of Failure

Steve Jones - SSC Editor — Wed, 30 Mar 2011 22:19:40 GMT

Comments posted to this topic are about the item [B]The Chance of Failure[/B]