Comments posted to this topic are about the item Disaster After Disaster

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com

The company that I work for provides services to monitor generators and we work closely with a company that provides maintenance on the generators. It is a recommended practice to start those generators at least once a month, allowing the switchgear to transfer the load from utility to the generator to make sure that all of the systems are working properly.

It is not unusual for customers to end the regular test periods because of the cost of fuel. I am aware of situations where generators have sat for years without any tests or maintenance and then, surprise, they did not function correctly when they were needed in an emergency situation.

I kind of equate this to taking regular database backups, but never restoring them to ensure that everything is working. It's easy to say that you are covered in an emergency, but you better be able to follow up.

I know of a company that tested its generators every month. Then, when disaster struck, it turned out they had half an hour's worth of fuel left, because they'd never refilled the tanks after all those tests. The power outage lasted something like four hours.

Can't say what company it was, but their servers being down actually had important law enforcement implications of a very negative sort.

They also had set up replication to a co-lo. They'd never tested it. So, when the gennies went down, the co-lo failed to come up.

Worked with another company that, during a hurricane, moved all employees to a hotel in another city, moved the servers to a co-lo, then found out that the type of connection the hotel's business center had didn't allow their applications to connect to their servers. The servers were up and running, the workstations were up and could connect to websites, but the two couldn't talk to each other. (I don't know enough about internet connections to know why this was so. But it definitely was.) A couple of salespeople with laptops and cellular data plans (this was before smartphones became popular) were the only ones able to connect apps to data.

Again, lack of taking "everything" into account.

I also worked with a company that, when an earthquake took down their primary data center, they had about 1 second of downtime, and then just kept on going. They had a plan, it was complete, it was drilled (practiced), it was clearly understood by and known to everyone involved, it had as much automation to it as made sense, and so on. Worked like a dream. A few years later, another data center had a blackout (someone apparently crashed a truck into a transformer a few blocks away), and, again, no significant downtime, no measurable interruption in work.

It really does depend on the level of detail of planning, and how well everyone knows the plan and has certainty on it.

- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread

"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon

It reminds us that there are all sorts of problems that must be considered. Our particular site (in a former manufacturing site) has 3 Caterpillar diesel generators and a storage tank large enough to run for a month.

We did have a close call with the T3 lines however. We have two completely independent lines, one from a conventional telecom and one from a cable company. The cable company's system went down and stayed down for a week. Fortunately the other one did not (though it HAS failed in the past for other reasons). If that one had failed as well, we would be sunk, and it's possible that key employees could not even get to our DR site.

My neighborhood was down for 11 days (I had by pure luck purchased a generator 2 weeks before Sandy. That same cable company was out in my neighborhood, and they actually placed a portable generator by a utility pole to power their customer's phones which was considered critical because of no 911 service.)

...

-- FORTRAN manual for Xerox Computers --

My company is in New Jersey and lost power during Hurricane Sandy. We had a meticulous business continuation plan and our colocation site had us up and running within 24 hours. The plan assumed that we would just log in from home and keep the company running. But in north central Jersey, nearly every customer had no electricity for the first few days, literally 4-5 million people with no lights and no heat, (I was one of them,) but also no Internet connections, no wifi, etc. By the time the colo site was up, most of our workers could not be contacted because their cell phones had run down and most landlines these days rely on cable or optical modems that don't run without power. No one had Internet access for the same reason.

If you have a BC plan, great, but you should examine every contingency and not just assume "that won't happen." Our assumption that the business might be down but "of course" the workers could just log in from home was, in hindsight, rather stupid on our part.

I just wanted to emphasize Steve's comment:

Sigerson

"No pressure, no diamonds." - Thomas Carlyle

A few comments on disasters.

If you are critical, then you need a real landline and a phone that doens't require power to operate. This is one huge reason we have an old fashioned land line at home. It always works.

Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.

If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered

A good point and one that applies to having a hard landline as well.

Sigerson

"No pressure, no diamonds." - Thomas Carlyle

Actually for many NJ residents, landlines were out for days. At least a cellphone can be charged from a car battery.

...

-- FORTRAN manual for Xerox Computers --

We have a home generator that runs a self-test once a week, running for 10 minutes. I also try to check fuel before vacation because for our business (horses) this is critical.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com

I've always made this a priority. In a disaster, you can't assume that people can be contacted, or that they will come. I've told my boss that if my family is in trouble, I'm not coming to work.



We use Propane here, but you have to be careful with nat gas. Lots of people assume the lines will be working in a disaster, and they may get shut down. A local supply on site is needed.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com