Disaster After Disaster

  • Comments posted to this topic are about the item Disaster After Disaster

  • The company that I work for provides services to monitor generators and we work closely with a company that provides maintenance on the generators. It is a recommended practice to start those generators at least once a month, allowing the switchgear to transfer the load from utility to the generator to make sure that all of the systems are working properly.

    It is not unusual for customers to end the regular test periods because of the cost of fuel. I am aware of situations where generators have sat for years without any tests or maintenance and then, surprise, they did not function correctly when they were needed in an emergency situation.

    I kind of equate this to taking regular database backups, but never restoring them to ensure that everything is working. It's easy to say that you are covered in an emergency, but you better be able to follow up.

  • I know of a company that tested its generators every month. Then, when disaster struck, it turned out they had half an hour's worth of fuel left, because they'd never refilled the tanks after all those tests. The power outage lasted something like four hours.

    Can't say what company it was, but their servers being down actually had important law enforcement implications of a very negative sort.

    They also had set up replication to a co-lo. They'd never tested it. So, when the gennies went down, the co-lo failed to come up.

    Worked with another company that, during a hurricane, moved all employees to a hotel in another city, moved the servers to a co-lo, then found out that the type of connection the hotel's business center had didn't allow their applications to connect to their servers. The servers were up and running, the workstations were up and could connect to websites, but the two couldn't talk to each other. (I don't know enough about internet connections to know why this was so. But it definitely was.) A couple of salespeople with laptops and cellular data plans (this was before smartphones became popular) were the only ones able to connect apps to data.

    Again, lack of taking "everything" into account.

    I also worked with a company that, when an earthquake took down their primary data center, they had about 1 second of downtime, and then just kept on going. They had a plan, it was complete, it was drilled (practiced), it was clearly understood by and known to everyone involved, it had as much automation to it as made sense, and so on. Worked like a dream. A few years later, another data center had a blackout (someone apparently crashed a truck into a transformer a few blocks away), and, again, no significant downtime, no measurable interruption in work.

    It really does depend on the level of detail of planning, and how well everyone knows the plan and has certainty on it.

    - Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
    Property of The Thread

    "Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon

  • It reminds us that there are all sorts of problems that must be considered. Our particular site (in a former manufacturing site) has 3 Caterpillar diesel generators and a storage tank large enough to run for a month.

    We did have a close call with the T3 lines however. We have two completely independent lines, one from a conventional telecom and one from a cable company. The cable company's system went down and stayed down for a week. Fortunately the other one did not (though it HAS failed in the past for other reasons). If that one had failed as well, we would be sunk, and it's possible that key employees could not even get to our DR site.

    My neighborhood was down for 11 days (I had by pure luck purchased a generator 2 weeks before Sandy. That same cable company was out in my neighborhood, and they actually placed a portable generator by a utility pole to power their customer's phones which was considered critical because of no 911 service.)

    ...

    -- FORTRAN manual for Xerox Computers --

  • My company is in New Jersey and lost power during Hurricane Sandy. We had a meticulous business continuation plan and our colocation site had us up and running within 24 hours. The plan assumed that we would just log in from home and keep the company running. But in north central Jersey, nearly every customer had no electricity for the first few days, literally 4-5 million people with no lights and no heat, (I was one of them,) but also no Internet connections, no wifi, etc. By the time the colo site was up, most of our workers could not be contacted because their cell phones had run down and most landlines these days rely on cable or optical modems that don't run without power. No one had Internet access for the same reason.

    If you have a BC plan, great, but you should examine every contingency and not just assume "that won't happen." Our assumption that the business might be down but "of course" the workers could just log in from home was, in hindsight, rather stupid on our part.

    I just wanted to emphasize Steve's comment:

    When a disaster gets large enough, it doesn't matter what you have contracted for.

    Sigerson

    "No pressure, no diamonds." - Thomas Carlyle

  • A few comments on disasters.

    If you are critical, then you need a real landline and a phone that doens't require power to operate. This is one huge reason we have an old fashioned land line at home. It always works.

    Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.

    If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered

  • Ross McMicken (12/12/2012)


    If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare.

    A good point and one that applies to having a hard landline as well.

    Sigerson

    "No pressure, no diamonds." - Thomas Carlyle

  • Ross McMicken (12/12/2012)


    A few comments on disasters.

    If you are critical, then you need a real landline and a phone that doens't require power to operate. This is one huge reason we have an old fashioned land line at home. It always works.

    Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.

    If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered

    Actually for many NJ residents, landlines were out for days. At least a cellphone can be charged from a car battery.

    ...

    -- FORTRAN manual for Xerox Computers --

  • Scott Arendt (12/12/2012)


    The company that I work for provides services to monitor generators and we work closely with a company that provides maintenance on the generators. It is a recommended practice to start those generators at least once a month, allowing the switchgear to transfer the load from utility to the generator to make sure that all of the systems are working properly.

    It is not unusual for customers to end the regular test periods because of the cost of fuel. I am aware of situations where generators have sat for years without any tests or maintenance and then, surprise, they did not function correctly when they were needed in an emergency situation.

    I kind of equate this to taking regular database backups, but never restoring them to ensure that everything is working. It's easy to say that you are covered in an emergency, but you better be able to follow up.

    We have a home generator that runs a self-test once a week, running for 10 minutes. I also try to check fuel before vacation because for our business (horses) this is critical.

  • Ross McMicken (12/12/2012)


    Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.

    I've always made this a priority. In a disaster, you can't assume that people can be contacted, or that they will come. I've told my boss that if my family is in trouble, I'm not coming to work.

    If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered

    We use Propane here, but you have to be careful with nat gas. Lots of people assume the lines will be working in a disaster, and they may get shut down. A local supply on site is needed.

  • Steve Jones - SSC Editor (12/12/2012)


    Ross McMicken (12/12/2012)


    Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.

    I've always made this a priority. In a disaster, you can't assume that people can be contacted, or that they will come. I've told my boss that if my family is in trouble, I'm not coming to work.

    For anticipated disasters (storms), some employers just send people off to another city before the problem occurs. If you've been designated as a DR resource, you don't get a lot of choice in that situation.

    We use Propane here, but you have to be careful with nat gas. Lots of people assume the lines will be working in a disaster, and they may get shut down. A local supply on site is needed.

    In a city, it may be impossible to legally store enough propane to power a generator for any length of time. The gas infrastructure here in Texas is well developed, and generally doesn't require any power to operate. I've never had the gas supply go out. After Ike, we didn't have power for 10 days, but always had gas.

  • Actually for many NJ residents, landlines were out for days. At least a cellphone can be charged from a car battery.

    I'm in NJ but our company is spread out with 4 big data centers in different parts of the country. One is 20-30 miles north of NYC but I don't think had much problem during or after the storm.

    I'm was one of the lucky ones with power and able to work remotely during Sandy.

    Ken

  • I live on the gulf coast and consider natural diasters (hurricanes) and DR recovery plans to be a way of life. My employer has a diesel generator that is capable of supplying our facility with uninterrupted power for approximately 1 month. I personally think that this may be a bit of overkill because in the event of a disaster of this proportion, most of us will likely have issues (loss of life or property) that would at least temporarily have a higher priority than work; and the utilities would likely be restored by the time that we would resume normal operations.

    Two of the most important things that I've learned from my past experiences with natural disasters are: 1) Maintain a solid communications plan/infrastructure including a listing of contact information for all critical employees 2) If you decide to evacuate, you may encounter difficulty when attempting to return due to fuel shortages, debris blocking the roads, traffic and curfews.

    My brother works for a call center and whenever the area is threatened by a hurricane, they fly their critical support staff and their families to St. Louis and establish a temporary call center there. They also provide meals and plan fun activities for the families, and have a team that stays behind in order to inspect these employees homes for damage and if necessary perform repairs with building materials that are staged in tractor trailers.

    The important thing is to have a plan, critique your plan and when practical, periodically test your plan.

  • Mad Hacker (12/13/2012)


    My brother works for a call center and whenever the area is threatened by a hurricane, they fly their critical support staff and their families to St. Louis and establish a temporary call center there. They also provide meals and plan fun activities for the families, and have a team that stays behind in order to inspect these employees homes for damage and if necessary perform repairs with building materials that are staged in tractor trailers.

    This is so enlightened it's almost unbelievable that an American company could be this smart. Who are they? I want to work there.

    Sigerson

    "No pressure, no diamonds." - Thomas Carlyle

  • Two examples of disaster recovery plans going bad:

    - my former employer had a small office in Wall Street and then came 9/11. Nothing happened to office, but lower Manhattan was closed. Company had DR plan and there was DR site in somewhere Jersey but when people got there, it was already taken by some other company. Plan B was to fly employees to headquarters but all flights were grounded

    - other former employer, after hurricane Vilma in Miami, DR worked, generators started and services were up and running but then people realized nobody can deliver diesel. Luckily company didn't run out of diesel, electricity came back just when there were only fumes left in diesel tank

    - current employer has DR site somewhere in north and asking them how long companies typically stay there after disaster, their answer, they still have New Orleans area companies operating there, since hurricane Katrina.

Viewing 15 posts - 1 through 15 (of 16 total)

You must be logged in to reply to this topic. Login to reply