• I know of a company that tested its generators every month. Then, when disaster struck, it turned out they had half an hour's worth of fuel left, because they'd never refilled the tanks after all those tests. The power outage lasted something like four hours.

    Can't say what company it was, but their servers being down actually had important law enforcement implications of a very negative sort.

    They also had set up replication to a co-lo. They'd never tested it. So, when the gennies went down, the co-lo failed to come up.

    Worked with another company that, during a hurricane, moved all employees to a hotel in another city, moved the servers to a co-lo, then found out that the type of connection the hotel's business center had didn't allow their applications to connect to their servers. The servers were up and running, the workstations were up and could connect to websites, but the two couldn't talk to each other. (I don't know enough about internet connections to know why this was so. But it definitely was.) A couple of salespeople with laptops and cellular data plans (this was before smartphones became popular) were the only ones able to connect apps to data.

    Again, lack of taking "everything" into account.

    I also worked with a company that, when an earthquake took down their primary data center, they had about 1 second of downtime, and then just kept on going. They had a plan, it was complete, it was drilled (practiced), it was clearly understood by and known to everyone involved, it had as much automation to it as made sense, and so on. Worked like a dream. A few years later, another data center had a blackout (someone apparently crashed a truck into a transformer a few blocks away), and, again, no significant downtime, no measurable interruption in work.

    It really does depend on the level of detail of planning, and how well everyone knows the plan and has certainty on it.

    - Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
    Property of The Thread

    "Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon