SQLServerCentral Article

Disaster in the Real World - Hurricane

,

As you may have read in the newsletter, I'm lucky enough to live in the

Orlando area, right in the middle of the path Hurricane Charley followed. What

follows are some of the problems I've experienced both personally and

professionally over the last few days. Hindsight is 20-20 they say, but I'll

throw some of those comments in too.

My first observation is that most people in Florida haven't been through a

hurricane and as much as you might try to explain, it's not the same. I saw a

comment in the paper from a man who had been playing golf at 6 pm on Fri and had

been telling friends that he had played in far worse weather. Quite a different

story three hours later. I spent two months in Miami after Hurricane Andrew -

the pictures just don't do it justice. Catastrophic is a pretty good word.

Anyway, I think before you are allowed to live in Florida you should have to

tour a replica of a post hurricane city. Rant complete.

We were watching the storm at work and even as late as Thursday at noon we

were planning business as usual on Friday. At that point we would have been

catching the outer band of the storm, probably 30-50 mph winds, heavy rain. I

decided Thursday afternoon to have my team work at home on Friday, playing it

safe, but everyone else was scheduled to be in the office. Late Thursday we

decided that Friday would be a half day. About the same time we found out all

schools would be closed Friday. In hindsight (as promised), I think we would

have done better by our people if we had closed or went to minimal staffing.

It's tough, you balance losing billable hours against not giving yourself enough

margin. As we wrapped on Friday we were still expecting business as usual on

Monday.

I spent Friday afternoon prepping for the storm. Making sure we had

batteries, water, etc, plus securing all the loose stuff outside - Florida homes

tend to have lots of chairs, pool floats, and other stuff that becomes deadly in

full hurricane winds. At 2 pm the power went out. Just very light rain at that

point, no wind. The power company said it was a fuse out on a pole but that they

were deferring all fixes until after the storm passed. Mind you that the storm

wasn't due to arrive for another 7 hours. Summer in Florida without AC is

miserable. I sent my wife and baby girl to a local hotel, sat down next to a

window to catch up on some geek reading.

About 8 pm the power went out at the hotel! Ah well. At least the room was

very cool and the heat of the day was gone, too late to try moving them with the

storm an hour away. Just about 9 pm the storm arrived. Fairly serious wind, but

in my area I don't think it exceeded gusts of 50-60 mph. The whole thing was

over in an hour or less. No damage to the house, hotel fine, sit back and wait

for morning.

Sat morning I survey the area. Couple minor tree limbs down in my yard, tons

of leaves and small branches, nothing serious at all. Pretty similar in the rest

of the neighborhood. My neighbor across the street has his generator running -

must be nice! We've been house hunting and had planned to acquire a generator

then, so I had put off buying one because I wanted one permanently installed,

didn't make sense to do for a house I was going to sell and I hated to buy a

portable one that I wasn't going to need later. Hmmm...maybe not the best

decision. If having a generator installed at your home seems like overkill,

trying going without ice or AC or anything else electric for a few days. It

varies by area, but we've averaged 2-3 power outages of 12 hours or more where I

live per year for the past few years. Is there a lesson here? Perhaps, but that

20-20 thing...

Over the past couple years we've seen people become more and more reliant on

cell phones, in many cases opting to stop paying for a traditional land line.

Others have turned to VOIP services such as

Vonage that tend to need a broadband connection, usually cable to work. At 8

am Sat morning impossible to get through to anyone on a cell phone. Even today I

can't reach my mom across town on her cell phone, though in most other areas it

was ok. Local authorities were asking everyone to not use their cell phone as

they needed all the available bandwidth to help with the recovery efforts. We

use Nextel at work, phone calls would not go through during much of Saturday,

but the radio part seemed to work better. I don't know, but I wonder if cell

carriers don't have a way to prioritize emergency traffic. Landline worked the

entire time.

Mid morning Saturday I move wife and kids to my mom's, on the way back our

network dude calls. We had switched critical systems over to run on the building

generator, but the generator doesn't have enough power to run the AC - our DR

plan calls for us to rent portable AC units if power will be out for more than

few hours. He had the units, but somehow the last time the generator wiring was

changed it was done in such a way that the elevators had no power! It took four

of us to get two of these things up to the third floor. Two pulling, one

pushing, one holding a flashlight because there was no stairwell lighting

either. Brand new they cost about $2k - maybe we should have just bought them?

Even with only mission critical systems running, the server room was up to 99

degrees when we got the coolers plugged in. Just can't get the heat out even

with a decent sized fan. Four hours later it had dropped to a hot but manageable

92 degrees.

Driving is crazy. Enough major lights are out to make left hand turns a

danger, everyone seems to have forgotten the rule that if the lights are out to

treat as a four way stop.

Sat night, still no power. Not fun. I've got two large dogs and a cat, no

room a my mom's, I've got to stay at home.

Sun morning I stand in line at Lowe's to get a generator. They ran a first

class operation. People who had been there to get one on Sat had been given a

ticket to return for a truckload due in Sun morning. Sun morning they had two

lines, one for people who had tickets, one for people who did not. Officer

Morris from the Sheriff's office was on hand to keep order (I mention this even

though he may never see it, but he was polite, patient, helpful, cheery -

everything you could ask for in a civil servant). Several times the store

manager came out to make sure everyone knew what was going on. Once they had

everything organized, they had the people with tickets entire single file, then

the rest of us followed behind. We passed by stacks of 5 gal cans in case we

needed them, then went all the way to the back of the store. I'd say it probably

took an hour for them to hand out 300 generators, took another half an hour to

check out and head back home. Lowe's did a first class job, couldn't ask for

better. Not all businesses were as well managed or responsive, or understood how

to communicate to help manage the stress that comes with times like these.

I head up the road for gas, the station has power but no gas. Go home, drop

off the generator, go back out to get gas. Find gas, come back, generator starts

on the first try. Always a nice thing. The generator holds 5 gals and is

supposed to run for 11 hours, so I head back to Lowe's to get another couple

cans and then get more fuel. No 5 gal cans at Lowes, sold out. I grab a couple 2

gal cans. Spend 45 minutes looking for gas. It's always something isn't it? The

stations that had power sold out quickly, but there were tons of stations with

plenty of gas, no power.

Back at home, I throw out all the food from the refrigerator. What can you

do?

As I'm driving around Sun most of the major stop lights are working, either

on main power or a generator, a few officers directing traffic where there is no

power. There are a few fast food places open. Sun night we go out to dinner at a

local restaurant. At the same time we are hearing about the extent of the

damage, ice and fuel shortages, that the power might be out for a full week.

Power customers starting to get irritable, lot's of situations where the house

across the way had power but they didn't. Power companies prioritize their

efforts to get banks, hospitals, police, fire, other important things running,

residential customers last on the list. I don't doubt that they were working as

hard as they could, where they weren't doing a good job was conveying to the

customer base where they were making headway, what the challenges were, a better

guess at a timeline for specific areas. To me, this is one of those times where

you give out as much information as you can. People may not like what they hear,

but they should be able to see some progress and understand the complications. I

suspect the power companies don't like to get into that kind of detail for fear

that they will then spend more time defending those decisions (or defending why

it's broken to start with). People don't like unknowns, better to let them know.

Sun at noon our plan is to have our main office closed (no power), our remote

office open. We'll have everyone report there, figure out who we can use, send

the rest back home. Not my favorite plan, I thought we'd have done better by

everyone to just stay closed Mon. At noon there were still streets blocked in

some areas, a fuel shortage, schools closed on Monday.

Sun at 4 pm we get main power back at the office. We change our plan to have

everyone to report for work as usual.

Mon six of eight people on my team are able to make it in, not bad. Life is

pretty normal at the office. No damage to the building, no data loss, life is

pretty good. No where to go for lunch, but that's ok. Lot's and lot's of our

employees have no power, are struggling to find ice and fuel. By Mon night they

have made some progress restoring power, but it's going to be Saturday before

Central Florida is 100% restored, and Thurs/Fri for large pockets of residential

customers. Orange County (Orlando) next door to us announces that schools will

not resume until the following Tues. My wife finds a gas station, brings over 2

5 gal cans, enough to run the refrigerator off and on long enough for cold water

and ice for a couple days.

Tues (today) sees some progress with power restoration, just not my power!

Still forecasting Thurs or Sat depending on who you listen to. Local schools set

to resume Fri here in Seminole County. Work is normal, more places open.

We were lucky in my area, not much damage. Other areas not far away fared far

worse. The only real lesson I see here is that good communication - and the

ability to communicate - are huge parts of managing a disaster. I imagine a lot

of you can relate to the last time you had a major incident at work, how many

people stuck their head in the door saying "how much longer?". I talked to Steve

about this some and he had some nice bits of info about how he has seen

companies prepare for and handle incidents, hopefully he will find time to write

some of that up soon. Beyond that, I think a great DR plan is one where you

really war game scenarios, just to see what you might could do. If we had been

on the 20th floor that whole no power to the elevator thing would have really

hurt!

Rate

You rated this post out of 5. Change rating

Share

Share

Rate

You rated this post out of 5. Change rating