SQL Clone
SQLServerCentral is supported by Redgate
Log in  ::  Register  ::  Not logged in

Disaster in the Real World - Hurricane

By Andy Warren,

As you may have read in the newsletter, I'm lucky enough to live in the Orlando area, right in the middle of the path Hurricane Charley followed. What follows are some of the problems I've experienced both personally and professionally over the last few days. Hindsight is 20-20 they say, but I'll throw some of those comments in too.

My first observation is that most people in Florida haven't been through a hurricane and as much as you might try to explain, it's not the same. I saw a comment in the paper from a man who had been playing golf at 6 pm on Fri and had been telling friends that he had played in far worse weather. Quite a different story three hours later. I spent two months in Miami after Hurricane Andrew - the pictures just don't do it justice. Catastrophic is a pretty good word. Anyway, I think before you are allowed to live in Florida you should have to tour a replica of a post hurricane city. Rant complete.

We were watching the storm at work and even as late as Thursday at noon we were planning business as usual on Friday. At that point we would have been catching the outer band of the storm, probably 30-50 mph winds, heavy rain. I decided Thursday afternoon to have my team work at home on Friday, playing it safe, but everyone else was scheduled to be in the office. Late Thursday we decided that Friday would be a half day. About the same time we found out all schools would be closed Friday. In hindsight (as promised), I think we would have done better by our people if we had closed or went to minimal staffing. It's tough, you balance losing billable hours against not giving yourself enough margin. As we wrapped on Friday we were still expecting business as usual on Monday.

I spent Friday afternoon prepping for the storm. Making sure we had batteries, water, etc, plus securing all the loose stuff outside - Florida homes tend to have lots of chairs, pool floats, and other stuff that becomes deadly in full hurricane winds. At 2 pm the power went out. Just very light rain at that point, no wind. The power company said it was a fuse out on a pole but that they were deferring all fixes until after the storm passed. Mind you that the storm wasn't due to arrive for another 7 hours. Summer in Florida without AC is miserable. I sent my wife and baby girl to a local hotel, sat down next to a window to catch up on some geek reading.

About 8 pm the power went out at the hotel! Ah well. At least the room was very cool and the heat of the day was gone, too late to try moving them with the storm an hour away. Just about 9 pm the storm arrived. Fairly serious wind, but in my area I don't think it exceeded gusts of 50-60 mph. The whole thing was over in an hour or less. No damage to the house, hotel fine, sit back and wait for morning.

Sat morning I survey the area. Couple minor tree limbs down in my yard, tons of leaves and small branches, nothing serious at all. Pretty similar in the rest of the neighborhood. My neighbor across the street has his generator running - must be nice! We've been house hunting and had planned to acquire a generator then, so I had put off buying one because I wanted one permanently installed, didn't make sense to do for a house I was going to sell and I hated to buy a portable one that I wasn't going to need later. Hmmm...maybe not the best decision. If having a generator installed at your home seems like overkill, trying going without ice or AC or anything else electric for a few days. It varies by area, but we've averaged 2-3 power outages of 12 hours or more where I live per year for the past few years. Is there a lesson here? Perhaps, but that 20-20 thing...

Over the past couple years we've seen people become more and more reliant on cell phones, in many cases opting to stop paying for a traditional land line. Others have turned to VOIP services such as Vonage that tend to need a broadband connection, usually cable to work. At 8 am Sat morning impossible to get through to anyone on a cell phone. Even today I can't reach my mom across town on her cell phone, though in most other areas it was ok. Local authorities were asking everyone to not use their cell phone as they needed all the available bandwidth to help with the recovery efforts. We use Nextel at work, phone calls would not go through during much of Saturday, but the radio part seemed to work better. I don't know, but I wonder if cell carriers don't have a way to prioritize emergency traffic. Landline worked the entire time.

Mid morning Saturday I move wife and kids to my mom's, on the way back our network dude calls. We had switched critical systems over to run on the building generator, but the generator doesn't have enough power to run the AC - our DR plan calls for us to rent portable AC units if power will be out for more than few hours. He had the units, but somehow the last time the generator wiring was changed it was done in such a way that the elevators had no power! It took four of us to get two of these things up to the third floor. Two pulling, one pushing, one holding a flashlight because there was no stairwell lighting either. Brand new they cost about $2k - maybe we should have just bought them? Even with only mission critical systems running, the server room was up to 99 degrees when we got the coolers plugged in. Just can't get the heat out even with a decent sized fan. Four hours later it had dropped to a hot but manageable 92 degrees.

Driving is crazy. Enough major lights are out to make left hand turns a danger, everyone seems to have forgotten the rule that if the lights are out to treat as a four way stop.

Sat night, still no power. Not fun. I've got two large dogs and a cat, no room a my mom's, I've got to stay at home.

Sun morning I stand in line at Lowe's to get a generator. They ran a first class operation. People who had been there to get one on Sat had been given a ticket to return for a truckload due in Sun morning. Sun morning they had two lines, one for people who had tickets, one for people who did not. Officer Morris from the Sheriff's office was on hand to keep order (I mention this even though he may never see it, but he was polite, patient, helpful, cheery - everything you could ask for in a civil servant). Several times the store manager came out to make sure everyone knew what was going on. Once they had everything organized, they had the people with tickets entire single file, then the rest of us followed behind. We passed by stacks of 5 gal cans in case we needed them, then went all the way to the back of the store. I'd say it probably took an hour for them to hand out 300 generators, took another half an hour to check out and head back home. Lowe's did a first class job, couldn't ask for better. Not all businesses were as well managed or responsive, or understood how to communicate to help manage the stress that comes with times like these.

I head up the road for gas, the station has power but no gas. Go home, drop off the generator, go back out to get gas. Find gas, come back, generator starts on the first try. Always a nice thing. The generator holds 5 gals and is supposed to run for 11 hours, so I head back to Lowe's to get another couple cans and then get more fuel. No 5 gal cans at Lowes, sold out. I grab a couple 2 gal cans. Spend 45 minutes looking for gas. It's always something isn't it? The stations that had power sold out quickly, but there were tons of stations with plenty of gas, no power.

Back at home, I throw out all the food from the refrigerator. What can you do?

As I'm driving around Sun most of the major stop lights are working, either on main power or a generator, a few officers directing traffic where there is no power. There are a few fast food places open. Sun night we go out to dinner at a local restaurant. At the same time we are hearing about the extent of the damage, ice and fuel shortages, that the power might be out for a full week. Power customers starting to get irritable, lot's of situations where the house across the way had power but they didn't. Power companies prioritize their efforts to get banks, hospitals, police, fire, other important things running, residential customers last on the list. I don't doubt that they were working as hard as they could, where they weren't doing a good job was conveying to the customer base where they were making headway, what the challenges were, a better guess at a timeline for specific areas. To me, this is one of those times where you give out as much information as you can. People may not like what they hear, but they should be able to see some progress and understand the complications. I suspect the power companies don't like to get into that kind of detail for fear that they will then spend more time defending those decisions (or defending why it's broken to start with). People don't like unknowns, better to let them know.

Sun at noon our plan is to have our main office closed (no power), our remote office open. We'll have everyone report there, figure out who we can use, send the rest back home. Not my favorite plan, I thought we'd have done better by everyone to just stay closed Mon. At noon there were still streets blocked in some areas, a fuel shortage, schools closed on Monday.

Sun at 4 pm we get main power back at the office. We change our plan to have everyone to report for work as usual.

Mon six of eight people on my team are able to make it in, not bad. Life is pretty normal at the office. No damage to the building, no data loss, life is pretty good. No where to go for lunch, but that's ok. Lot's and lot's of our employees have no power, are struggling to find ice and fuel. By Mon night they have made some progress restoring power, but it's going to be Saturday before Central Florida is 100% restored, and Thurs/Fri for large pockets of residential customers. Orange County (Orlando) next door to us announces that schools will not resume until the following Tues. My wife finds a gas station, brings over 2 5 gal cans, enough to run the refrigerator off and on long enough for cold water and ice for a couple days.

Tues (today) sees some progress with power restoration, just not my power! Still forecasting Thurs or Sat depending on who you listen to. Local schools set to resume Fri here in Seminole County. Work is normal, more places open.

We were lucky in my area, not much damage. Other areas not far away fared far worse. The only real lesson I see here is that good communication - and the ability to communicate - are huge parts of managing a disaster. I imagine a lot of you can relate to the last time you had a major incident at work, how many people stuck their head in the door saying "how much longer?". I talked to Steve about this some and he had some nice bits of info about how he has seen companies prepare for and handle incidents, hopefully he will find time to write some of that up soon. Beyond that, I think a great DR plan is one where you really war game scenarios, just to see what you might could do. If we had been on the 20th floor that whole no power to the elevator thing would have really hurt!

Total article views: 6255 | Views in the last 30 days: -1
Related Articles

Power BI for Office 365 now available!

Microsoft has announced today the general availability of Power BI for Office 365! Microsoft descri...


Power BI for Office 365

Announced at Microsoft’s Worldwide Partner Conference 2013 is Power BI for Office 365: Power BI for...


Power BI for Office 365 preview released

I first blogged about Power BI for Office 365 two months ago (see Power BI for Office 365) when it w...


Power BI for Office 365 Released

Today Microsoft has officially announced the release of Power BI for Office 365.  You can read the P...


Power BI for Office 365 FAQ

As I have been giving demos on Power BI for Office 365, I have received a tremendous amount of posit...