March 23, 2014 at 10:05 pm
Comments posted to this topic are about the item Worst Day as a DBA Story Finalists
March 24, 2014 at 4:24 pm
As much as I love to tell my story about renaming everyone Jong, wow. How can any story ever written compete with the Karate Kid story? 😀
March 25, 2014 at 11:43 am
The company I was working for at the time was fairly large and after the incident I had plenty of opportunities to re-tell the story. It happened as I said it did but I glossed over the fact that I spent the night in jail (not fun) and narrowly escaped prison/probation (the judge sympathized with me - the security guard had several citations against him for physically confronting other people). The only reason why I wasn't bankrupted was due to my company agreeing with me that I was performing my job (up to the assault) and supplied the lawyer.
I use the story now as a funny anecdote in don't-get-caught-like-this sorts of training sessions.
March 27, 2014 at 10:46 am
Well, so far, "Karate Kid" is the runaway favorite. Is this due to some sub-conscious accumulation of violent frustration on the part of DBA's? Hmmm.
My personal favorite was the Early Morning Drive Disappearance. Frankly, that seems like a bigger nightmare to me.
March 27, 2014 at 11:06 am
The karate kid story is one of my favourite stories of all time. It made me laugh. I can just imagine the sinking feeling as you realize you jusk took out the data center... :laugh:
The only other great story I have to share is the day we lost our ERP. At the time, the company I was working for was running without an IT manager. My position was in IS, on the development side, so I had little to no pull with that team. The IT team was running by committee, and they had one loud mouth who tended to railroad the team into bad ideas.
On this given Sunday, our building maintenance team needed to take the power in the server room offline for an hour to do some upgrades to the system. The only system impacted was our old SAN, which our ERP databases just happened to be running on (it was the only system left to migrate). The maintenance guys, being good guys and fairly smart to boot, offered to build us a power cord to keep the system online during the outage, at significant expense (220V cable, over 75 feet long, not cheap!). Our mighty hero, though, figured no. We can just take the SAN offline during the outage. And he made this call despite the IT guys saying it was a bad idea, and the maintenance guys saying it was a bad idea.
Sunday rolls around, and the power upgrade goes well. Maintenance cruises through it, and has everything back online ahead of schedule. But then the SAN is brought back online. And you can guess how this ends. I get a call after hour two of the SAN outage to let me know that it was gone. The SAN lost 24 of the 26 disks, and the vendor has no idea how to get them back. Apparently, the firmware on it has a bug that caused it to lose the entire configuration, and there is no fix. Enter the IS team, who conferred for half an hour to come up with a plan to do an emergency migration of our ERP system to a new server. Luckily, we had a new system in place, but had not started testing any migration plan to it yet.
What started off as a horrible day had a good ending. We were able to get our entire ERP back online within 6 hours, with zero data loss. And our mighty IT anti-hero? It was revealed that he was bilking the company for OT that he didn't work, and was fired about a month later. The lesson here is, when even the non-IT staff think it is a bad idea, it's time to reevaluate your thinking!
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply