August 25, 2008 at 9:05 pm
Comments posted to this topic are about the item Working Under Pressure
August 26, 2008 at 5:59 am
Steve,
Years ago I worked as an Emergency Medical Tech -- it put things in perspective. Now when I'm under pressure working on stuff I can easily handle the pressure of a DB being down - yeah, some money might be lost due to downtime but generally speaking no one is gonna die.
Thats not to say you don't feel the urgency, it just helps you to keep your cool under pressure. It's the guys who don't panic who can solve the problem without doing any more damage.
Mark
August 26, 2008 at 6:43 am
I wish you'd been out here! That was some adventure, but it turned out OK. Horse is still alive, about 9 days now.
August 26, 2008 at 7:23 am
I remember a crisis with my former employer I wish I would have had help to resolve...
About three weeks before the fall semester was due to start (I worked for a school district), I had the servers fully installed/configured and just ready to turn on. Unfortunately, I was a one person shop. When I turned everything on, the servers started re-booting in about ten minute intervals. Everything I could see pointed to the vendor app causing the problem. My vendor remoted in and it looked to them the same way so they poured through code only to find their code was solid. I spent more than 100 hours that week in the server room on this problem.
I finally told my boss she needed to pay for a support call to Microsoft to have them help me decipher the problem. It ended up being the Novell client I had to use for our network. In fact, Novell had identified/corrected and reported the problem only that morning. The Microsoft rep told me had I called a day earlier, they would not have been able to solve the problem either. After downloading their fix, everything worked as I expected, late Friday night. At least I had the weekend to enjoy.
August 26, 2008 at 7:31 am
Your story reminds me of what happened to my dogs 3 weeks ago. They were playing and the little one got his jaw under the collar of the big one, it got twisted and the big dog passed out. My wife heard the little one scream, went out, cut the collar off and then called me. She was hysterical and I couldn't understand a word she was saying. Once the big dog got up on his own, she started making sense so I drove home to take him to the vet. He weighs 120 lbs so my wife can't take him on her own. Both dogs are fine.
One of the reasons I left my last job was the unreasonable expectation that I sit through long implementations when I'm not required. They didn't have a good concept of crisis management since they just wanted every single person who might be of value to be on site in case they need them from start to finish. The assistance provided would generally be one of getting yelled at for having bugs then having a manager tell you how to fix them. One implementation was 18 hours straight on a Saturday, followed by a long time on the Sunday. They wanted me there because there was a SQL Server piece being implemented along with a kludgy Oracle piece that was doomed to failure and unrelated to the SQL Server piece. The SQL Server piece worked perfectly. It was installed and I did some testing in 3 hours. The IT manager wanted me to stay until everything (including Oracle) was fully tested and operational since I volunteered to be a part of the implementation team. I told him that I only signed on for the SQL Server piece and negotiated my release (said I'd be at home with no plans to go elsewhere so he could call) since I didn't feel like spending the rest of my day at work. Everyone else got to be a part of the "all hands on deck" sitting around a terminal while they found and tried to fix bugs. From what I recall, they were fixing the bugs for weeks.
I didn't feel like burning myself out, especially when there really was nothing for me to do and there wasn't a reward either. Apparently there were hurt feelings but those didn't get presented to me until my annual review (4 months later), by which point I had one foot out the door.
August 26, 2008 at 7:53 am
Steve,
At my last job I worked for a Telecom company and when anything happened. The boss would have everybody come into one office and sit around one computer while we all discussed the problem. Sometimes we got to leave at 12:00 midnight and sometimes we left 3 days later. But everybody stayed. We would see who would come in and work the next day, and trade times when we got sleep or not.
PS: What is the deal with the gay pride on this article? It does not go with the article.
[font="Tahoma"]John Burris:hehe:
MCITP Database Administrator[/font]
August 26, 2008 at 8:13 am
Sorry about the image. The link was to something else and it must have changed on the other end.
New image loaded.
August 26, 2008 at 8:36 am
[p]My wife, children and myself operate a ranch in central Texas. Right now, our primary livestock are "commercial" goats, raised for slaughter. All too often, we have an animal emergency. Very few, however, result in a call to the vet - just the cost alone would put us out of business. Besides, it is an hour drive for him to get here, and that is only if he is at the clinic. Make it almost two, if he is coming from home. Longer, if he is at another ranch.[/p]
[p]Goats are rather hardy, so 99% of the emergencies will not result in the death of the animal, if the situations are handled promptly. These have taught the greatest lessons in crisis handling ever: quickly assess the situation, then dispatch the minimum personnel, but no less than two, to handle the situation, while the rest of ranch operations, ie: life, flows on around the problem. Don't panic. Don't worry. Do your job and do it right. Call for help, if you need it; After you have solved the problem, clean up, restock the emergency kit, report the incident and its solution to the boss-lady (my wife) and enter it into the medical record for the injured animal (ie: do the paperwork). Afterwards, discuss the problem and attempt to solve the longer term issue to prevent a recurrence of the problem[/p]
[p]They learned that panic, fear, self-doubt, arguments over who is in charge, finger-pointing, delay and over-reliance on others complicate the outcome. Any of these can be fatal for the animal, if its head is caught, it is bleeding or there is a kid stuck in the birth canal.[/p]
[p]Many times in the office environment, IT "emergencies" are nothing more than an inconveinence, albeit sometimes an expensive one. Putting pressure on those solving the problem seldom results in strong long term solution and the same crisis, with some variations to make it interesting, is played out periodically - every pay period, every month, every quarter, every new product launch or every year. The result is that management pays over and over again to fix the crisis, without ever fixing the problem. Panic and finger pointing, on the part of management and coworkers just wastes time and resources, makes tempers flare and delay the solution of the so-called "crisis"[/p].
August 26, 2008 at 8:44 am
A nice article and some good stories here, too.
On the fatigue factor. I've been there too many times to count, and made my share of big boo-boos as well.
I have learned over time, that when writin code scriptlets that modify data, or structure in an on-line production environment, there are a couple of easy and smart things I can do that help to prevent me from looking like a fool.
1. Start the script with a code block of BEGIN TRANSACTION; ROLLBACK TRANSACTION. Usually effective while testing pre-rollback results.
2. If a WHERE clause will be involved. SCRIPT THAT FIRST!!! That may not seem like much, but it could have and actually has prevented a few major semi-disasters.
August 26, 2008 at 9:20 am
Excellent.
IT positions are required by nature to be able to perform under pressure. It has been so much that way in my professional life that I have asked questions about performing under pressure in interviews.
Thanks for taking the time to present and discuss this.
Miles...:)
Not all gray hairs are Dinosaurs!
August 26, 2008 at 10:00 am
It just occurred to me that my previous post could be considered a bit of a complaint. Please let me assure you I'm thankful for my stress since it pays the bills.
My only complaint from my previous employer is that there wasn't any kind of compensation for the extra hours and I know many of us do work that way. However, I do know in many companies where overtime is not "officially compensated," they do provide some comp time so at least staff don't feel abused. Perhaps a topic for another discussion but...
August 26, 2008 at 10:04 am
It may pay the bills, but it still has to be managed. Hopefully they're not driving you too hard, Jim.
August 26, 2008 at 10:55 am
I've been in life-or-death situations a few times, and it does put DBA emergencies into perspective. Having been held at knife-point by muggers kind of makes a crashed hard drive seem less panic-inducing.
I like to follow the maxim that "disasters are emergencies that you haven't prepared for". Prepare, plan, acknowledge that you can't predict everything. General emergency training, drilling, planning and preparation will make most things less disastrous.
On the "hope for the best, plan for the worst", you also have to plan for the best, and for the most likely. Most lottery winners don't plan, and end up in bankruptcy after a short time. That's a failure to plan for the best, turning it into the worst.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
August 26, 2008 at 1:00 pm
Tom Garth (8/26/2008)
A nice article and some good stories here, too.On the fatigue factor. I've been there too many times to count, and made my share of big boo-boos as well.
I have learned over time, that when writin code scriptlets that modify data, or structure in an on-line production environment, there are a couple of easy and smart things I can do that help to prevent me from looking like a fool.
1. Start the script with a code block of BEGIN TRANSACTION; ROLLBACK TRANSACTION. Usually effective while testing pre-rollback results.
2. If a WHERE clause will be involved. SCRIPT THAT FIRST!!! That may not seem like much, but it could have and actually has prevented a few major semi-disasters.
Thank you, this is great advice. I always follow the above steps, and this advice has saved me in the past.
I also script my DELETE/UPDATE statements as a SELECT first so I can be sure that my WHERE clause does exactly what I want it to.
August 26, 2008 at 3:29 pm
Ahhhh ... those times in which DELETES with WHERE clauses that were not highlighted ...
* Noel
Viewing 15 posts - 1 through 15 (of 18 total)
You must be logged in to reply to this topic. Login to reply