March 21, 2014 at 3:55 am
Some years back we had a real big software release during the weekend, working long days.
Part of the release involved adding a column on some tables and filling it with a default value.
While running the script, late on the saturday evening, the log file started growing, no matter executing log backups, it wouldn't reuse any log space.
Lots of stress, until (an hour later) I remembered we put a cdc instance on some of those tables ...
I won't make that mistake again.
March 21, 2014 at 5:56 am
I had worked at my present company for 3 years training people on computers and in 2001 was asked to join the IT department as the help desk. Unknown to me they fired the other IT person the same day. So I was the only IT staff.
A month later the accounting server died. Main app was Microsoft Dynamics on pervasive SQL. I managed to get the server up but had to re-initialize the RAID which wiped everything. I was already over my head. I thought I could use the backups to restore things. Unfortunately I had never looked to see what was being backed up, just assumed the previous person knew what to do. Well, you know what happens when you assume. There was very little that was backed up. I thought my career here was done.
Fortunately our payroll was handled offsite so the vendor was able to pay us. But we lost all the HR, GL, etc. My boss said it was not my fault as I was only help desk, not responsible for server maintenance.
I worked 12 hour days for the next three months ensuring things got fixed and backups happened at night. I also explained to the people in charge that our network was crap (Windows NT4) and needed to be upgraded.
Six months later we installed a new network, good tape backup, and hired a network admin.
Now I'm network admin, DBA, etc. but have learned much from all of you and test my backups and restores. 😀
March 21, 2014 at 6:29 am
I am not a DBA but it is a DBA horror story.
Working on a development project a long time ago in a company far, far away, the development team asked for the development defect database and the source code database be backed up on a regular basis. Someone on the project was told that it was in hand.
It was not.
Server fails. A months worth of UAT lost. 9 months worth of source code history lost.
Fortunately, a person on the team who had been moaning that he had done no development tasks except reviewing was the only person with a known perfect copy of the source code from the last release. So all history information was lost but we could continue development. UAT was restarted and, unbelievably, the management blamed the delay onto the development team (although there was evidence to the contrary). That did not go down very well and relationships between the various teams were strained for the rest of the project (and possibly longer but I was only there for the project).
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
March 21, 2014 at 6:55 am
Hmm this could be lots of pages with long stories of woe.
Which should I tell...
? the times when someone ran a DELETE in production without a WHERE and the fun getting the data back.
? when I was forced to go on vacation in the middle of creating a huge data scrub script, only to return and find that it was assigned to someone who just ran it. 4 weeks of nights and weekends creating mini-data cleanse projects to fix the 3 million records affected.
? the time after we just got dual monitors and someone hit F5 with focus on the wrong screen. The first statement was DROP DATABASE ...
I cannot go on.
March 21, 2014 at 7:17 am
Who ever supports in PROD environment experience like "butterflies flying in my stomach" and I guess that would be the BAD DAY!
---------------------------------------------------
"Thare are only 10 types of people in the world:
Those who understand binary, and those who don't."
March 21, 2014 at 10:23 am
A tough story, submitted as an anonymous post:
My comments: Anonymous, a tough spot to be in and one that challenges you professionally and morally. Honestly, I'd be making plans for a transition to another job. I realize that can take time, and I'd want to move to a place/position that I think fits me, but a move is something I'd look to do in the next year.
== =========
My Bad Day is, unfortunately for me, is still happening. I work for a non-profit agency under contract to the City to provide some initial counseling for low income people in need of the services we provide and treatment we fund. We have some databases in which we track data points concerning the counseling and also track the City’s money as it’s used to fund our customers’ treatment. I’ve been involved in much of the writing of the applications we use and helped writing many of the views and stored procedures used, as well as designing tables and coming up with procedures to handle the business. Due to the nature of our business and the clientele that we serve, their need will always exceed the ability to meet it. It’s unfortunate, but true. Therefore, in an effort to be fiscally responsible and because we are stewards of the public’s funds, many years ago we instituted a plan of dividing the annual allotment of funds into quarterly allotments. Through our hard won experience we had learned that if you don’t do this, then you’ll run out of money to fund treatment 6 to 8 weeks before the end of the fiscal year. This way, by dividing the annual allotment up into quarterly allotments, we’d likely run out the last 2 weeks of a quarter, but that was easier for those agencies providing treatment we reimbursed using the City’s funds. It was better to be out of money for just 2 weeks, then 6 to 8 weeks at the end of the fiscal year.
Around 2007 or so the people at the City who managed our agency, retired. The management that took over have an entirely different philosophy, concerning trying to meet the needs of our clients. They want to always meet the needs regardless of how much it costs. That’s great, so long as you have an infinite amount of resources, but the City doesn’t. Naturally the recession only exacerbated the situation. Our agency’s operating budget has gone down almost 45%, since the retirement of the original managers. And we’ve lost more than 50% of staff through layoffs since 2007/2008. In the IT department we were at 4, we’re now down to two. Since I’m now the accidental DBA, I have to do all of the things the previous IT manager used to do, that the new City management asks, and that includes what I call stealing from Peter to pay Paul. They’ll ask that we move money from a later quarter to the current quarter so that there’s no break in coverage for financing treatment. Even though the current City management was told, repeatedly by my former IT manager, that we had a system in place so that funds would stretch through the fiscal year, City management has consistently ignored our advice. I’ve learned from my former boss that trying to get the current City management to see wisdom is useless. Naturally we run out of funds 6 to 8 weeks before the end of the fiscal year. What’s happened the last couple of years is the current City management has found funds from other departments at the City, through other agencies that contact to those other City departments, by taking money from their budgets to feed our own. I know of one agency which was closed, and I’ve reason to believe it was so that the City could continue to fund the treatment we provide. I feel awful that we were inadvertently responsible for closing another agency and laying those people off. Last fiscal year, I was told that my job might be eliminated, again to provide uninterrupted treatment for the needy public. I survived last year, but do not believe that I’ll survive this year. I’m likely to find out in the next month or at most two, if my prediction of losing my job, is correct.
What makes matters worse is my surviving IT colleague. I’ve been warned by other co-workers that my IT colleague is trying to damage my reputation. I believe this is likely possible, because she comes to me to bad mouth our other co-workers. If she comes to me to bad mouth others, I have to believe it’s likely she goes to others to bad mouth me. Certainly, if the issue is just money, I’m the larger target, since I’m a level 3 and my IT colleague is just a level 2. For the last 2 years I’ve lived with the fear of losing my job. Sometimes, it’s hard to sleep at night.
Steve has suggested that writing this down might be cathartic. I guess it might be. I won’t reply to any message posted in the thread for this article for obvious reasons, but I will follow with great interest what people, especially if any reply to my post (through Steve).
March 21, 2014 at 11:08 am
I accidentally scrambled production data in an effort to scramble DEV data once.
Lucky for me the data was recoverable from a cloned table on another database.
March 21, 2014 at 11:25 am
I had assumed responsibility for a process that paid a particular group of our employees outside the standard system (think expenses, as an example). I had to execute stored procedure to process the data and ultimately generate an Excel file that was submitted to the payroll company. The parameters for the various procedures were a bit clunky and it was all too easy to make a mistake – even with all the notes I had put together. Bottom line, I had not set the parameters correctly one month and it blew up the following month causing the payroll to increase by $750K, an increase of well over 50%. The file went through two other gatekeepers, neither of whom noticed the payroll had exploded, and the employees were paid. It didn’t even bubble up at the highest level (CFO) because the threshold for “whoa, something’s wrong!” was set at 1 million, so I was under the threshold. After discovering a problem, I was able to identify, dollar for dollar, exactly who was overpaid and by how much. Since they had already received nicely inflated paychecks, what I had to do was deduct the overage from their subsequent reimbursement checks over the next several months. Some people never had another submission so they never had their overage deducted and got a nice little windfall. When all was said and done, after about 10 months, Accounting gave me the OK to not try to recover the remaining overage amount. I’m no longer there (not for this reason!) and it’s been a while, but I think I was able to “recover” all but maybe $30K. Finance and Accounting, not to mention the CIO had every reason to hang me out to dry and did not. When they communicated to the employees, they blamed the problem on a system upgrade. 😀
March 21, 2014 at 3:16 pm
I agree with Steve on Anonymous. Of course it's easy to say "find a new job" but much more stressful if you're already living paycheck to paycheck, have debts or family depending on you, or if your marketable skills have slipped in favour of the proprietary systems & data you work with.
But whatever you need to do to start growing your skills, networking, going to SQL PASS chapter meetings, etc. Blogging, update your LinkedIn. And certainly don't put in overtime!
Leonard
Madison, WI
March 21, 2014 at 3:24 pm
I have chuckled about this several times already today! This is great!
Manic Star (3/21/2014)
I accidentally scrambled production data in an effort to scramble DEV data once.
As if I don't have enough shame from my other post, a loooong time ago when I was green, I bulk copied some data from production to move to dev, then truncated the table on dev. You guessed it....I truncated production instead! Thankfully I had just bulk copied the data so I just bulk copied it back. I truly felt all the blood rush to my feet when that happened.
March 21, 2014 at 5:49 pm
phonetictalk (3/21/2014)
I agree with Steve on Anonymous. Of course it's easy to say "find a new job" but much more stressful if you're already living paycheck to paycheck, have debts or family depending on you, or if your marketable skills have slipped in favour of the proprietary systems & data you work with.
The problem with abandoning ship in his case is not that it is necessarily a bad job, but more that the management has mismanaged the program/funds that he's working with.
So it isn't a technical issue, but rather a management issue. No dead server; or deleted DB; or an error that he has some responsibility over.
But when it comes to job hunting, it is really a full time job on it's own. I was let go from my last job because of an AD scripting error and not a DB problem. I had been looking for a job fro the prior year while I was at my current job and had only one interview.
The five weeks between jobs had three phone interviews and two face-to-face. The second F2F had me a job offer. I'm making more, now able to telecommute and am essentially on par with my manager for a respect level.
----------------
Jim P.
A little bit of this and a little byte of that can cause bloatware.
March 24, 2014 at 7:24 am
Someone decides to run a script that updates 100 million rows in a billion row table; if you're the DBA, then you're going to have a handful of very bad days.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
March 24, 2014 at 9:53 am
LSAdvantage (3/21/2014)
I have chuckled about this several times already today! This is great!Manic Star (3/21/2014)
I accidentally scrambled production data in an effort to scramble DEV data once.As if I don't have enough shame from my other post, a loooong time ago when I was green, I bulk copied some data from production to move to dev, then truncated the table on dev. You guessed it....I truncated production instead! Thankfully I had just bulk copied the data so I just bulk copied it back. I truly felt all the blood rush to my feet when that happened.
Yep, i walked into boss's office right after i realized what happened, and he was actually pretty cool about helping me recover from the error.
Lesson learned, always double-check your connection and use transactions even for 'routine stupid stuff'.
March 25, 2014 at 2:42 am
How's this for bad practice. I needed to restore a dev database, so intended on taking it offline prior to the restore. Like a fool I was multitasking, and on one monitor had an instance of of SSMS connected to our live server to do some reporting, and on my other monitor a second SSMS connected to dev. Our dev database had the same name as our live database, guess which one got taken offline!
It took about a minute to realise what I had done, by which time we had taken our 6th call from users who had been kicked out of our main 'line of business' system
We blamed the shoddy network for dropping out 😉
Viewing 15 posts - 1 through 15 (of 24 total)
You must be logged in to reply to this topic. Login to reply