Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
Log in  ::  Register  ::  Not logged in
Home       Members    Calendar    Who's On

Add to briefcase 123»»»

Bad Days Expand / Collapse
Posted Thursday, March 20, 2014 8:27 PM



Group: Administrators
Last Login: Today @ 11:42 AM
Points: 31,667, Visits: 16,128
Comments posted to this topic are about the item Bad Days

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1553298
Posted Friday, March 21, 2014 3:55 AM


Group: General Forum Members
Last Login: Today @ 5:51 AM
Points: 411, Visits: 1,351
Some years back we had a real big software release during the weekend, working long days.

Part of the release involved adding a column on some tables and filling it with a default value.

While running the script, late on the saturday evening, the log file started growing, no matter executing log backups, it wouldn't reuse any log space.

Lots of stress, until (an hour later) I remembered we put a cdc instance on some of those tables ...

I won't make that mistake again.
Post #1553383
Posted Friday, March 21, 2014 5:56 AM


Group: General Forum Members
Last Login: Friday, October 24, 2014 7:52 AM
Points: 1,859, Visits: 596
I had worked at my present company for 3 years training people on computers and in 2001 was asked to join the IT department as the help desk. Unknown to me they fired the other IT person the same day. So I was the only IT staff.
A month later the accounting server died. Main app was Microsoft Dynamics on pervasive SQL. I managed to get the server up but had to re-initialize the RAID which wiped everything. I was already over my head. I thought I could use the backups to restore things. Unfortunately I had never looked to see what was being backed up, just assumed the previous person knew what to do. Well, you know what happens when you assume. There was very little that was backed up. I thought my career here was done.
Fortunately our payroll was handled offsite so the vendor was able to pay us. But we lost all the HR, GL, etc. My boss said it was not my fault as I was only help desk, not responsible for server maintenance.
I worked 12 hour days for the next three months ensuring things got fixed and backups happened at night. I also explained to the people in charge that our network was crap (Windows NT4) and needed to be upgraded.
Six months later we installed a new network, good tape backup, and hired a network admin.

Now I'm network admin, DBA, etc. but have learned much from all of you and test my backups and restores.
Post #1553447
Posted Friday, March 21, 2014 6:29 AM



Group: General Forum Members
Last Login: Today @ 12:13 PM
Points: 6,051, Visits: 3,994
I am not a DBA but it is a DBA horror story.

Working on a development project a long time ago in a company far, far away, the development team asked for the development defect database and the source code database be backed up on a regular basis. Someone on the project was told that it was in hand.

It was not.

Server fails. A months worth of UAT lost. 9 months worth of source code history lost.

Fortunately, a person on the team who had been moaning that he had done no development tasks except reviewing was the only person with a known perfect copy of the source code from the last release. So all history information was lost but we could continue development. UAT was restarted and, unbelievably, the management blamed the delay onto the development team (although there was evidence to the contrary). That did not go down very well and relationships between the various teams were strained for the rest of the project (and possibly longer but I was only there for the project).


-- Stop your grinnin' and drop your linen...they're everywhere!!!
Post #1553456
Posted Friday, March 21, 2014 6:55 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Yesterday @ 5:22 AM
Points: 41, Visits: 385
Hmm this could be lots of pages with long stories of woe.

Which should I tell...

? the times when someone ran a DELETE in production without a WHERE and the fun getting the data back.

? when I was forced to go on vacation in the middle of creating a huge data scrub script, only to return and find that it was assigned to someone who just ran it. 4 weeks of nights and weekends creating mini-data cleanse projects to fix the 3 million records affected.

? the time after we just got dual monitors and someone hit F5 with focus on the wrong screen. The first statement was DROP DATABASE ...

I cannot go on.
Post #1553466
Posted Friday, March 21, 2014 7:17 AM


Group: General Forum Members
Last Login: Wednesday, January 21, 2015 1:05 AM
Points: 2,631, Visits: 2,069
Who ever supports in PROD environment experience like "butterflies flying in my stomach" and I guess that would be the BAD DAY!

"Thare are only 10 types of people in the world:
Those who understand binary, and those who don't."
Post #1553475
Posted Friday, March 21, 2014 10:23 AM



Group: Administrators
Last Login: Today @ 11:42 AM
Points: 31,667, Visits: 16,128
A tough story, submitted as an anonymous post:

My comments: Anonymous, a tough spot to be in and one that challenges you professionally and morally. Honestly, I'd be making plans for a transition to another job. I realize that can take time, and I'd want to move to a place/position that I think fits me, but a move is something I'd look to do in the next year.

== =========
My Bad Day is, unfortunately for me, is still happening. I work for a non-profit agency under contract to the City to provide some initial counseling for low income people in need of the services we provide and treatment we fund. We have some databases in which we track data points concerning the counseling and also track the City’s money as it’s used to fund our customers’ treatment. I’ve been involved in much of the writing of the applications we use and helped writing many of the views and stored procedures used, as well as designing tables and coming up with procedures to handle the business. Due to the nature of our business and the clientele that we serve, their need will always exceed the ability to meet it. It’s unfortunate, but true. Therefore, in an effort to be fiscally responsible and because we are stewards of the public’s funds, many years ago we instituted a plan of dividing the annual allotment of funds into quarterly allotments. Through our hard won experience we had learned that if you don’t do this, then you’ll run out of money to fund treatment 6 to 8 weeks before the end of the fiscal year. This way, by dividing the annual allotment up into quarterly allotments, we’d likely run out the last 2 weeks of a quarter, but that was easier for those agencies providing treatment we reimbursed using the City’s funds. It was better to be out of money for just 2 weeks, then 6 to 8 weeks at the end of the fiscal year.

Around 2007 or so the people at the City who managed our agency, retired. The management that took over have an entirely different philosophy, concerning trying to meet the needs of our clients. They want to always meet the needs regardless of how much it costs. That’s great, so long as you have an infinite amount of resources, but the City doesn’t. Naturally the recession only exacerbated the situation. Our agency’s operating budget has gone down almost 45%, since the retirement of the original managers. And we’ve lost more than 50% of staff through layoffs since 2007/2008. In the IT department we were at 4, we’re now down to two. Since I’m now the accidental DBA, I have to do all of the things the previous IT manager used to do, that the new City management asks, and that includes what I call stealing from Peter to pay Paul. They’ll ask that we move money from a later quarter to the current quarter so that there’s no break in coverage for financing treatment. Even though the current City management was told, repeatedly by my former IT manager, that we had a system in place so that funds would stretch through the fiscal year, City management has consistently ignored our advice. I’ve learned from my former boss that trying to get the current City management to see wisdom is useless. Naturally we run out of funds 6 to 8 weeks before the end of the fiscal year. What’s happened the last couple of years is the current City management has found funds from other departments at the City, through other agencies that contact to those other City departments, by taking money from their budgets to feed our own. I know of one agency which was closed, and I’ve reason to believe it was so that the City could continue to fund the treatment we provide. I feel awful that we were inadvertently responsible for closing another agency and laying those people off. Last fiscal year, I was told that my job might be eliminated, again to provide uninterrupted treatment for the needy public. I survived last year, but do not believe that I’ll survive this year. I’m likely to find out in the next month or at most two, if my prediction of losing my job, is correct.

What makes matters worse is my surviving IT colleague. I’ve been warned by other co-workers that my IT colleague is trying to damage my reputation. I believe this is likely possible, because she comes to me to bad mouth our other co-workers. If she comes to me to bad mouth others, I have to believe it’s likely she goes to others to bad mouth me. Certainly, if the issue is just money, I’m the larger target, since I’m a level 3 and my IT colleague is just a level 2. For the last 2 years I’ve lived with the fear of losing my job. Sometimes, it’s hard to sleep at night.

Steve has suggested that writing this down might be cathartic. I guess it might be. I won’t reply to any message posted in the thread for this article for obvious reasons, but I will follow with great interest what people, especially if any reply to my post (through Steve).

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1553595
Posted Friday, March 21, 2014 11:08 AM

SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Today @ 7:37 AM
Points: 278, Visits: 1,493
I accidentally scrambled production data in an effort to scramble DEV data once.

Lucky for me the data was recoverable from a cloned table on another database.
Post #1553615
Posted Friday, March 21, 2014 11:25 AM


Group: General Forum Members
Last Login: Tuesday, March 3, 2015 8:29 AM
Points: 198, Visits: 965
I had assumed responsibility for a process that paid a particular group of our employees outside the standard system (think expenses, as an example). I had to execute stored procedure to process the data and ultimately generate an Excel file that was submitted to the payroll company. The parameters for the various procedures were a bit clunky and it was all too easy to make a mistake – even with all the notes I had put together. Bottom line, I had not set the parameters correctly one month and it blew up the following month causing the payroll to increase by $750K, an increase of well over 50%. The file went through two other gatekeepers, neither of whom noticed the payroll had exploded, and the employees were paid. It didn’t even bubble up at the highest level (CFO) because the threshold for “whoa, something’s wrong!” was set at 1 million, so I was under the threshold. After discovering a problem, I was able to identify, dollar for dollar, exactly who was overpaid and by how much. Since they had already received nicely inflated paychecks, what I had to do was deduct the overage from their subsequent reimbursement checks over the next several months. Some people never had another submission so they never had their overage deducted and got a nice little windfall. When all was said and done, after about 10 months, Accounting gave me the OK to not try to recover the remaining overage amount. I’m no longer there (not for this reason!) and it’s been a while, but I think I was able to “recover” all but maybe $30K. Finance and Accounting, not to mention the CIO had every reason to hang me out to dry and did not. When they communicated to the employees, they blamed the problem on a system upgrade.
Post #1553622
Posted Friday, March 21, 2014 3:16 PM

SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: 2 days ago @ 10:07 AM
Points: 274, Visits: 1,424
I agree with Steve on Anonymous. Of course it's easy to say "find a new job" but much more stressful if you're already living paycheck to paycheck, have debts or family depending on you, or if your marketable skills have slipped in favour of the proprietary systems & data you work with.

But whatever you need to do to start growing your skills, networking, going to SQL PASS chapter meetings, etc. Blogging, update your LinkedIn. And certainly don't put in overtime!

Madison, WI
Post #1553681
« Prev Topic | Next Topic »

Add to briefcase 123»»»

Permissions Expand / Collapse