October 15, 2010 at 3:01 am
Whether you have been a consultant or been on the receiving end of consultancy there are always horror stories to tell.
Anyone care to share a few?
October 15, 2010 at 12:16 pm
Not a horror story, but definitely a cringe moment.
A job where the Dev instance was on the same server as the Production Instance. SQL Security was not their strongest knowledge set.
'Nuff said.
October 15, 2010 at 12:53 pm
Walking into a new job (as I did 3 years ago) where all the developers have the sa password for ALL instances of SQL (it was the same on each server and hadn't changed in years - yikes!). Changes to objects were being made on the fly, no dev or QA environments to speak off. I took baby-steps initially, as I was the new guy. Now I am the true owner of all but one server, they all have dev and QA and only I promote to production after a thorough code review. I've gained their respect, they (and I) know what to expect. It delays the process a bit but the end result is well worth it.
-- You can't be late until you show up.
October 15, 2010 at 12:59 pm
We had a database consulting company come in recently and develop a database and application exactly as specified by the VP of development.
Of course what he told them had nothing to do with solving the user requirements, so it was completely unusable.
October 15, 2010 at 1:51 pm
Walked into a startup and they'd been making backups before all deployments to their mission critical database for their product. They were having a crisis and couldn't restore the db.
Watching them do the backup, I saw that they constantly added new files to the backup dialog in EM. There were about 7 files in there, and 4 or 5 of which were no longer on the server. The old accidental-striped-backup issue.
Ugh.
October 16, 2010 at 7:41 am
I was on a consultancy job where we were the co-ordinator of all the clients' vendors. It was agreed with the data centre manager and the client that my colleague and I would be on site at 08:00 to install and configure a new system. This was to take 2-3 days.
To get to the datacentre required getting an 04:55 train.
We arrived at 07:55 to find a locked gate and a security guard warning us that no-one would be available in reception until 08:30. At 08:30 reception opened and we signed in and waited........and waited.......and waited. At 09:30 we asked the receptionist what was going on and she answered that the data centre manager wasn't answering his phone and she had left a message on his voice mail.
At 10:00 we insisted on speaking to either his boss or his 2nd in command. Eventually a dishevelled underling turned up and said that today the data centre was moving x thousand servers and rewiring various parts of the building so we couldn't possibly install our system. At this point we rang our boss, who rang the client and eventually we were met by the data centre managers assistant who told us that the guy had gone on 3 weeks leave last Thursday.
About 11:45 and after a lot of full and frank exchanges of view between our client, the owners of the data centre and ourselves we were allowed into the main server room where we were to install our software onto brand new kit. Well it had certainly been new at some point in its life but the stuff we were looking at looked as if it had been dragged behind a monster truck. Never mind, we decided to install our software, only to find that the SQL Server install disks that were supposed to have been provided by the client to the data centre directly weren't available. We were told that if we went to lunch and came back about 15:00 the disks would be "found".
We arrived back on time and 7 hours after we were supposed to have begun a complex installation and configuration process we actually begun work. We figured that we had better work on through the night to get ourselves back on track. After all a data centre is a 24/7 operation isn't it.
No. we were booted out at 17:30 sharp and with foreboding we asked which hotel the data centre manager had booked us into. Predictably no-one knew what we were talking about and as there is a poem dedicated to the local area (see http://www-cdr.stanford.edu/intuition/Slough.html) we didn't hold out much hope.
Eventually we were told that a hotel had been found for us and we were given an address.
This hotel turned out to be a couple of town-houses bought by an opportunistic family who had knocked doors between the two and converted every available space into rooms. My colleague drew the short straw and got the room that had been build where the staircases of the two houses met. There was basically a bed, shelves and everything else pulled out of the wall on TV brackets. there was literally enough floor space to open the door and that was it. I got the attic conversion only the conversion consisted of chip board loft boards and white washed lathe and plaster. I could see daylight through the gaps in the tiles. Given that this was winter I slept in the bed without taking off my shoes, clothes or coat.
We got back to the data centre at 08:30 the next day and finished our install only to see a red skull and crossbones light up on a panel of one of the "new" servers.
At this point I noticed that my colleague was looking a funny shade of green. Sure enough he we in the early stages of food poisoining from our hotel adventure. Of course being locked in a server room doesn't present many options and unfortunately he faced the embarassing need to relieve himself in a waste paper bin. Not wanting to leave such an object in the lanes between the servers he deposited his stinking bucket in the back of a rack cabinet.
Fortunately we were let out of the server room before the need for the bucket manifested itself again and we duly left the premises. Due to snow and bad weather the 3 hour train journey was a 7 hour journey with a dramatically and volcanically ill colleague. It was only when I dropped him off at his house that I realised that we had left the bucket that he had used actually inside the server cabinet at the data centre.
Under normal circumstances I would like to be able to say that I was sorry however I feel that a bucket of shit in a server cabinet was well deserved, but only partial payback for such incompetence on behalf of the data centre and the hapless manager.
October 19, 2010 at 9:26 am
David.Poole (10/16/2010)
I was on a consultancy job where we were the co-ordinator of all the clients' vendors. It was agreed with the data centre manager and the client that my colleague and I would be on site at 08:00 to install and configure a new system. This was to take 2-3 days....
One for the ages, thanks for sharing. :sick:
October 19, 2010 at 9:57 am
Brandie Tarvin (10/15/2010)
Not a horror story, but definitely a cringe moment.A job where the Dev instance was on the same server as the Production Instance. SQL Security was not their strongest knowledge set.
'Nuff said.
hi, could you explain abit more about your comment please?
what the actual situation was and what the problems that would arise from it were?
I dont really understand what you are saying.:ermm:
October 19, 2010 at 10:18 am
davidandrews13 (10/19/2010)
Brandie Tarvin (10/15/2010)
Not a horror story, but definitely a cringe moment.A job where the Dev instance was on the same server as the Production Instance. SQL Security was not their strongest knowledge set.
'Nuff said.
hi, could you explain abit more about your comment please?
what the actual situation was and what the problems that would arise from it were?
I dont really understand what you are saying.:ermm:
Dev is on the same server as prod. Everyone has db_owner access to both. What happens when someone's pointed to the wrong DB?
Worse, what happens when you write an WHILE loop in Dev that you forget to add a counter to stop the WHILE Loop? Or write poorly performing code that you're troubleshooting in Dev, but it's pegging the CPU or Memory at 100%? What do you think happens to the production instance?
If you still don't understand, let me know and I'll explain more fully. But in the spirit of horror stories, I like to let people's imagination work overtime. @=)
Viewing 9 posts - 1 through 8 (of 8 total)
You must be logged in to reply to this topic. Login to reply