Blog Post

Worst Day As A DBA



I remember the day pretty vividly, it was in the summer months and I was as green as green can be coming into the technology field. Walking through the doors to the start of my career  I was ready to tackle the world. The training for new hire sessions had already been completed and it was game time. I didn’t know what a SQL High Five meant at the time but I would have given one to everybody as that is how I was feeling walking through the doors.

The introductions began and I got the normal “new guy”, “fresh meat”, “greenhorn” labels that most people call newbies. As we turned the corner I noticed something that looked a bit off to me. A pen cap stuck in the keyboard by one of the developers. Hmm that is odd, they didn’t teach anything about pen caps in keyboards in college. That person was not at their desk at the time but we did find them in the break room and he was introduced to me as one of the developers.

Time past and the introductions complete I went back to my desk in my little cubicle and couldn’t help but think back to that pen cap. The curiosity was too great so off I go; back over to the desk with the pen cap, mind you this is about an hour later.

Hey man, I got to ask you a question. What’s up with that pen cap?

“Oh, yeah I was building an import process and I forgot and left a MSG box in there. I started to load the files in and instead of stopping it I figured this was a quick way to get through it.”

Hmm, interesting tactic but the red flags and sirens started to go off in my head. Being the new kid on the street and the youngster I went to one of my peers and started poking around a bit. Explaining what I saw I was amazed to learn that this in fact had happened before.

“Before”, little did I know those 6 letters would start to build the foundation of my DBA career. What, wait a minute….”before” you say? Yeah, ole Billy (no not his real name) over there has done that “before”. Nice, so I go back to my desk again and sit down. I take out a pen and paper and start writing down my questions.

1. Where is Billy loading this data?

2. What kind of data is Billy loading?

3. What kind of access does Billy have?

4. Does my boss know this?

5. What method is being used to import the data?

6. Who is the business owner?

Now realize I hadn’t even turned my computer on yet to get the lay of the land. Off I go with my questions.

Um Hey Billy, just out of curiosity where you loading that data? Prod he replies

My heart sunk, I knew the writing on the walls and where this was going. What kind of data are you using? Client data for our system. Back to my desk I go; sit down flip on the computer. I started researching, digging, and sure enough my thoughts were now a reality.

Dev Ops had gone rogue and had access directly to prod. Remember the pen cap; well after realizing that the import was loading more data than the file had in it we discovered the app didn’t have a stopping mechanism and no duplicate checks. In the end we were loading a 100,000 record file 8 times!

Light bulb goes off in my head, as I turn to colleague. Hey where is the last backup? It is on x drive but it won’t do you no good. Why is that I ask; yeah it’s a week old. We run them manually before we leave for the day.

In the end that worst day started off my worst week but looking back I believe that worst day started the groundwork for a solid foundation. How or why is that you ask?

1. Security – I’m a huge proponent of it and probably rightfully so after enduring the major cleanup that ensued.

2. Documentation – no documentation was found anywhere; we all can do a better job of this; me included.

3. Don’t be afraid to speak up; if something is off to you question it. Research it. Dig in and figure it out.

4. Just because something is done one way for years doesn’t mean it is the right way. Evolve and become more efficient. Do you think having a pen cap on an enter key to load data is efficient?

5. If you are a newbie and seasoned vet review your systems on a routine scheduled basis.

6. Backups – are you taking them? If so are you in turn testing them or validating them?

Some of the things I know now that I didn’t then are handy utilities such as Red Gate’s SQL Backup Pro that could have benefited me; take a look at their arsenal for the data professional. A wide range of products that will allow you to streamline your processes and tasks.

I look back on my time there and we brought it so far. We righted the ship but it was no easy task and is not for the faint of heart. It taught me to chip away at the wrongs and turn them to rights. I speak a lot about being a game changer. That means if you see something amiss go after it. Make it right.

While I have had a few “worst” days since then, I’ve learned one thing about being a Data Professional ~ being one comes with a price tag of having great responsibility. Don’t abuse it.




You rated this post out of 5. Change rating




You rated this post out of 5. Change rating