I was talking to a DBA friend of mine, reminiscing about some of the hard lessons we learned early on in our career. The more we talked, the more we realized that there should probably be a one-page cheat-sheet that you’re required to read before you open SQL Server Management Studio.
1. The name of the job isn’t necessarily what it does. That “Backup All Databases” job probably doesn’t, and it probably has a shrink step in there for good measure.
2. A completed backup job doesn’t mean anything. Maybe the job isn’t set up to back up all of the databases. Maybe it’s a homegrown script that has a bug. Maybe it’s writing the backups to the very same drive where the databases live.
3. A lack of failure emails doesn’t mean success. It can also mean the failure emails stopped working, or they were being sent to a distribution list that has been deleted, or that the mail server is down, or that the email filter you set up the other day is wrong.
4. The last admin meant well. They weren’t incompetent, just overworked and undertrained.
5. Software vendors aren’t psychic. You can complain all you want about their crappy performance, but the reality is that your users might be using the software in a totally different way than anybody else. If you don’t give them clear, easy-to-understand performance data about query and index issues in your environment, they’re not going to be able to guess about it, much less fix it.
6. For maximum learning, you need peers and challenges. If you’re the only DBA in a shop, and you get your servers under control, you’re not going to grow. You need to tackle new challenges that you haven’t seen before, and you need outside opinions to challenge what you think you already know. You might be a big fish in a little pond today, but when you take a job in a bigger pond, be humble about what you think you know. You might be wildly incorrect.
What about you? What do you wish someone would have told you earlier?
30 Comments. Leave new
There is no postage involved when it comes to Log Shipping.
But seriously, I wish that I would have documented my SQL Server environment from the get go. I spent a lot of time learning where things sat and ran as requests would pop up as opposed to having a diagram or at least a list of my SQL Servers, their databases and which applications / users accessed them for quick reference.
Some very good points. I especially like four, five, and six. At some point, you are going to be the last admin that someone may be cursing out because of the way you did some things. That’s something to keep in perspective. For the fifth point, it’s been my experience that software vendors don’t know how to fix what they don’t know is broken. If we as users can give them honest critical feedback along with some suggestions, I’m hopeful that a lot of them would work on implementing the solutions. And for the sixth point, do you have any suggestions on finding new challenges to help you grow if you they’re not available in your current environment? I try to attend SQL Saturdays and do a lot of reading of blog posts and such, but I find it difficult to find hands-on environments to try out all the new things I learn.
Number 6 hits home really hard for me. Such an important lesson to learn.
To add to your list. One of my bigger lessons learned was learning how to say no. Early on I was always saying yes no matter what flaming hoops I had to jump through to make things happens. For some reason I didn’t realize I should say no, that’s a bad idea. It is a crucial part of our job to point out to people when they have really bad ideas.
Agreed Chris Albert. I try to break the “DBA’s favorite word is no” view but many, many times no is the correct answer.
Don’t wait for something to break to test your backup.
Agreed 10 fold with Russ but add to it test the process frequently.
Other points we good also.
When you don’t follow change procedures for “simple, insignificant changes,” interaction of features will bite you–every single time.
Just because you have a checklist of tasks (for example to set up a new instance) does not mean the other person did them all. Double-check. Better yet have them double-check and give some proof.
My lesson is… It does not matter how hard you try to make something idiot proof the idiots will still find a way to screw things up… The idiots are ingenious!
“You did what? …”
“And that worked?!?!!?”
“How did you even … ”
“That doesn’t sound right …”
#4 is the best…I always here when I start doing work with a new client how bad the previous DBA was. They stress how little they knew and so on……blah, blah, blah…….then as I start working in the environment, I see poor communication, expectations not being communicated well, no defined goals, IT is not aligned with business, no change control, so on…..The previous DBA did not increase his/her skill level and could not keep up with the last minute demands…….
* if you are not a DBA but you have some competent DB skills, expect that you will always be assigned with DB tasks
* even official titled DBAs sometimes dont know what to do in a meltdown, thqt is why they play safe and are reluctantant to do what they have to do. So you do it anyway.
* sql replication if lefrlt unmaintained will become a monster in a year or two. Leaving you, the person that just happened to get things handed down to, getting the flak of everyone when you are not a part of the team that conceptualized and implemented it
* consider using bigint if your databases would be gulping down TBs and TBs of data ib the future. We learned this thw hardway whwn sql int reached 2.1B
* you must know how to use BCP if u intend to work with very large DBs, select * into with 2B records????
* you must leaen how to read query plans.
*you must separate the datafiles of DBs that have intensive I/O in other drives. That distribution DB that has a zillion publishers will push your I/O to the limit leaving you with SUSPENDED tasks in SPWHO2
* learn to use activity monitor and dm tables, aside from sp_who2
I could write a book here but for now, these arw my contributions
I especially like number six. Coming from a culture (former emloyer) where “knowledge = power”, it took some time to get used to the sql family where knowledge is shared.
What i missed early on was a form of standardization. Things like naming convention of your instances, database settings, number of files and how to log your best queries. Sounds like the accidental DBA thrown in the deep end? Yes. Google taught me a lot but in the end there’s no substitute for experience (= learning from failures)., And I’m very happy that my boss doesn’t mind me failing, as long as i learn from it and don’t repeat it 😉
There will be days where you are a hero. There will be days when you are a zero. Don’t take either too personally. It’s the nature of this job.
Upvote for the comment about the alert emails not working. I used to set up an alert that triggered every day just so I could have confidence no one had screwed the mail system.
Documenting is good, but keep the documentation up to date is the point. Google is your best friend!
They almost always mean well and many are overworked / undertrained but we also see a lot of great DBAs that had the answer but no support from their team / management to implement the changes.
Always get someone else to test your theories/processes/scripts/etc. It doesn’t matter how many times you’ve tested it, someone else will always find a way to break it.
Love 4. From a system perspective over the years I have seen it all, and then been on the other-side a few times when people tell you about an environment you set up, and you are like “yeah” thanks for that.
My favourite was on a site that had a business system on a 50GB db supporting 50 users only, and after 7 years they never changed the underlying hardware and DB un-maintained was closer to 500GB, support 150 filling up all its environment, memory, DB and CPU etc, and they complained the environment was not suitable! Hello Crystal ball?, we are 7 years in the future now??
My lessons are two:
1) subscribe to blog newsletters or RSS feed about SQL or whatever you like and learn every day. Go on the blogs and comment and ask questions. Do this every day and you will learn.
2) No software will survive at the contact with the user
The Windows Server Team and Storage Admins are not your enemies. Everyone should be working to the same goal. If you are clashing heads all the time, then the chances are that is is the process that is broken.
If someone comes to me and says that they typed DO THIS and it did THAT, or they ran it on server A and the results show up on server B and there is no output or no log messages, etc., I say: Since I didn’t see what you did with my own two eyes, I can’t believe or not believe you. Lets just get things working the way should. And then I let management assign blame.
“A lazy DBA is a good DBA…?” – Unknown
I can really relate to #6!! It’s similar to another axiom that I’ve always tried to live by: Never get a chip on your shoulder!
Number 6 struck me! What should I do to face more challenge without living small pond because the small pond would need time to get big 😉
Good documentation will take care of you if you take care of it.
Commenting your code will definitely help someone, somewhere, someday.
If you don’t understand something, just listen to the explanation of someone who does. In full. All of their explanation. Then ask questions once they’re done.
“It’s a really quick change, just make it directly in production”, makes you feel funny for a reason.
I would group 1, 2 and 3 together, Brent: AssUMe makes an Ass of U and Me… a.k.a. Assumption is the mother of all f*ck-ups…
A variation of 4 we also call ACT: Almost Closing Time… Maybe some things aren’t perfect and should have been done better… But you don’t know under what conditions the job needed to be done… Maybe something went horribly wrong and it had to be fixed a.s.a.p. and quality code was not a requirement… Never assume that one bad job makes someone a bad person… 😉
Don’t be afraid to argue with anyone. You are probably who most know about SQL, so let them know that.
I would suggest an alternative version of that: always be curious, open, and sharing. Share what you’ve learned, and at the same time, be open and interested in what others have learned. You are probably *not* the one who knows the most about SQL.
I say that because in any given room of IT people, there’s somebody who knows something I don’t know about SQL Server. Your Venn diagram of knowledge isn’t a big circle for you that encompasses everything everyone else knows. Everyone’s circle of knowledge has an area that isn’t covered by yours.
That performance tuning follows the scientific method — hypothesis, test, and iterate, and that tuning a database means keeping all metrics of the system visible, because an improvement in one area can cause a problem in another area.