Many years ago, I was on call for a production system. This was common for me since I was one of several production DBA’s for the company. It was a normal weekend for me, and I got up and started playing with my kids. They were heavily into trains, and we were laying out our latest large train set in the living room. We would cover most of the floor with train tracks. The kids would later use the train to pick up little people from various cities, and they played various games. As I assembled the tracks, I got a message on my phone.
“Sys alert: DB-01 Server is not responding”
Since this was one of the primary databases, I was very concerned. This was the days before Slack, so we didn’t have good “channels” to post a message and ask what was going on. I logged in and started troubleshooting to figure out what was going on. At the same time, I started reaching out to some of the operations folks to see if they knew anything. I got a response back.
Ops: “Hey Pat, we took that down since we are doing upgrades on the system right now; it should be done soon”.
Obviously, my first thought was WTF????? But….Instead.
Me: “Ok, I wasn’t aware of this. Do you need anything from me?”
Ops: “No, we should be done in about 20 minutes, and then you will probably need to validate”
Me: “Ok, I’ll wait and see”
20 minutes later,
Kids: “Dad, dad, dad, can you finish the train tracks…”
Me: “A few more minutes, I’ll help soon”
30 minutes past…….
Kids: “Dad! Dad, Dad!......can we get the train tracks finished”
Me: “Almost done…..just a few more minutes”
Me: “Hey guys, still waiting here by the computer. Do we have things back up and running”
Ops: “Oh ya, all good. We were done about 30 minutes ago”...........
Some of you out there will be nodding your heads right now since you lived through days like this as well. Anyone who’s been on call for a system has dealt with issues when the system goes down or problems arise. It’s especially hard when simple communication issues make you lose time with family, friends, or just relaxation.
To help, I thought I would share some simple communication tips and tricks. Now keep in mind, technical folks like engineers, operations, database people, and network people are NOT typically great at communication. They are usually very introverted, and the last thing they want is to talk about something that might cause downtime or a problem in the system. These guidelines help with that.
- Do not be afraid to over communicate. If something is large enough to impact the entire system, then YES, the General channel in Slack with an @here is acceptable! Your company may say otherwise, but start with this idea and modify it later.
- Don’t fault people for over communicating! I see this very often! People will complain that they don’t want to be alerted about something that doesn’t impact them. If you own the change and it will impact systems, it is your job to communicate it. If someone complains that they shouldn’t have to worry about it, agree and tell them anyways.
- Be clear and concise in what you are doing. Avoid. “The system will be down”. Which system? How long? Will it impact product X? Is it the whole system or just one part? It’s much better to be clear. “The order system will be down from 9:00 a.m. to 9:15 a.m. This will only impact orders; all other aspects of the system will work”
- Be clear about who you need involved and what you might need from them. “We will need the DBA team after we complete at 9:15 am” “We need engineering to validate this change at 9:15 a.m. " This message also sets up clear lines of communication. “We will use this “channel” to discuss all items related to this”. The channel can be a Slack channel, Telegram, WhatsApp, or sms text group. Anything that supports a group of people talking and coordinating.
- Follow up clearly and concisely. NEVER just post “It’s done”. Much better would be. “The update is complete, and the system is back up and running. We have validated these items, and if you see any issues, please contact us”.
I know these all sound like communication tips for incidents and system updates, but they're really guidelines for all the communication you do. You can use these in meetings, when working with other teams, or even when organizing things with friends. The more you practice these ideas of clear communication, the better things will be for you in your career.
Hope these help, and you have uneventful weekends, spending time doing the things you love!