SQLServerCentral / Editorials / SQLServerCentral.com / The On-call Demands / Latest Posts

RE: The On-call Demands

Lynn Pettis — Tue, 10 Apr 2012 09:22:33 GMT

[quote][b]TravisDBA (4/10/2012)[/b][hr][quote] For off-hours issues they're the first one called but since it's not an official on call (ie not paid for it and don't carry pager) we're not required to respond. If whomever is on call doesn't respond then the other person gets called (which rarely happens). It works well since we're both dedicated and respond if we can[/quote]I had this exact same situation at Citrix where my partner was a 21 year old kid that slept so hard a hand grenade could not wake him up and I got called as his backup all the time so I was doing on-call constantly. Citrix was very political and the boss liked the kid and made every excuse for him like for example "Well Travis he takes prescription medication at night". Well I immediately said "Well boohoo, I have a couple of beers at night does that let me off the hook when I am on call and fall asleep?" Citrix managment is some of the worst I have ever seen in over 28 years in this industry. It's almost fast-food (Burger King) oriented managment, Just terrible. :-D[/quote]At a previous employer we had an on-call rotation as well. At one point we had 4 developers, so on-call went three levels deep giving one person a week off from any any on-call responsibility. The only problem we had was one developer. If you followed this person in the rotation you were guaranteed to get the call as this person never answered their phone at night. Oh, and this person never got in trouble for not responding.

RE: The On-call Demands

TravisDBA — Tue, 10 Apr 2012 09:11:48 GMT

[quote] For off-hours issues they're the first one called but since it's not an official on call (ie not paid for it and don't carry pager) we're not required to respond. If whomever is on call doesn't respond then the other person gets called (which rarely happens). It works well since we're both dedicated and respond if we can[/quote]I had this exact same situation at Citrix where my partner was a 21 year old kid that slept so hard a hand grenade could not wake him up and I got called as his backup all the time so I was doing on-call constantly. Citrix was very political and the boss liked the kid and made every excuse for him like for example "Well Travis he takes prescription medication at night". Well I immediately said "Well boohoo, I have a couple of beers at night does that let me off the hook when I am on call and fall asleep?" Citrix managment is some of the worst I have ever seen in over 28 years in this industry. It's almost fast-food (Burger King) oriented managment, Just terrible. :-D

RE: The On-call Demands

cfradenburg — Tue, 10 Apr 2012 07:19:26 GMT

Our other DBA and I alternate who is on call. The DBA "on call" handles incoming issues during the day and watching monitoring. For off-hours issues they're the first one called but since it's not an official on call (ie not paid for it and don't carry pager) we're not required to respond. If whomever is on call doesn't respond then the other person gets called (which rarely happens). It works well since we're both dedicated and respond if we can. For any time spend working a page we get time off to compensate and our boss is very flexible with that.Since it's rare to get paged (every few weeks if that) what we have works. My only concern is that at some point either both of us won't be available or something will happen off hours that our monitoring utility catches but since we're not officially on call we won't get paged for and result in downtime only because it's not handled before users come in. However, both my boss and his boss are aware of the concern and say it's acceptable, mostly because every department in our hospital has downtime procedures that allow them to operate in the event of any system failure. Which means if anything does happen I won't need to worry as much about defending the fact that it didn't get caught.

RE: The On-call Demands

jeff.stanlick — Tue, 10 Apr 2012 07:00:39 GMT

At my organization I am the only fulltime DBA, responsible for over 50 instances and 700+ SQL Server databases. Thus, I'm on call 24x7x365 (that includes vacations). Prior to being the DBA, I was one of the developers and we had no DBA. Since I was fixing most of the database issues when they happened anyway, I asked to be the fulltime DBA. I used to get calls and pages after hours 4-5 times a week. After 4 years of being the DBA, I get actual phone calls about database issues 1-2 times a year, and pages from automated monitoring I setup 4-5 times a month. The biggest reason for the downturn is getting the developers to use source code control and automating development environment refreshes for them so they're coding in an environment that matches the production environment they're going to deploy to. They catch a lot more of their errors before the code gets to production and they write a lot better code now. Most of my pages now are warnings about impending disk space issues.

RE: The On-call Demands

Bill Brightman — Mon, 09 Apr 2012 16:17:04 GMT

That's a great record! In my experience, it's usually the DBAs who a) don't know what they're doing or b) get lazy and take shortcuts that find themselves getting lots of calls during the off-hours. There are exceptions, but as they say, an ounce of prevention is worth a pound of cure!

RE: The On-call Demands

andy.taylor 44187 — Mon, 09 Apr 2012 15:43:49 GMT

I'm one of a pair of DBAs - between us we look after 50 SQL Servers and 35 Oracle Databases. We are on call 24/7, alternating one week on one week off.In the year I've held this post, I've received just one out of hours call for a broken SQL Server, and at least 5 for Oracle!

RE: The On-call Demands

TravisDBA — Mon, 09 Apr 2012 10:11:05 GMT

[quote][b]JP Dakota (4/9/2012)[/b][hr]I am never on call. I made it a condition of employment. [/quote]I am glad for you, but in reality, most DBA's would not get the job in the first place if they made this a condition up front. It's just part of the job for most DBA's.:-D

RE: The On-call Demands

TravisDBA — Mon, 09 Apr 2012 10:00:15 GMT

[quote]If your company doesn't have controls and your cowboy developers can run roughshod over the production environment, become a champion for controls and methods. Do it now.[/quote]This makes me remember an interview I once had where the interviewer started out the interview with "All my developers have sysadmin access to the production environment, Do you have a problem with that?" My immediate return question was "Who restores the production database back to point in time if they screw it up. His answer was "Well..... the DBA, of course." That pretty much terminated the interview for me right there. You can champion all you want, but if management doesn't back your controls and methods, you are pretty much left in the ditch, and it is time to just move on. :-D

RE: The On-call Demands

JP Dakota — Mon, 09 Apr 2012 08:26:27 GMT

I am never on call. I made it a condition of employment. Been there, done that, got the t-shirt a long time ago. My work is highly specialized, so I do get called occasionally when an issue is very complicated or a DBA is stuck trying to figure something out. By occasionally I mean a couple of times a year. We have a very proficient technical staff where I work, and there are firm controls in place to prevent "accidental code" from assaulting production.

RE: The On-call Demands

phegedusich — Fri, 06 Apr 2012 19:37:10 GMT

I'm not an enterprise DBA, but I support a couple of applications with off-farm SQL instances. SQL has been rock-solid for the duration of my tenure; most middle-of-the-night calls result from application snafus. I have to give a huge "THIS" to proper development and code promotion procedures. SQL isn't excused. Adopt a strong test and promotion methodology, and stick to it. At my current job, this isn't an option and is mandated by regulation, so you can be sure it's followed without exception.If your company doesn't have controls and your cowboy developers can run roughshod over the production environment, become a champion for controls and methods. Do it now.

RE: The On-call Demands

IowaTechBear — Fri, 06 Apr 2012 14:57:58 GMT

[quote][b]Lynn Pettis (4/6/2012)[/b][hr][quote][b]TravisDBA (4/6/2012)[/b][hr][quote]We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.[/quote]I totally agree with most of what you say. However, the problem is with that, is the people reporting the issue(error), as well as HelpDesk personnel on duty many times cannot always determine if the issue is database related, network related, or application(even Access) related. In many companies the Help Desk department has a constant turnover of people, so seasoned individuals that have the experience to detect the difference have often moved on or left. So, as a result, many times a DBA will get called for a wild goose chase anyway. Fortunately, in my company the Help desk people have been around for awhile and can usually ferret this out before the call is made. :-D[/quote]My experience has been it is ALWAYS a database issue until proven otherwise beyond any reasonable doubt.[/quote]I agree. I don't want to give the impression that we only receive pages or calls about SQL Server. But we are third level support; a problem has to pass through two other levels before we are paged. :doze:Usually the service desk does what they can to help. If necessary it gets passed along to the application analyst on call who is responsible for the specific application. If the application analyst runs into problem then they contact our layer and the analyst are pretty good at notifying the teams that they think would be the most likely to help such as networking, server, application or client device people. :cool:;-) :hehe:Our service center does a pretty good job of working out who to contact for a problem, which for SQL Server issues is about 95%+ of the time. The other 5% we scratch our head and wonder why it was sent to us.:crazy:

RE: The On-call Demands

TravisDBA — Fri, 06 Apr 2012 13:10:37 GMT

Exactly my point, we get called on a lot of wild goose chases that had nothing to do with the database.:-D

RE: The On-call Demands

Lynn Pettis — Fri, 06 Apr 2012 13:08:51 GMT

[quote][b]TravisDBA (4/6/2012)[/b][hr][quote]We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.[/quote]I totally agree with most of what you say. However, the problem is with that, is the people reporting the issue(error), as well as HelpDesk personnel on duty many times cannot always determine if the issue is database related, network related, or application(even Access) related. In many companies the Help Desk department has a constant turnover of people, so seasoned individuals that have the experience to detect the difference have often moved on or left. So, as a result, many times a DBA will get called for a wild goose chase anyway. Fortunately, in my company the Help desk people have been around for awhile and can usually ferret this out before the call is made. :-D[/quote]My experience has been it is ALWAYS a database issue until proven otherwise beyond any reasonable doubt.

RE: The On-call Demands

TravisDBA — Fri, 06 Apr 2012 13:05:43 GMT

[quote]We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.[/quote]I totally agree with most of what you say. However, the problem is with the above, is the people reporting the issue(error), as well as HelpDesk personnel on duty many times [b]cannot always determine if the issue is database related, network related, or application(even Access) related[/b]. In many companies the Help Desk department has a constant turnover of people, so seasoned individuals that have the experience to detect the difference have often moved on or left. So, as a result, many times a DBA will get called for a wild goose chase anyway. Fortunately, in my company the Help desk people have been around for awhile and can usually ferret this out before the call is made. :-D

RE: The On-call Demands

cjones 47715 — Fri, 06 Apr 2012 13:01:41 GMT

How funny, I got called this morning! I seem to get 2-3 calls a month. We have 200+ employees and a call center. I'm our software dev, so if something happens with the database it tends to affect my applications. So I get called on every glitch to either help troubleshoot the issue, fix it, or make the data right if our dialer didn't get loaded properly. It never fails though, Mondays and Fridays are rip call days and always when on vacations.

RE: The On-call Demands

IowaTechBear — Fri, 06 Apr 2012 12:54:39 GMT

When I started at my current company I was the 3rd DBA so I was on call every third week. Before I joined the two DBAs would alternate, one would take day call the other would take night call and then switch back and forth every other week. Now we have 5 DBAs so I am on call every 5th week. When we are on call it is 24/7. The night time calls are rare, maybe once a week or less on average, but an average is just that. We can go a week without night time calls and a week when the service center calls with a problem after midnight 3 or 4 times in the same week. (Sleep, what is sleep?) :hehe:We strictly handle SQL Server support. No Access, no applications, just SQL Server. We have well over 100 instances of SQL Server to administrate spread throughout the state of Iowa.Our manager wants whoever is on call to do just that - take care of issues as they happen through the day, freeing up the other DBAs to do their project work. It also frees the on call person from project work and allows them to do things that usually get pushed to the side such as documentation. :-DThis is the first time where I am part of a team of DBAs and I like it. There can be more stress, but it is a stress that I can deal with rather than at smaller companies where I would have to be DBA, Developer, Analyst, and have non SQL Server problems shoved in my direction and they want all problems solved at the same time. :w00t:I just plan my schedule around the on-call week. No trips, no outings, nothing that would keep me from responding to a call within 20 minutes.So it is a planned stress vs. a chaotic stress. :cool:

RE: The On-call Demands

TravisDBA — Fri, 06 Apr 2012 12:04:33 GMT

[quote] That's what happens when you have 10 times as many developers as DBAs and the DEVELOPERS are the ones writing all stored procedures and creating/modifying tables and databases [i]without[/i] DBA consultation or interaction.Yes, you read that correctly. Welcome to my hell. [/quote]I feel your pain. I can tell you though at my shop one of the first things I convinced my manager of is that we do not just hit F5 on any SQL code coming into our QA environment. We review all incoming SQL code for standards and performance metrics and if ANY developer code fails that review, then that code is returned to development for retooling. Period. There are no exceptions to this rule and we got managment buy in on it as well. Without management buy-in its useless. The code does not rollout unless the DBA group bless it, period. Some see this as a bottleneck, sure, but let a developer lock up an important production database with a badly coded stored procedure and the phones start lighting up, then managment sees the light real fast. :-D

RE: The On-call Demands

Steve Jones - SSC Editor — Fri, 06 Apr 2012 09:28:17 GMT

[quote][b]bclyde-1080677 (4/6/2012)[/b][hr]... Then once we have everything dialed in we should have more time to [strike]prevent[/strike] [i]help[/i] the developers [strike]from destroying[/strike] [i]architect[/i] the databases.[/quote]:hehe:

RE: The On-call Demands

bsclyde — Fri, 06 Apr 2012 09:05:33 GMT

We have a team of 4 DBAs who are more or less on-call 24/7. As the team lead I usually get the first calls and will route them to the proper DBA as necessary. We average about 2-3 after-hours calls a week. About 40% of the time it is an actual database or SQL Server issue. That's what happens when you have 10 times as many developers as DBAs and the DEVELOPERS are the ones writing all stored procedures and creating/modifying tables and databases [i]without[/i] DBA consultation or interaction.Yes, you read that correctly. Welcome to my hell. We also currently do not have ANY real monitoring and alerting in place except for what we have been able to slap together in between projects and emergencies. It is improving, however, since we will be bringing on one of the industry's leading SQL monitoring tools in the next few weeks. Then once we have everything dialed in we should have more time to [strike]prevent[/strike] [i]help[/i] the developers [strike]from destroying[/strike] [i]architect[/i] the databases.

RE: The On-call Demands

TravisDBA — Fri, 06 Apr 2012 07:45:54 GMT

2 DBA's, and we are on call every other month. The key to on-call is [b]proactive automation and monitoring[/b]. Once you get that set up on all your db servers, you are emailed issues before they become major issues and you can deal with them during regular hours, thus preventing those late night calls like a T-Log that finally filled up a disk in the middle of the night. It doesn't prevent all of them mind you, but it does cut the volume down considerably. Also, having a well trained help-desk staff that can ferret out silly non-issues and simple connectivity issues before they get to you can help out considerably as well. If you have that all setup first then being on call is not such a nightmare. I have often found over the years through sheer experience that alot of after hours issues could have been caught during business hours if the checks and monitoring were already in place. There will always be issues in SQL Server that crop up quite suddenly and you must deal with them as they occur. However, I have found that 95% of most issues in SQL Server are as the result of [b]accumulation[/b]. In other words they develop over time. This is where monitoring and automation come in. :-D

RE: The On-call Demands

Jack Corbett — Fri, 06 Apr 2012 07:16:03 GMT

At my current company call is about every 10 weeks. The only calls I've gotten are for batch processes that aren't automated and have regular problems for which there are standard fix scripts. Don't ask why the regular problems haven't been fixed :-D. That's only 5 times a week and at hours you are normally awake anyway.At the last company I had call duties for we started with 9 in the IT department and ended up with 5 before my position was eliminated. We were on call for everything IT, from PC to network to Exchange to SQL Server. 99% of the calls were non-SQL related and had documented solutions. It still sucked at 3am. Averaged maybe 4-5 calls a week.

RE: The On-call Demands

Steve Jones - SSC Editor — Fri, 06 Apr 2012 06:47:31 GMT

[quote][b]Grubb (4/6/2012)[/b][hr]I'd say we handle around 3-5 after-hour calls a week. At this point, 95% of the issues are NOT MS-SQL :)[/quote]That's always fun. I used to work on a mixed team (DB, AD, Exchange, etc) and most of my calls were non-SQL Server. Always interesting to learn something new, frustrating when it's under pressure.

RE: The On-call Demands

@SixStringSQL — Fri, 06 Apr 2012 06:31:30 GMT

Our 8 DBAs take turns, a week at a time. We have a mix of RDBMS species, so if we handle an after-hours call on a system we don't understand, we will bring another DBA into the conversation.I'd say we handle around 3-5 after-hour calls a week. At this point, 95% of the issues are NOT MS-SQL :)

RE: The On-call Demands

GWAk — Fri, 06 Apr 2012 06:23:23 GMT

Depending on departmental churn rate and courtesy swaps, my on-call period falls on every 20-32 weeks. Where call frequency is concerned, we've managed to hire strong skillsets recently for 3rd shift so it's rare that we get more than 2 calls during the week, usually falling on the slot where there's no shift coverage (Sunday late PM). It's gotten a lot better, so no complaints.

RE: The On-call Demands

rutherford.jeff — Fri, 06 Apr 2012 05:14:24 GMT

Being a DBA has changed my mind about IT. I am on call as probably most 24/7. I maintain 39 SQL servers. I am lucky however so far I have yet had an evening disturbed by a phone call, email that one of my servers has an issue. I am the only DBA. We have a lot of Exchange personnel and a couple of Enterprise Vault guys. It seems they are constantly being called in to fix this a or multiple problems after hours, don't get me wrong I have my fair share of issues, but I contribute a lot of my success to my father, he taught long ago to do a job right. I don't make a guess or let me see what this does. I want it done right before I turn a product out or leave for the day. So I feel lucky that even though after hours the clock ticks on down but because I try to be proactive while I am at work, my family time doesn't suffer. I also know eventually there are potential issues that will arise and the midnight hour will come and go where I will have to go in and fix this or that issue.

The On-call Demands

Steve Jones - SSC Editor — Thu, 05 Apr 2012 21:29:28 GMT

Comments posted to this topic are about the item [B]The On-call Demands[/B]