August 13, 2018 at 5:29 pm
HBollah - Friday, August 10, 2018 6:35 AMWorking for a large international company when we got hit by ransomware. Kissed goodbye to 60,000 laptops, 10,000 servers and then spent the next 3 months rebuilding all the SQL instances in our data centres (approx 500 instances of SQL).
How I'd love to go into more detail but I can't, but needless to say I wouldn't wish that on anyone and wouldn't want to go through that ever again
Wow, that's a bad day. We had a bad day with Slammer, shut down our network of 10,000+nodes for a couple days, but nothing was lost.
August 14, 2018 at 2:33 am
HBollah - Friday, August 10, 2018 6:35 AMWorking for a large international company when we got hit by ransomware. Kissed goodbye to 60,000 laptops, 10,000 servers and then spent the next 3 months rebuilding all the SQL instances in our data centres (approx 500 instances of SQL).
How I'd love to go into more detail but I can't, but needless to say I wouldn't wish that on anyone and wouldn't want to go through that ever again
You are not alone.
Thomas Rushton
blog: https://thelonedba.wordpress.com
August 14, 2018 at 7:33 am
I'm reminded of another bad day. One department at a former employer was using unlicensed software and a disgruntled former employee reported them for software piracy. The president mandated that all software had to be documented, so I spent the next several days hunting down all of the old purchase orders and licenses for every single piece of software that our entire department used.
Drew
J. Drew Allen
Business Intelligence Analyst
Philadelphia, PA
August 14, 2018 at 8:39 am
mig28mx - Friday, August 10, 2018 10:36 AMInteresting topic.
I allways believed that the DBA´s are adicted to the peril, the adrenaline, the caffeine shot. Because if not, I don´t find another reazon why we can cut off vacations, work on weekends, Christmas day or any other long week period to perform some tasks that usually we can not do in office hours.
I really enjoy my work as DBA. It is when our wisdom are challenged and it is good. I end the day, feeling like House, MD, when I correctly diagnosed a "desease" and applied the correct the medicine. And eager to see the next hard case to solve.
I know, it is a child thinking, but helped me a lot.
I don't think any DBAs are "addicted to the peril, the adrenaline." I think the reason a lot of DBAs do put in the weekends, the after-hours, the holiday work is more because the better DBAs are "customer focused." Our customers are the end-users and the business, and if the database goes down, a lot of things go down. As Steve has often said, "we ain't saving babies" (well, if you're in a healthcare-related business, you might be,) but we do often match the old XKCD cartoon about sysadmins...
I know in my case, I do my production patching on the weekends to minimize the impact on the end users, not because I *have* to, but because it's what's best for me. I think in a lot of businesses, the databases could be seen as the foundation of a lot of the business. If you have to take down the foundation during the day, a lot of things come to a halt...
As for the cutting vacations short, well, to me, that just says that either the business needs to wake up and get more people who can do the job, or you've got a DBA who likes to take the "knowledge hoarding" = "job security" mindset and not share with co-workers / subordinates how to do things (I've had a boss like this, didn't enjoy it.) Smaller companies though, will be the exception, then you might have a one-person IT "team" responsible for everything...
August 14, 2018 at 9:38 am
jasona.work - Tuesday, August 14, 2018 8:39 AMI don't think any DBAs are "addicted to the peril, the adrenaline."
I'll disagree slightly. I think we get a rush from the excitement and pressure of working (and solving ) something important to the business, or at least our managers/management. Not that we try to get into situations, but many of us get excited when they happen. I see this regularly, from lots of people, who don't want to go home. When we have major incidents, we need to stagger shifts, and it can be hard to get people to take a break or go home and come back.
However, it's not just DBAs. I find lots of infrastructure people are like this, and some developers. Not most, because I've known too many that hate being called in or having their hours moved, but some. Even managers can get like this, especially if they're promoted from a tech role.
I think that the issues with working too hard/long are a mix of different reasons. Some people are terrified of losing a job and feel pressure from management (overt or implied) to get things done. We also feel some obligation to get things working. We might just have an active brain and are constantly thinking about the issue, so we keep going.
August 14, 2018 at 11:25 am
Steve Jones - SSC Editor - Tuesday, August 14, 2018 9:38 AMI'll disagree slightly. I think we get a rush from the excitement and pressure of working (and solving ) something important to the business, or at least our managers/management. Not that we try to get into situations, but many of us get excited when they happen. I see this regularly, from lots of people, who don't want to go home. When we have major incidents, we need to stagger shifts, and it can be hard to get people to take a break or go home and come back.However, it's not just DBAs. I find lots of infrastructure people are like this, and some developers. Not most, because I've known too many that hate being called in or having their hours moved, but some. Even managers can get like this, especially if they're promoted from a tech role.
I think that the issues with working too hard/long are a mix of different reasons. Some people are terrified of losing a job and feel pressure from management (overt or implied) to get things done. We also feel some obligation to get things working. We might just have an active brain and are constantly thinking about the issue, so we keep going.
Hmm. I think my read of the original post was more about doing what could be considered "normal tasks" versus the occasional "oh crap" tasks.
Granted, when I start working a problem (even if it's not a mission-critical-need-it-yesterday sort of thing) I'll tend to keep at it. But, once the end of my day rolls around, I'll shut down and head home (unless I'm partway into something, then I'll finish up first.) I'm not a fan of the weekend work to patch my servers, but I do it anyways to minimize the impact on the end users.
Now, a crisis? Yeah, I'm keeping going until it's fixed, I'll skip "fun stuff" that might get me out of work for a couple hours (our organization picnic comes to mind,) and I'll (generally) enjoy the journey to fix the problem.
OK, I guess I am somewhat addicted.
August 15, 2018 at 8:59 am
jasona.work - Tuesday, August 14, 2018 11:25 AMNow, a crisis? Yeah, I'm keeping going until it's fixed, I'll skip "fun stuff" that might get me out of work for a couple hours (our organization picnic comes to mind,) and I'll (generally) enjoy the journey to fix the problem.
OK, I guess I am somewhat addicted.
This is often what makes us successful, or really good at our jobs. Unfortunately, it can impact others around us and throw us out of life balance if we don't recognize and balance this.
August 15, 2018 at 9:19 am
Steve Jones - SSC Editor - Wednesday, August 15, 2018 8:59 AMThis is often what makes us successful, or really good at our jobs. Unfortunately, it can impact others around us and throw us out of life balance if we don't recognize and balance this.
I'm sorry, what's this "vacation" of which you speak?
And who the heck are those people in my house, they're there every time I get home and one of them is constantly kissing me and sleeping in the same bed with me? I don't recognize them at all!
Thankfully, I've never gotten that bad...
August 15, 2018 at 10:14 am
I feel that my outlook on 'crisis management' changed and matured over my years in IT. When I was young and energetic and pursuing an opportunity, I was essentially willing to do whatever it took to handle things day or night to keep production going. And it was literally day or night, as I managed alone a small 24-hour shop with developers and operators. I saw it as a personal challenge to prove to myself that I was capable. However, over a 42-year career I came to realize that it was not all my responsibility, but that the responsibility was also to be shared by other management. At that point I was not so gung ho on always being the one to handle problems. Of course, I always stepped up in emergencies, but at the same time I became much more confident in providing forewarning of problems and preventing them, and expecting in return increased financial benefit and such things as comp time and more vacation. The net effect of this on my life is that I did truly forfeit time with my family while they were growing and maturing, but the up side is that now I do try to provide more and more support and encouragement in their careers as a retired parent (three of four sons are also deeply into IT careers). Since my family is scattered across the US now, I can and do provide travel opportunities for children and grandchildren for visits which get them away from their busy lives. We enjoy family time at each other's hometowns and at various vacation locations together. Now the advice from this: What has made lots of this possible is that my wife and I faithfully put away funds both on our own and through employer plans for our retirement. We made sure we did not need to depend wholly on social security for our retirement income. One of the main features we expected in employment was the employer's participation in that retirement plan. But we never depended entirely on that.
Rick
Disaster Recovery = Backup ( Backup ( Your Backup ) )
August 21, 2018 at 6:51 am
This is a fascinating topic for programmers who, even when they belt-and-suspender everything, still have to deal with plenty of worst days caused by disasters in production. I've forgotten most of the ones I dealt with over 2 decades of programming, but I well recall the stomach-dropping feeling that came with them. I've posted a blog about the worst day I remember - and why it wasn't the time to recovery that made it the worst.
Viewing 10 posts - 16 through 24 (of 24 total)
You must be logged in to reply to this topic. Login to reply