January 3, 2012 at 6:28 am
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
...Fire. Someone did an unauthorized mod on the prod server's vm (ERP 3000 employees on stand still). Now it won't even boot. Worse than that, the only way it might boot is to kill our only backup VM. And the guy's not answering his phone! :crazy:
Absconded 😉
No he's on the phone now. His boss's not too happy!
Poor chap... New Year didn’t bring good news for him. 🙁
January 3, 2012 at 6:33 am
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
...Fire. Someone did an unauthorized mod on the prod server's vm (ERP 3000 employees on stand still). Now it won't even boot. Worse than that, the only way it might boot is to kill our only backup VM. And the guy's not answering his phone! :crazy:
Absconded 😉
No he's on the phone now. His boss's not too happy!
Poor chap... New Year didn’t bring good news for him. 🙁
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
January 3, 2012 at 6:43 am
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
...Fire. Someone did an unauthorized mod on the prod server's vm (ERP 3000 employees on stand still). Now it won't even boot. Worse than that, the only way it might boot is to kill our only backup VM. And the guy's not answering his phone! :crazy:
Absconded 😉
No he's on the phone now. His boss's not too happy!
Poor chap... New Year didn’t bring good news for him. 🙁
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
Agree. My last post was not professional, just personal. And that’s what we (at least I) consider when we post here (in NOT SQL section).
January 3, 2012 at 6:45 am
Nothing pers Dev.
PS, it's 300-ish people down, not 3000.
January 3, 2012 at 7:47 am
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
Dev (1/3/2012)
Ninja's_RGR'us (1/3/2012)
...Fire. Someone did an unauthorized mod on the prod server's vm (ERP 3000 employees on stand still). Now it won't even boot. Worse than that, the only way it might boot is to kill our only backup VM. And the guy's not answering his phone! :crazy:
Absconded 😉
No he's on the phone now. His boss's not too happy!
Poor chap... New Year didn’t bring good news for him. 🙁
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
Agree. My last post was not professional, just personal. And that’s what we (at least I) consider when we post here (in NOT SQL section).
Vent away.....better to vent here and regain composure to discuss reocvery options.
The reason for rules is to try and avoid the chaos of disaster.
Not that they are just a reference for everyone else.
January 3, 2012 at 8:18 am
Brandie Tarvin (1/3/2012)
So, skirting the edge of the "no tech speak in The Thread" rule...How is your first week of 2012 shaping up work wise? Are you in firefighting mode with month end / year end stuff? Has it breezed by because everyone's too busy with their own stuff?
Are you starting in on the pile o' projects for 2012? Or are you bored out of your skull because peeps are still on vacation and this is the only break you expect to get for the rest of the year?
I'm halfway between firefighting and pile o' projects.
Continuing with middle-of-the-night maintenance because we don't want to put a patch in place until after the holidays. I will be implementing the fix today! Then awaiting new projects. Should be soon!
--------------------------------------
When you encounter a problem, if the solution isn't readily evident go back to the start and check your assumptions.
--------------------------------------
It’s unpleasantly like being drunk.
What’s so unpleasant about being drunk?
You ask a glass of water. -- Douglas Adams
January 3, 2012 at 8:51 am
Welcome back, everyone! Hope you had a good break.
January 3, 2012 at 9:31 am
Ninja's_RGR'us (1/3/2012)
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
If thats true, how did he do it? Relaxed permissions?
January 3, 2012 at 9:34 am
MysteryJimbo (1/3/2012)
Ninja's_RGR'us (1/3/2012)
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
If thats true, how did he do it? Relaxed permissions?
Domain admin.
Turns out it was another mistake (big one, but not as big). He had left live vm snapshots on for 2 weeks. Managed to fill up a drive with 5-10 TB of free space in less than 2 weeks.
Which only 9 of our prod servers share! Only 1 of them was down this morning, the prod ERP!
Everything's back to normal-ish.
January 3, 2012 at 9:36 am
Ninja's_RGR'us (1/3/2012)
Domain admin.
Ahhh
January 3, 2012 at 9:38 am
Ninja's_RGR'us (1/3/2012)
MysteryJimbo (1/3/2012)
Ninja's_RGR'us (1/3/2012)
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
If thats true, how did he do it? Relaxed permissions?
Domain admin.
Turns out it was another mistake (big one, but not as big). He had left live vm snapshots on for 2 weeks. Managed to fill up a drive with 5-10 TB of free space in less than 2 weeks.
Which only 9 of our prod servers share! Only 1 of them was down this morning, the prod ERP!
Everything's back to normal-ish.
Wow, 5 TB in 2 weeks!
--------------------------------------
When you encounter a problem, if the solution isn't readily evident go back to the start and check your assumptions.
--------------------------------------
It’s unpleasantly like being drunk.
What’s so unpleasant about being drunk?
You ask a glass of water. -- Douglas Adams
January 3, 2012 at 9:40 am
Stefan Krzywicki (1/3/2012)
Ninja's_RGR'us (1/3/2012)
MysteryJimbo (1/3/2012)
Ninja's_RGR'us (1/3/2012)
New year has nothing to do with his incompetence (ignorance) and his not following strict orders. That server is on total lock down and it requires 3 people's yes to do anything on it of that nature on it. 2 for anything else and he's not allowed to sign on anything for it. Only execute.
If thats true, how did he do it? Relaxed permissions?
Domain admin.
Turns out it was another mistake (big one, but not as big). He had left live vm snapshots on for 2 weeks. Managed to fill up a drive with 5-10 TB of free space in less than 2 weeks.
Which only 9 of our prod servers share! Only 1 of them was down this morning, the prod ERP!
Everything's back to normal-ish.
Wow, 5 TB in 2 weeks!
Just a guess but might be less, we moved stuff around to try a save the server without deleting our ONLY backup (possibly, no way to know).
That snapshot saves the whole VM, then all the changes as they happen. Which then includes all the backups. That adds up fast!!
January 3, 2012 at 9:49 am
Ninja's_RGR'us (1/3/2012)
Turns out it was another mistake (big one, but not as big). He had left live vm snapshots on for 2 weeks. Managed to fill up a drive with 5-10 TB of free space in less than 2 weeks.Which only 9 of our prod servers share! Only 1 of them was down this morning, the prod ERP!
Impressive. Almost as good as the server admin I worked with last year who dropped a VM. Yes, you read that right, he deleted the virtual machine. It was only the production SQL Server, nothing too important.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
January 3, 2012 at 9:52 am
GilaMonster (1/3/2012)
Ninja's_RGR'us (1/3/2012)
Turns out it was another mistake (big one, but not as big). He had left live vm snapshots on for 2 weeks. Managed to fill up a drive with 5-10 TB of free space in less than 2 weeks.Which only 9 of our prod servers share! Only 1 of them was down this morning, the prod ERP!
Impressive. Almost as good as the server admin I worked with last year who dropped a VM. Yes, you read that right, he deleted the virtual machine. It was only the production SQL Server, nothing too important.
Wow, how did you guys girls recover form that?
Viewing 15 posts - 33,046 through 33,060 (of 66,816 total)
You must be logged in to reply to this topic. Login to reply