February 11, 2017 at 3:48 pm
Comments posted to this topic are about the item Problems With Database Problems
February 12, 2017 at 7:45 pm
Forgiveness is certainly divine but I wonder how may people's tone would be different if they were a part of the group that suffered data losses.
This brings up a good point, though. If you trust someone with your data, what avenues do you have if they lose your data or suffer a breach? What if you have a small company and ALL of your data (and code) is lost? Will the company you entrusted help you stay in business and pay you for lost business? ... without having to sue them for damages? ... and what can you get out of a dead horse if they go out of business because of it all?
Heh... I'd say it's time to review for all that in any contracts that you may have with 3rd parties. Even if they guarantee recovery (as so many do), you'd better make sure there's a "Plan B" for when they fail. More likely, though, there will be some clause in the contract somewhere that exonerates them from any such responsibility.
A small company I did some consulting for went through that. Ironically, I hate made the suggestion that they needed a "Plan B". The CIO bought it and tried to take it up the chain of command. She was told that she was being "emotional". 3 Days later... BOOM! The data disappeared on one of the critical systems. They did restores but couldn't get the previous week to restore. It took the entire finance department (about 40 people) over a week to recover most of it through source documents, spreadsheets, and other sources. Of course, they didn't recover it all.
The company that failed to provide the necessary correct and viable backups/restoes, even though the victimized company was paying out the nose for such services, wasn't made to suffer a penny of loss because of some stupid clause that everyone overlooked or brushed off as a "would never happen".
--Jeff Moden
Change is inevitable... Change for the better is not.
February 13, 2017 at 1:22 am
I tend to go for the two geographical sources of backup being the minimal level of acceptability. Files that are on my computer as well as files stored in the cloud must have a second geographical location set. I also use a second repository type i.e. different geographical locations withing the same cloud provider is also unacceptable.
Paranoid? Maybe. But remember, data often cannot be rebuilt or bought.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 13, 2017 at 6:21 am
Jeff Moden - Sunday, February 12, 2017 7:45 PMForgiveness is certainly divine but I wonder how may people's tone would be different if they were a part of the group that suffered data losses.This brings up a good point, though. If you trust someone with your data, what avenues do you have if they lose your data or suffer a breach? What if you have a small company and ALL of your data (and code) is lost? Will the company you entrusted help you stay in business and pay you for lost business? ... without having to sue them for damages? ... and what can you get out of a dead horse if they go out of business because of it all?
Heh... I'd say it's time to review for all that in any contracts that you may have with 3rd parties. Even if they guarantee recovery (as so many do), you'd better make sure there's a "Plan B" for when they fail. More likely, though, there will be some clause in the contract somewhere that exonerates them from any such responsibility.
A small company I did some consulting for went through that. Ironically, I hate made the suggestion that they needed a "Plan B". The CIO bought it and tried to take it up the chain of command. She was told that she was being "emotional". 3 Days later... BOOM! The data disappeared on one of the critical systems. They did restores but couldn't get the previous week to restore. It took the entire finance department (about 40 people) over a week to recover most of it through source documents, spreadsheets, and other sources. Of course, they didn't recover it all.
The company that failed to provide the necessary correct and viable backups/restoes, even though the victimized company was paying out the nose for such services, wasn't made to suffer a penny of loss because of some stupid clause that everyone overlooked or brushed off as a "would never happen".
Jeff,
In almost every similar situation I've seen, at one point someone has raised a red flag and said, "What about this issue or that?" and were either ignored or ridiculed. I've experienced this multiple times in my career and you learn to notify, document and go on. And I'm usually on the team that has to clean up the resulting mess... 🙁
Most recently I was ignored for months raising a few critical issues. Management admitted they ignored me, but upper management has changed the game, making them responsible if they don't address issues promptly and accurately. So it is getting better... :rolleyes:
February 13, 2017 at 6:43 am
... The second item that I worry about is the care someone takes when performing a task. In this case, any of us might have been tired at 9pm. Especially if we'd spent the day working on a replication setupwe'd spent the day working on a replication setup, which can be frustrating. Responding to a page, especially for a security incident can be stressful. Solving an issue like that, and then having performance problems crop up is disturbing ...
Having to fight production issues on multiple fronts is problematic, especially if you started the day fighting one issue that requires your full attention and stretches your skillset. Every IT organization, especially smaller ones, should have a 3rd party resource like a local consultant they can call in when they get swamped.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
February 13, 2017 at 8:19 am
chrisn-585491 - Monday, February 13, 2017 6:21 AM...were either ignored or ridiculed. I've experienced this multiple times in my career and you learn to notify, document and go on...
The only way to go in my experience.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 13, 2017 at 8:23 am
Eric M Russell - Monday, February 13, 2017 6:43 AM...Every IT organization, especially smaller ones, should have a 3rd party resource like a local consultant they can call in when they get swamped.
Depends on what they can manage cost-wise. Sometimes it just isn't feasible to have someone retained.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 13, 2017 at 8:28 am
Regarding: I saw a note from Brent Ozar that the GitLab job description for a database specialist doesn't mention backups. It does say a solid understanding of the parts of the database, which should include backups.
I would add (which I think may be implied later in the paragraph I quoted above) that a DBA/SysAdmin's understanding needs to go beyond backup. Backups "always" work, right? The real test is restorability/recoverablity. How many of us have ever been bit by the assumption that backups are working fine but when it comes right down to actually restoring someone's data, all of a sudden we find that things haven't been working so smoothly. We then have to explain to our customer why we can't get their information back. Enter the database and you compound the issues with recoverability because you are no longer dealing with just files, but databases, transaction logs and sleuthing out where things went wrong in the first place.
February 13, 2017 at 10:07 am
A slightly different angle on the 'DR' theme, but I like google's approach. They do not mention backups at all -- instead they focus on restores. The test is can you restore the required data on demand? That of course trickles back to the backup regimen, but the ultimate test is RESTORE.
February 13, 2017 at 4:04 pm
The goal isn't backups or restores, per se, it's recovery. Backup and restore are tasks, recovery is a state.
/important nitpick
February 13, 2017 at 8:13 pm
chrisn-585491 - Monday, February 13, 2017 6:21 AMJeff Moden - Sunday, February 12, 2017 7:45 PMForgiveness is certainly divine but I wonder how may people's tone would be different if they were a part of the group that suffered data losses.This brings up a good point, though. If you trust someone with your data, what avenues do you have if they lose your data or suffer a breach? What if you have a small company and ALL of your data (and code) is lost? Will the company you entrusted help you stay in business and pay you for lost business? ... without having to sue them for damages? ... and what can you get out of a dead horse if they go out of business because of it all?
Heh... I'd say it's time to review for all that in any contracts that you may have with 3rd parties. Even if they guarantee recovery (as so many do), you'd better make sure there's a "Plan B" for when they fail. More likely, though, there will be some clause in the contract somewhere that exonerates them from any such responsibility.
A small company I did some consulting for went through that. Ironically, I hate made the suggestion that they needed a "Plan B". The CIO bought it and tried to take it up the chain of command. She was told that she was being "emotional". 3 Days later... BOOM! The data disappeared on one of the critical systems. They did restores but couldn't get the previous week to restore. It took the entire finance department (about 40 people) over a week to recover most of it through source documents, spreadsheets, and other sources. Of course, they didn't recover it all.
The company that failed to provide the necessary correct and viable backups/restoes, even though the victimized company was paying out the nose for such services, wasn't made to suffer a penny of loss because of some stupid clause that everyone overlooked or brushed off as a "would never happen".
Jeff,
In almost every similar situation I've seen, at one point someone has raised a red flag and said, "What about this issue or that?" and were either ignored or ridiculed. I've experienced this multiple times in my career and you learn to notify, document and go on. And I'm usually on the team that has to clean up the resulting mess... 🙁
Most recently I was ignored for months raising a few critical issues. Management admitted they ignored me, but upper management has changed the game, making them responsible if they don't address issues promptly and accurately. So it is getting better... :rolleyes:
I've been through that many times and, unfortunately, continue to go through that. It's taught me a great deal about people and patience. Many times (after me warning them about something, of course), I realize that they can actually recover from the mistake they're about to make. It'll take them some time but that helps drive the lesson home. There are other times when I know they won't be able to recover. With all the due diligence in the world and code examples along with alternative solutions, etc, etc, they still march on down the primrose path and so, even though it costs me a lot of time, I setup for the "save". There are some things (especially when it comes to being ethical/moral) that I simply won't allow. Of course, those are very rare situations and I've only had to suffer through three such things. Two of the companies went bankrupt because of their actions and inactions. A third company was "too big to fail" but a lot of people got hurt. More than half the people in the company got laid off... on the month I predicted... two years before it happened. It's all made me a little sour on "Business Intelligence", which I now consider to be an oxymoron. 😉
--Jeff Moden
Change is inevitable... Change for the better is not.
February 14, 2017 at 7:03 am
Gary Varga - Monday, February 13, 2017 8:23 AMEric M Russell - Monday, February 13, 2017 6:43 AM...Every IT organization, especially smaller ones, should have a 3rd party resource like a local consultant they can call in when they get swamped.Depends on what they can manage cost-wise. Sometimes it just isn't feasible to have someone retained.
If an IT organization is complex enough to warrant 3rd party expertise on occasion, but they can't afford a day or two of 3rd party consulting, then that is a severe issue that needs to be addressed.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
February 14, 2017 at 8:21 am
Eric M Russell - Tuesday, February 14, 2017 7:03 AMGary Varga - Monday, February 13, 2017 8:23 AMEric M Russell - Monday, February 13, 2017 6:43 AM...Every IT organization, especially smaller ones, should have a 3rd party resource like a local consultant they can call in when they get swamped.Depends on what they can manage cost-wise. Sometimes it just isn't feasible to have someone retained.
If an IT organization is complex enough to warrant 3rd party expertise on occasion, but they can't afford a day or two of 3rd party consulting, then that is a severe issue that needs to be addressed.
I don't disagree. It is just that these days the margin has become so thin for some companies that they can no longer survive some issues.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
February 14, 2017 at 9:16 am
Gary Varga - Tuesday, February 14, 2017 8:21 AMEric M Russell - Tuesday, February 14, 2017 7:03 AMGary Varga - Monday, February 13, 2017 8:23 AMEric M Russell - Monday, February 13, 2017 6:43 AM...Every IT organization, especially smaller ones, should have a 3rd party resource like a local consultant they can call in when they get swamped.Depends on what they can manage cost-wise. Sometimes it just isn't feasible to have someone retained.
If an IT organization is complex enough to warrant 3rd party expertise on occasion, but they can't afford a day or two of 3rd party consulting, then that is a severe issue that needs to be addressed.
I don't disagree. It is just that these days the margin has become so thin for some companies that they can no longer survive some issues.
Yeah, when the database server crashes, and the only in-house staff member with SQL Server experience doesn't know all the steps required to recover and get the database back online. That's an issue for which the company will not survive.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
Viewing 15 posts - 1 through 15 (of 15 total)
You must be logged in to reply to this topic. Login to reply