March 5, 2014 at 2:02 am
I am inclined to disagree with everyone and yet I feel that I agree too. I guess I am in a contrary mood today.
Was it a success? Yes.
Was it a failure? Yes.
The outcome was satisfactory. No more, no less.
The process, as well as the execution of it, was a success as, due to the rollback, the process was successfully completed i.e. to a known process completion state.
The aspect overlooked was a failure but, as others have pointed out, may not be too bad if learnt from.
A foreseeable failure occurred but recovered to a operational state in a managed way. Outcome: satisfactory.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
March 5, 2014 at 7:44 am
mnelson-699385 (3/4/2014)
Part of a good migration plan is the Roll Back or Backout
We take the paranoia a stage further (financial institution :))- not just a Backout plan, but a Proven Backout Plan. We always do: deploy, test, rollback, test, redeploy, test as an integral part of pre-production testing.
Discovering something wrong during a production implementation is an inconvenience; subsequently discovering that you can't get back is a disaster.
April 3, 2014 at 7:14 am
The old adage, you learn by your mistakes, is worth bearing in mind here.
April 3, 2014 at 8:35 am
I think it's not a purely intellectual decision, there are a lot of external factors that could tip the balance one way or the other. For example can money influence the decision?
- If you were being paid hourly you might benefit from being cautious.
- If it was a contract with a fixed cost you may have to bear it, resulting in a loss.
- If you were permanent money likely would not have made any difference.
The other filter I put things through is: what is everyone else in the organisation doing? If you're working with cowboy coders pushing out a new build to clients every day then why should your database team be held to a higher standard? Your caution may become a point of ridicule and derision from others, and it would be best to don the cowboy hat and push on. That's what the company expects, supports, and rewards.
Conversely, if you're working for banks, health care, or the military, you might take a much more cautious approach, and nobody is going to attack you for it because that's the standard that is expected.
April 5, 2014 at 8:05 am
Cody, that's a great point. I'll think on that. Just confirms for me that having a plan in place for making the decision is just as important as the rollback plan itself.
December 14, 2017 at 2:08 am
New to this topic. Rollback is not failing. If there was no rollback option advances going forward would slow to a standstill with people terrified of change!
December 14, 2017 at 4:42 am
A rollback in Production is a failure. A rollback in a sub-Prod environment is a success. You've used your environments to find an issue that would have burned you in a Prod deployment.
December 14, 2017 at 4:58 am
Victor Kirkpatrick - Thursday, December 14, 2017 4:42 AMA rollback in Production is a failure. A rollback in a sub-Prod environment is a success. You've used your environments to find an issue that would have burned you in a Prod deployment.
Thank you! What I should really have added to my post! 🙂
December 14, 2017 at 7:14 am
Failure in a test environment? That's why it's called *testing*. 🙂
There's a reason QA environments exist. The author's anecdote was a glaring example of why we have QA environments.
Your team missed something. It happens. That's why QA is a thing. It DID NOT happen in production. This is not a failure, this is the system acting exactly as designed.
Sure, kick yourselves for obvious stupidity 🙂 but pat yourselves on the back for not being stupid in production.
December 14, 2017 at 8:23 am
mjh 45389 - Thursday, December 14, 2017 2:08 AMNew to this topic. Rollback is not failing. If there was no rollback option advances going forward would slow to a standstill with people terrified of change!
So you wouldn't mind a rollback occurring on the process that builds your paycheck, right? 😉 As recently stated, rollbacks in production ARE failures and may indicate a bigger problem associated with the entire process of code development, testing, and deployment.
Rollbacks in test are a development "failure" but such things happen because no one can write perfect code all the time for all the scenarios (that's not an excuse for getting lazy, though). The purpose of testing (QA/UAT, etc) is to test for such things before they get to prod so it's not a "failure" if it's caught in a test environment. Testing did what it was supposed to do and the people that developed the code just need to fix it.
But, the bottom line is (in my humble opinion), any and all rollbacks in production are, in fact, a failure.
--Jeff Moden
Change is inevitable... Change for the better is not.
December 14, 2017 at 3:49 pm
Anyone who says he/she has never had a failure is not being honest with themselves. We all have failures from time to time. The important thing is to be able to handle the failure with grace.
Recover from the failure and get everything working again. Acknowledge the failure and why it happened, document it fully so it will not happen again.
Just look at Apollo 13, all of the engineers worked together to deal with a failure and found a way to bring our men back home, today that failure is also one of NASA's greatest accomplishments.
I seem to remember a quote "if you have not made a mistake you have not done anything" don't know who said it but I often tell it to the young ones who have made a mistake and feel bad about it.
December 14, 2017 at 9:06 pm
Frank W Fulton Jr - Thursday, December 14, 2017 3:49 PMAnyone who says he/she has never had a failure is not being honest with themselves. We all have failures from time to time. The important thing is to be able to handle the failure with grace.
Recover from the failure and get everything working again. Acknowledge the failure and why it happened, document it fully so it will not happen again.
Just look at Apollo 13, all of the engineers worked together to deal with a failure and found a way to bring our men back home, today that failure is also one of NASA's greatest accomplishments.
I seem to remember a quote "if you have not made a mistake you have not done anything" don't know who said it but I often tell it to the young ones who have made a mistake and feel bad about it.
My old boss used to say, "If you never make a mistake, it's because I'm not pushing you hard enough".
--Jeff Moden
Change is inevitable... Change for the better is not.
Viewing 13 posts - 16 through 27 (of 27 total)
You must be logged in to reply to this topic. Login to reply