SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Is Rolling Back The Same as Failing?


Is Rolling Back The Same as Failing?

Author
Message
david.wright-948385
david.wright-948385
SSCommitted
SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)

Group: General Forum Members
Points: 1708 Visits: 991
don c-309367 (3/4/2014)
I'm going to disagree. ... don't make an excuse for the failure.

Understandable vs acceptable.
Rod
Rod
SSCertifiable
SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)

Group: General Forum Members
Points: 7359 Visits: 2186
I recently went through this. We have a somewhat older ASP.NET website, that doesn't want to work with Internet Explorer 11. But more and more of our external users are getting new PCs that come with IE 11 installed. I had made changes to the web site so that it would work. I tested it as thoroughly as I could and then pushed it out to production, but that turned out to be a disaster. Not only did it not work for IE 11, if didn't work for any browser. I spent an hour or so trying to get it to work, and finally decided to rollback. Better that users using IE 11 couldn't use the site, than no one at all couldn't use it.

It wasn't a satisfactory choice, but I felt it the only one open to me. I'm still puzzled as to why it didn't work, because it worked perfectly in testing. And unfortunately for me, as often happens in my job, other more urgent priorities have come up requiring my attention. This website/IE11 issue has been pushed to the back burner. However, I intend to put up a staging site on a server that's not visible to the outside world, where we can test it again, before putting it out for the rest of the world to use. One that will mirror the production web server as closely as possible.

When I have the time to get back to it.

Kindest Regards,Rod
Connect with me on LinkedIn.
dlander525
dlander525
SSC Rookie
SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)SSC Rookie (48 reputation)

Group: General Forum Members
Points: 48 Visits: 914
To use Monty Python's Holy Grail as a metaphor, every major effort in a database requires a 'run away' option.
Stephanie J Brown
Stephanie J Brown
Ten Centuries
Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)

Group: General Forum Members
Points: 1284 Visits: 1103
I had a similar experience last year, except it was a production install we had to roll back. And yes, the team was somewhat depressed initially about that - we had planned and tested and done "everything right". Well, almost, as it turned out.

Testing of a specific feature component had been missed in the QA environment, but was performed in the production validation step. By accident so to speak, which tells me our test planning needs to be improved.

Any set of professionals will be upset when something goes wrong. That's part of what makes us professionals, and is also what drives us to improve. It's OKAY to be depressed, bummed out, even angry, as long as that gets translated into the energy to fix the issue and improve the process.

How you react, as a manger, team leader, scrum master, teammate, whatever, become very important at that time (IMHO). I do think it's important to remind the team that the rollback itself is a success - we prevented buggy code from existing in production. And that is our job! We prefer to do it earlier in the process. That's what I told my team, and depression gave way to thoughtfulness, then confidence.

That was the important take-away for me. I suppose I could have beat the development team up for not enough unit testing, or the QA testers for not having the scenario in their test plan. But what would that have accomplished? I don't need team members with an inferiority complex or who point fingers at other team members ("it was their fault, I'm off the hook") - I need competent, confident team members who work together. By pointing out the positive aspects of the roll-back, that's what I got. The whole team spent the next week correcting the problem and at the next implementation window we successfully installed a major upgrade that's been working smoothly ever since.

So was the rollback indicative of success or failure? It depends on which goal you're talking about. The TEAM was successful; a small (but important) part of our PROCESS failed.

- we successfully prevented non-working code from existing in production
- all team members were aware of which part of our PROCESS failed
- all team members worked enthusiastically to correct the issue
- we successfully implemented a week later
- we successfully corrected our process and all team members are now more aware of what is needed process-wise to increase the likelihood of a successful implementation

That's a lot of success, and it's only a failure if you fail to learn from it. That's my philosophy, anyway.


Here there be dragons...,

Steph Brown
mnelson-699385
mnelson-699385
Grasshopper
Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)Grasshopper (16 reputation)

Group: General Forum Members
Points: 16 Visits: 47
Part of a good migration plan is the Roll Back or Back Out plan and the fact that you had one was commendable in and of itself. You would be surprised at how many times I have seen migrations without a Roll Back plan. Unforeseeable things happen all the time, especially in an environment where there are many people and even teams working a migration, and that is what the Back Out plan is for. Now, the fact that this was caught in a Test environment, to me, should be considered a Success. The success is that we uncovered an issue that hadn't been thought of or covered in the Development phase and that we prevented this from happening in the Production environment. Whenever doing these things we need to have these "Dry Runs" in the Test environment, which should be as close to mimicking the Production environment as possible, and as many Dry Runs as it takes before we have something that we are 99.99% sure is going to work in our Production environment because we worked out ALL the kinks in the Test environment.

Great Job to you and your team(s) for having the Back Out plan and for executing it when it was necessary.
Hopefully your next Dry Run will be the one that tells you that you are now ready for a push to Production.

Merrill
Rod
Rod
SSCertifiable
SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)SSCertifiable (7.4K reputation)

Group: General Forum Members
Points: 7359 Visits: 2186
I like your analysis, Stephanie. Very good.

Kindest Regards,Rod
Connect with me on LinkedIn.
Gary Varga
Gary Varga
One Orange Chip
One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)One Orange Chip (27K reputation)

Group: General Forum Members
Points: 27741 Visits: 6556
I am inclined to disagree with everyone and yet I feel that I agree too. I guess I am in a contrary mood today.

Was it a success? Yes.
Was it a failure? Yes.

The outcome was satisfactory. No more, no less.

The process, as well as the execution of it, was a success as, due to the rollback, the process was successfully completed i.e. to a known process completion state.
The aspect overlooked was a failure but, as others have pointed out, may not be too bad if learnt from.

A foreseeable failure occurred but recovered to a operational state in a managed way. Outcome: satisfactory.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
Andrew Watson-478275
Andrew Watson-478275
SSCrazy
SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)SSCrazy (2.2K reputation)

Group: General Forum Members
Points: 2221 Visits: 2756
mnelson-699385 (3/4/2014)
Part of a good migration plan is the Roll Back or Backout

We take the paranoia a stage further (financial institution Smile)- not just a Backout plan, but a Proven Backout Plan. We always do: deploy, test, rollback, test, redeploy, test as an integral part of pre-production testing.

Discovering something wrong during a production implementation is an inconvenience; subsequently discovering that you can't get back is a disaster.
marlon.seton
marlon.seton
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1157 Visits: 319
The old adage, you learn by your mistakes, is worth bearing in mind here.
Cody Konior
Cody Konior
Ten Centuries
Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)Ten Centuries (1.1K reputation)

Group: General Forum Members
Points: 1091 Visits: 1111
I think it's not a purely intellectual decision, there are a lot of external factors that could tip the balance one way or the other. For example can money influence the decision?
- If you were being paid hourly you might benefit from being cautious.
- If it was a contract with a fixed cost you may have to bear it, resulting in a loss.
- If you were permanent money likely would not have made any difference.

The other filter I put things through is: what is everyone else in the organisation doing? If you're working with cowboy coders pushing out a new build to clients every day then why should your database team be held to a higher standard? Your caution may become a point of ridicule and derision from others, and it would be best to don the cowboy hat and push on. That's what the company expects, supports, and rewards.

Conversely, if you're working for banks, health care, or the military, you might take a much more cautious approach, and nobody is going to attack you for it because that's the standard that is expected.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search