Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase ««123»»

Is Rolling Back The Same as Failing? Expand / Collapse
Author
Message
Posted Tuesday, March 4, 2014 8:02 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Saturday, September 13, 2014 4:06 AM
Points: 988, Visits: 804
don c-309367 (3/4/2014)
I'm going to disagree. ... don't make an excuse for the failure.

Understandable vs acceptable.
Post #1547351
Posted Tuesday, March 4, 2014 8:03 AM
Right there with Babe

Right there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with Babe

Group: General Forum Members
Last Login: Friday, June 20, 2014 8:23 AM
Points: 738, Visits: 1,305
I recently went through this. We have a somewhat older ASP.NET website, that doesn't want to work with Internet Explorer 11. But more and more of our external users are getting new PCs that come with IE 11 installed. I had made changes to the web site so that it would work. I tested it as thoroughly as I could and then pushed it out to production, but that turned out to be a disaster. Not only did it not work for IE 11, if didn't work for any browser. I spent an hour or so trying to get it to work, and finally decided to rollback. Better that users using IE 11 couldn't use the site, than no one at all couldn't use it.

It wasn't a satisfactory choice, but I felt it the only one open to me. I'm still puzzled as to why it didn't work, because it worked perfectly in testing. And unfortunately for me, as often happens in my job, other more urgent priorities have come up requiring my attention. This website/IE11 issue has been pushed to the back burner. However, I intend to put up a staging site on a server that's not visible to the outside world, where we can test it again, before putting it out for the rest of the world to use. One that will mirror the production web server as closely as possible.

When I have the time to get back to it.


Kindest Regards,

Rod
Connect with me on LinkedIn.
Post #1547352
Posted Tuesday, March 4, 2014 9:32 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Thursday, September 25, 2014 3:50 PM
Points: 27, Visits: 733
To use Monty Python's Holy Grail as a metaphor, every major effort in a database requires a 'run away' option.
Post #1547407
Posted Tuesday, March 4, 2014 10:36 AM
SSChasing Mays

SSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing Mays

Group: General Forum Members
Last Login: Wednesday, September 24, 2014 12:52 PM
Points: 610, Visits: 975
I had a similar experience last year, except it was a production install we had to roll back. And yes, the team was somewhat depressed initially about that - we had planned and tested and done "everything right". Well, almost, as it turned out.

Testing of a specific feature component had been missed in the QA environment, but was performed in the production validation step. By accident so to speak, which tells me our test planning needs to be improved.

Any set of professionals will be upset when something goes wrong. That's part of what makes us professionals, and is also what drives us to improve. It's OKAY to be depressed, bummed out, even angry, as long as that gets translated into the energy to fix the issue and improve the process.

How you react, as a manger, team leader, scrum master, teammate, whatever, become very important at that time (IMHO). I do think it's important to remind the team that the rollback itself is a success - we prevented buggy code from existing in production. And that is our job! We prefer to do it earlier in the process. That's what I told my team, and depression gave way to thoughtfulness, then confidence.

That was the important take-away for me. I suppose I could have beat the development team up for not enough unit testing, or the QA testers for not having the scenario in their test plan. But what would that have accomplished? I don't need team members with an inferiority complex or who point fingers at other team members ("it was their fault, I'm off the hook") - I need competent, confident team members who work together. By pointing out the positive aspects of the roll-back, that's what I got. The whole team spent the next week correcting the problem and at the next implementation window we successfully installed a major upgrade that's been working smoothly ever since.

So was the rollback indicative of success or failure? It depends on which goal you're talking about. The TEAM was successful; a small (but important) part of our PROCESS failed.

- we successfully prevented non-working code from existing in production
- all team members were aware of which part of our PROCESS failed
- all team members worked enthusiastically to correct the issue
- we successfully implemented a week later
- we successfully corrected our process and all team members are now more aware of what is needed process-wise to increase the likelihood of a successful implementation

That's a lot of success, and it's only a failure if you fail to learn from it. That's my philosophy, anyway.



Here there be dragons...,

Steph Brown
Post #1547442
Posted Tuesday, March 4, 2014 11:59 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Thursday, June 26, 2014 10:39 AM
Points: 14, Visits: 47
Part of a good migration plan is the Roll Back or Back Out plan and the fact that you had one was commendable in and of itself. You would be surprised at how many times I have seen migrations without a Roll Back plan. Unforeseeable things happen all the time, especially in an environment where there are many people and even teams working a migration, and that is what the Back Out plan is for. Now, the fact that this was caught in a Test environment, to me, should be considered a Success. The success is that we uncovered an issue that hadn't been thought of or covered in the Development phase and that we prevented this from happening in the Production environment. Whenever doing these things we need to have these "Dry Runs" in the Test environment, which should be as close to mimicking the Production environment as possible, and as many Dry Runs as it takes before we have something that we are 99.99% sure is going to work in our Production environment because we worked out ALL the kinks in the Test environment.

Great Job to you and your team(s) for having the Back Out plan and for executing it when it was necessary.
Hopefully your next Dry Run will be the one that tells you that you are now ready for a push to Production.

Merrill
Post #1547482
Posted Tuesday, March 4, 2014 12:42 PM
Right there with Babe

Right there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with BabeRight there with Babe

Group: General Forum Members
Last Login: Friday, June 20, 2014 8:23 AM
Points: 738, Visits: 1,305
I like your analysis, Stephanie. Very good.

Kindest Regards,

Rod
Connect with me on LinkedIn.
Post #1547501
Posted Wednesday, March 5, 2014 2:02 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 2:35 AM
Points: 5,465, Visits: 3,239
I am inclined to disagree with everyone and yet I feel that I agree too. I guess I am in a contrary mood today.

Was it a success? Yes.
Was it a failure? Yes.

The outcome was satisfactory. No more, no less.

The process, as well as the execution of it, was a success as, due to the rollback, the process was successfully completed i.e. to a known process completion state.
The aspect overlooked was a failure but, as others have pointed out, may not be too bad if learnt from.

A foreseeable failure occurred but recovered to a operational state in a managed way. Outcome: satisfactory.


Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
Post #1547672
Posted Wednesday, March 5, 2014 7:44 AM
Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Friday, September 19, 2014 10:12 AM
Points: 1,234, Visits: 2,218
mnelson-699385 (3/4/2014)
Part of a good migration plan is the Roll Back or Backout

We take the paranoia a stage further (financial institution :))- not just a Backout plan, but a Proven Backout Plan. We always do: deploy, test, rollback, test, redeploy, test as an integral part of pre-production testing.

Discovering something wrong during a production implementation is an inconvenience; subsequently discovering that you can't get back is a disaster.
Post #1547796
Posted Thursday, April 3, 2014 7:14 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Tuesday, April 15, 2014 8:03 AM
Points: 825, Visits: 319
The old adage, you learn by your mistakes, is worth bearing in mind here.
Post #1557961
Posted Thursday, April 3, 2014 8:35 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Friday, September 19, 2014 2:15 AM
Points: 213, Visits: 854
I think it's not a purely intellectual decision, there are a lot of external factors that could tip the balance one way or the other. For example can money influence the decision?
- If you were being paid hourly you might benefit from being cautious.
- If it was a contract with a fixed cost you may have to bear it, resulting in a loss.
- If you were permanent money likely would not have made any difference.

The other filter I put things through is: what is everyone else in the organisation doing? If you're working with cowboy coders pushing out a new build to clients every day then why should your database team be held to a higher standard? Your caution may become a point of ridicule and derision from others, and it would be best to don the cowboy hat and push on. That's what the company expects, supports, and rewards.

Conversely, if you're working for banks, health care, or the military, you might take a much more cautious approach, and nobody is going to attack you for it because that's the standard that is expected.
Post #1558023
« Prev Topic | Next Topic »

Add to briefcase ««123»»

Permissions Expand / Collapse