• I had a similar experience last year, except it was a production install we had to roll back. And yes, the team was somewhat depressed initially about that - we had planned and tested and done "everything right". Well, almost, as it turned out.

    Testing of a specific feature component had been missed in the QA environment, but was performed in the production validation step. By accident so to speak, which tells me our test planning needs to be improved.

    Any set of professionals will be upset when something goes wrong. That's part of what makes us professionals, and is also what drives us to improve. It's OKAY to be depressed, bummed out, even angry, as long as that gets translated into the energy to fix the issue and improve the process.

    How you react, as a manger, team leader, scrum master, teammate, whatever, become very important at that time (IMHO). I do think it's important to remind the team that the rollback itself is a success - we prevented buggy code from existing in production. And that is our job! We prefer to do it earlier in the process. That's what I told my team, and depression gave way to thoughtfulness, then confidence.

    That was the important take-away for me. I suppose I could have beat the development team up for not enough unit testing, or the QA testers for not having the scenario in their test plan. But what would that have accomplished? I don't need team members with an inferiority complex or who point fingers at other team members ("it was their fault, I'm off the hook") - I need competent, confident team members who work together. By pointing out the positive aspects of the roll-back, that's what I got. The whole team spent the next week correcting the problem and at the next implementation window we successfully installed a major upgrade that's been working smoothly ever since.

    So was the rollback indicative of success or failure? It depends on which goal you're talking about. The TEAM was successful; a small (but important) part of our PROCESS failed.

    - we successfully prevented non-working code from existing in production

    - all team members were aware of which part of our PROCESS failed

    - all team members worked enthusiastically to correct the issue

    - we successfully implemented a week later

    - we successfully corrected our process and all team members are now more aware of what is needed process-wise to increase the likelihood of a successful implementation

    That's a lot of success, and it's only a failure if you fail to learn from it. That's my philosophy, anyway.


    Here there be dragons...,

    Steph Brown