SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Is Rolling Back The Same as Failing?


Is Rolling Back The Same as Failing?

Author
Message
Andy Warren
Andy Warren
SSChampion
SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)SSChampion (11K reputation)

Group: Moderators
Points: 11463 Visits: 2730
Comments posted to this topic are about the item Is Rolling Back The Same as Failing?

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
David.Poole
David.Poole
SSCertifiable
SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)SSCertifiable (7.5K reputation)

Group: General Forum Members
Points: 7500 Visits: 3282
A rollback in a non-production environment should be celebrated as preventing a rollback scenario in the production environment. It means that your pre-production environments have justified their existence!

Both deployments and rollbacks should be rehearsed.

Yes we always kick ourselves when things don't work perfectly first time. Its a matter of professional pride after all.

If it is people feeling sheepish and embarrassed over having to do a rollback that is a good thing. You are working with people with a good attitude.
If it is people feeling "oh well it doesn't matter, it's not production" then that is a bad attitude.

LinkedIn Profile

Newbie on www.simple-talk.com
david.wright-948385
david.wright-948385
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1234 Visits: 979
No situation - especially this one - should be allowed to drag a team down in this way. Technically is was the right thing to do, and it ensured the migration went well.

In my view the problem here is motivation. It is as much the manager's or team leader's responsibility to maintain morale as it is to manage timescales, cost and quality, and it sounds like they were sleeping on the job :-) All it needed was perspective, encouragement and a positive "can do" attitude.
scott mcnitt
scott mcnitt
SSC-Enthusiastic
SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)SSC-Enthusiastic (102 reputation)

Group: General Forum Members
Points: 102 Visits: 436
When you fail more often you'll get used to it Wink

"In theory, there is no difference between theory and practice. In practice, there is."
-Yogi Berra
Todd Townley
Todd Townley
Old Hand
Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)Old Hand (309 reputation)

Group: General Forum Members
Points: 309 Visits: 478
In a test environment? This is exactly what they are for! In hindsight, you may kick yourself for not seeing things that should have been incorporated. But you can celebrate the fact that it was caught at a noncritical stage and can be corrected.

I manage a team that does nothing but one-offs (custom one-time data conversions and miscellaneous custom database scripts). It is the nature of our work that when we run the final result in the production environment, it is typically the first time that code is run in that exact environment. That, in and of itself, carries risk. We do back up the databases prior to execution. If something goes wrong, we evaluate to see if we can correct now or correct later. In a worst-case scenario, we'll have to restore the database, reschedule, and reevaluate.

Prior to running in production, we perform many tests and many rollbacks in a test environment. If we can get the process to run perfectly in the test environment, then we've greatly reduced (but not eliminated) the risk as we move this to the production environment.

A significant aspect of our job is to protect the data. If a process executes against the data in such a way that the data is damaged and cannot be corrected back to it's original state, then a rollback is the right thing to do (if possible).

Long ago in my career (back when it took two hours to back up a database to multiple tapes, and you did double backups because you didn't trust the tapes) I had an experience where a rollback was not possible. It involved a conversion within a financial system. Some data was damaged and could not be corrected. There was no time to restore a database and try again, because this would have brought the system down for 24 hours or more. I spent the next four weeks rebuilding the data as close to original as possible. It wasn't perfect, but the accountants (who were aware of the situation) accepted the result. That was a very expensive lesson.
Jim P.
Jim P.
SSC Eights!
SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)SSC Eights! (901 reputation)

Group: General Forum Members
Points: 901 Visits: 2215
Then there are the rollbacks caused by stuff that didn't happen in the test environment.

We had a software package that we upgraded the DB and the clients in a test environment with no problem.
Then production Monday we upgraded the DB and as the users logged on in the morning, it would install the updated client. We had pretty much automated the whole thing that about the only manual client install was on the terminal servers.

Well we started getting calls that the client install was failing. We go start looking at the clients and it was failing even with manual installs. Our company was one of the "first" installers for upgrades of the SW. So the vendor had no idea what was going on either.

Luckily we had a two week leeway before regulations needed the SW to be upgraded, and only about five clients needed to be rolled back along with a DB restore.

Then it is a matter of troubleshooting. Turned out the prior version had made the desktop shortcut icon as read only. So when the installer went to replace it, it was choking. It took us two days to find the issue.



----------------
Jim P.

A little bit of this and a little byte of that can cause bloatware.
SQLRNNR
SQLRNNR
SSC-Dedicated
SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)SSC-Dedicated (32K reputation)

Group: General Forum Members
Points: 32104 Visits: 18551
I read a blog post on this topic recently.

I like the approach depicted in the article - try to resolve and then rollback after having run there enough troubleshooting to hopefully identify the problem and shortcomings. In addition, I would say there needs to be a go/nogo point in the rollout where the rollback needs to be considered.


http://tjaybelt.blogspot.com/2014/02/be-tenacious.html



Jason AKA CirqueDeSQLeil
I have given a name to my pain...
MCM SQL Server, MVP


SQL RNNR

Posting Performance Based Questions - Gail Shaw

OCTom
OCTom
Hall of Fame
Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)Hall of Fame (3.1K reputation)

Group: General Forum Members
Points: 3109 Visits: 4152
I think it's kind of odd that the team felt badly about this. The team should feel satisfied that their procedures are correct.

Is any of us perfect that everything we do works right the first time? That's we have have test environments. You were able to rollback, discuss the issue, find a resolution, and test again.
david.wright-948385
david.wright-948385
Ten Centuries
Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)Ten Centuries (1.2K reputation)

Group: General Forum Members
Points: 1234 Visits: 979
OCTom (3/4/2014)
I think it's kind of odd that the team felt badly about this. The team should feel satisfied that their procedures are correct.

Is any of us perfect that everything we do works right the first time? That's we have have test environments. You were able to rollback, discuss the issue, find a resolution, and test again.


+1. See my comment above about motivation/morale.
don c-309367
don c-309367
Forum Newbie
Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)Forum Newbie (6 reputation)

Group: General Forum Members
Points: 6 Visits: 10
I'm going to disagree. I think in this situation is was a failure. Sure you can tell yourself that it should be caught now (and it should) but don't consider a rollback successful after you had reviews and meetings and still missed something. That is a failure. Of course, you won't do the same thing again and you'll learn but a 'good' team will feel bad that they missed something. You (the team leader) can celebrate that the team cared enough to feel bad and cares about the work but don't make an excuse for the failure.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search