Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 123»»»

Is Rolling Back The Same as Failing? Expand / Collapse
Author
Message
Posted Monday, March 3, 2014 9:14 PM
SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: Moderators
Last Login: Yesterday @ 7:46 AM
Points: 6,784, Visits: 1,898
Comments posted to this topic are about the item Is Rolling Back The Same as Failing?

Andy
SQLAndy - My Blog!
Connect with me on LinkedIn
Follow me on Twitter
Post #1547152
Posted Tuesday, March 4, 2014 1:39 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, September 26, 2014 2:45 PM
Points: 2,907, Visits: 1,829
A rollback in a non-production environment should be celebrated as preventing a rollback scenario in the production environment. It means that your pre-production environments have justified their existence!

Both deployments and rollbacks should be rehearsed.

Yes we always kick ourselves when things don't work perfectly first time. Its a matter of professional pride after all.

If it is people feeling sheepish and embarrassed over having to do a rollback that is a good thing. You are working with people with a good attitude.
If it is people feeling "oh well it doesn't matter, it's not production" then that is a bad attitude.


LinkedIn Profile
Newbie on www.simple-talk.com
Post #1547193
Posted Tuesday, March 4, 2014 4:01 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Saturday, September 13, 2014 4:06 AM
Points: 988, Visits: 804
No situation - especially this one - should be allowed to drag a team down in this way. Technically is was the right thing to do, and it ensured the migration went well.

In my view the problem here is motivation. It is as much the manager's or team leader's responsibility to maintain morale as it is to manage timescales, cost and quality, and it sounds like they were sleeping on the job All it needed was perspective, encouragement and a positive "can do" attitude.
Post #1547233
Posted Tuesday, March 4, 2014 5:35 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Thursday, September 18, 2014 5:39 AM
Points: 41, Visits: 378
When you fail more often you'll get used to it ;)

"In theory, there is no difference between theory and practice. In practice, there is."
-Yogi Berra
Post #1547266
Posted Tuesday, March 4, 2014 5:55 AM
SSC Veteran

SSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC VeteranSSC Veteran

Group: General Forum Members
Last Login: Thursday, September 18, 2014 2:56 PM
Points: 246, Visits: 383
In a test environment? This is exactly what they are for! In hindsight, you may kick yourself for not seeing things that should have been incorporated. But you can celebrate the fact that it was caught at a noncritical stage and can be corrected.

I manage a team that does nothing but one-offs (custom one-time data conversions and miscellaneous custom database scripts). It is the nature of our work that when we run the final result in the production environment, it is typically the first time that code is run in that exact environment. That, in and of itself, carries risk. We do back up the databases prior to execution. If something goes wrong, we evaluate to see if we can correct now or correct later. In a worst-case scenario, we'll have to restore the database, reschedule, and reevaluate.

Prior to running in production, we perform many tests and many rollbacks in a test environment. If we can get the process to run perfectly in the test environment, then we've greatly reduced (but not eliminated) the risk as we move this to the production environment.

A significant aspect of our job is to protect the data. If a process executes against the data in such a way that the data is damaged and cannot be corrected back to it's original state, then a rollback is the right thing to do (if possible).

Long ago in my career (back when it took two hours to back up a database to multiple tapes, and you did double backups because you didn't trust the tapes) I had an experience where a rollback was not possible. It involved a conversion within a financial system. Some data was damaged and could not be corrected. There was no time to restore a database and try again, because this would have brought the system down for 24 hours or more. I spent the next four weeks rebuilding the data as close to original as possible. It wasn't perfect, but the accountants (who were aware of the situation) accepted the result. That was a very expensive lesson.
Post #1547274
Posted Tuesday, March 4, 2014 6:17 AM


SSChasing Mays

SSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing MaysSSChasing Mays

Group: General Forum Members
Last Login: Tuesday, September 23, 2014 7:42 PM
Points: 635, Visits: 2,215
Then there are the rollbacks caused by stuff that didn't happen in the test environment.

We had a software package that we upgraded the DB and the clients in a test environment with no problem.
Then production Monday we upgraded the DB and as the users logged on in the morning, it would install the updated client. We had pretty much automated the whole thing that about the only manual client install was on the terminal servers.

Well we started getting calls that the client install was failing. We go start looking at the clients and it was failing even with manual installs. Our company was one of the "first" installers for upgrades of the SW. So the vendor had no idea what was going on either.

Luckily we had a two week leeway before regulations needed the SW to be upgraded, and only about five clients needed to be rolled back along with a DB restore.

Then it is a matter of troubleshooting. Turned out the prior version had made the desktop shortcut icon as read only. So when the installer went to replace it, it was choking. It took us two days to find the issue.




----------------
Jim P.

A little bit of this and a little byte of that can cause bloatware.
Post #1547286
Posted Tuesday, March 4, 2014 7:32 AM


SSCoach

SSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoachSSCoach

Group: General Forum Members
Last Login: Yesterday @ 11:01 PM
Points: 17,694, Visits: 15,561
I read a blog post on this topic recently.

I like the approach depicted in the article - try to resolve and then rollback after having run there enough troubleshooting to hopefully identify the problem and shortcomings. In addition, I would say there needs to be a go/nogo point in the rollout where the rollback needs to be considered.


http://tjaybelt.blogspot.com/2014/02/be-tenacious.html




Jason AKA CirqueDeSQLeil
I have given a name to my pain...
MCM SQL Server


SQL RNNR

Posting Performance Based Questions - Gail Shaw
Posting Data Etiquette - Jeff Moden
Hidden RBAR - Jeff Moden
VLFs and the Tran Log - Kimberly Tripp
Post #1547332
Posted Tuesday, March 4, 2014 7:39 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Friday, September 26, 2014 3:27 PM
Points: 2,575, Visits: 3,872
I think it's kind of odd that the team felt badly about this. The team should feel satisfied that their procedures are correct.

Is any of us perfect that everything we do works right the first time? That's we have have test environments. You were able to rollback, discuss the issue, find a resolution, and test again.

Post #1547335
Posted Tuesday, March 4, 2014 7:44 AM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Saturday, September 13, 2014 4:06 AM
Points: 988, Visits: 804
OCTom (3/4/2014)
I think it's kind of odd that the team felt badly about this. The team should feel satisfied that their procedures are correct.

Is any of us perfect that everything we do works right the first time? That's we have have test environments. You were able to rollback, discuss the issue, find a resolution, and test again.


+1. See my comment above about motivation/morale.
Post #1547339
Posted Tuesday, March 4, 2014 7:57 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, April 3, 2014 7:19 AM
Points: 2, Visits: 10
I'm going to disagree. I think in this situation is was a failure. Sure you can tell yourself that it should be caught now (and it should) but don't consider a rollback successful after you had reviews and meetings and still missed something. That is a failure. Of course, you won't do the same thing again and you'll learn but a 'good' team will feel bad that they missed something. You (the team leader) can celebrate that the team cared enough to feel bad and cares about the work but don't make an excuse for the failure.
Post #1547345
« Prev Topic | Next Topic »

Add to briefcase 123»»»

Permissions Expand / Collapse