• In a test environment? This is exactly what they are for! In hindsight, you may kick yourself for not seeing things that should have been incorporated. But you can celebrate the fact that it was caught at a noncritical stage and can be corrected.

    I manage a team that does nothing but one-offs (custom one-time data conversions and miscellaneous custom database scripts). It is the nature of our work that when we run the final result in the production environment, it is typically the first time that code is run in that exact environment. That, in and of itself, carries risk. We do back up the databases prior to execution. If something goes wrong, we evaluate to see if we can correct now or correct later. In a worst-case scenario, we'll have to restore the database, reschedule, and reevaluate.

    Prior to running in production, we perform many tests and many rollbacks in a test environment. If we can get the process to run perfectly in the test environment, then we've greatly reduced (but not eliminated) the risk as we move this to the production environment.

    A significant aspect of our job is to protect the data. If a process executes against the data in such a way that the data is damaged and cannot be corrected back to it's original state, then a rollback is the right thing to do (if possible).

    Long ago in my career (back when it took two hours to back up a database to multiple tapes, and you did double backups because you didn't trust the tapes) I had an experience where a rollback was not possible. It involved a conversion within a financial system. Some data was damaged and could not be corrected. There was no time to restore a database and try again, because this would have brought the system down for 24 hours or more. I spent the next four weeks rebuilding the data as close to original as possible. It wasn't perfect, but the accountants (who were aware of the situation) accepted the result. That was a very expensive lesson.