We had an interesting discussion about deployments in databases and how you go forward or back from the point when you discover a problem. You can watch the episode and see what you think, but one thing that Pat asked was about rolling back a broken release.
I’ve seen a few broken releases that were rolled back immediately in my career. However, in a lot of cases, I’ve also been a part of semi-failed releases where we had to roll things forward.
I learned early on to smoke test the system post-deployment. Either get an account, or run known queries after applying a patch to ensure things worked BEFORE I let someone know the deployment was complete.
In one case, we applied a patch, restarted the application and started receiving errors immediately. We knew then that whatever things had changed, those changes were not in sync with the application. In this case, the application code and the database code had a small typo in a name, but late at night, we didn’t realize it was this simple.
In the days of outage windows, we couldn’t debug for long, so I decided immediately that getting back online was important. We replaced the new .exe with the old one and I looked at each of the database commands and wrote a reversing one to reset the database. Since the system was down, there weren’t any data changes.
In recent years, a few clients have had an easier time as they use feature flags to enable new functionality. When they’ve had an issue like the wrong name in code, they just flip the toggle to disable the feature. This rolls back the code.
I highly recommend using feature flags to anyone working with database software changes. Coupled with zero-downtime architectures for database changes, this lets us rollback things quickly.