SQL Clone
SQLServerCentral is supported by Redgate
Log in  ::  Register  ::  Not logged in

Don't Throw Something New In At Production Deployment

This post comes from a conversation I had with a developer last week as well as my experiences over the weekend. My developer friend and I were hashing over war stories. He's a senior development/application architect level and he's seen his fair share of poor production deployments, one of which was finding a totally new architecture (untested) in a about to go to production deployment because a junior developer decided to show a colleague about this new, fancy technology he just learned about. If I remember the war story right, the original code had cleared QA and was about to go to production when the junior developer worked his "magic." Good thing he was intercepted because his changes revealed several things:

  • The changes didn't work.
  • The changes revealed the developer didn't actually understand the new technology.
  • The junior developer would put a deployment at risk to obtain personal "glory."
  • The build process needed significant strengthening.

This one was quite surprising to me. Having been a developer for a number of years, I wouldn't personally roll any new code into a production deployment after the initial code passed QA testing unless we found something significantly wrong and we just couldn't wait. "Just couldn't wait" is usually defined as regulatory requirement, but it could also be a significant feature release where you're trying to beat a competitor. However, in both cases I can count on one hand the number of times I've been part of a project where this kind of change was done. It should be common sense how much risk one incurs by doing something like this. But yet, that one developer did it. And as a result, the company not only lost his hours as he re-architected part of the production deployment, they lost the hours of the senior developers that then had to go in to check the code and ensure everything was rolled back properly and the right build was put forth for the production deployment.

Now to this weekend. I wasn't directly supporting a particular performance at my church, but got pulled into it because there were issues with the group's computers and they didn't know how to solve them. As it turns out, they had bought a brand new laptop with a brand new operating system they weren't familiar with (WIndows 7) and had gone from an older version to a newer version of a key piece of software (2.5 to 3.5). That key piece of software is what drives all the sound and video files for the performance. And, of course, the people involved had never used Windows 7 nor were they up to speed with the latest version of the software. In fact, they had downloaded the upgrade the night before because they wanted to switch laptops and the old version (2.5) didn't run on Windows 7. Actually, the new version (3.5) doesn't run perfectly on Windows 7, either, but apparently they were licensed for it and not the latest version (4.0), which does run on Windows 7. It runs well enough, but there are enough glitches that I would upgrade further.

At most churches this group would have went to, they would have been in some trouble. I don't mean with the operating system. Windows 7 is not that big of a deal for anyone who is a power user or works in IT; it's just a matter of actually looking a bit more carefully at a couple of screens when setting up that secondary video output because some of the navigation has changed. But that software isn't straight forward and there are places where you have to look at settings inside that software to toggle important things like desired display settings. It doesn't just conform to whatever the monitor is. The group clearly had no idea that those settings existed (I don't believe they were in the older version) or that those settings were causing all of their media to display incorrectly. We just happen to run the software they were using at the 3.5 version. So we knew what tweaks had to be made, and we were able to get them up and running. It's a good thing we were familiar with the software because I don't believe the vendor has weekend support hours and the first performance was last night (Sunday). Had that not been the case, they would have had issues putting on the presentation. I am assuming they would have dropped back to the old laptop, but they didn't feel it was "stable" but didn't elaborate as to why. I didn't bother to ask why, as I had ministry preparation that I needed to get done which coincided with the performance. And while we're not takling about putting together a new software build, what they did was essentially the same thing... throwing something new in at production deployment (in this case, a performance).

The moral of these two stories is a simple one: don't roll a change into production at the last minute which hasn't been tested. I know there are exceptions. There are always exceptions. But that's what they are supposed to be: exceptions. The rule should be everything that goes in has gone through a testing cycle. It's just the right way to do things. And it saves hours, stress, and reputations to do it the right way.


K. Brian Kelley - Databases, Infrastructure, and Security

IT Security, MySQL, Perl, SQL Server, and Windows technologies.


Posted by AjarnMark on 22 March 2010

WHOLEHEARTEDLY agree with this Brian!  This is exactly why, when I took over at my current job, I implemented a new formal process for deployment that involves creation of an explicit build package and it gets moved from BVT to QA to PROD and is NEVER EVER EVER modified along the way.  The rule is that if ANYTHING changes, even a simple typo on a label, a new package is built (and given a new number) and it has to go all the way back to the start and make its way through the process again.

Posted by RL on 26 March 2010

that is common practice at my day job.  but on the weekends when i too support the PC's at my church we don't have a QA box in the sound/video booth, but i think i will recommend it before we upgrade to win7 and media shout 4 ( i am assuming that is the software you were referring to....)

Posted by peter.j.hanlon on 26 March 2010

AjarnMark said:

WHOLEHEARTEDLY agree with this Brian!

Do you?  What Brian is suggesting is that there are exceptions, when the benefit outwieghs the risks, and you do put something in that hasn't been through the usual level of testing.  But that they should remain exceptions.  Your reply implies you don't accept any exceptions.

I agree with Brians approach and I think the degree to which it applies varies by industry sector.  

Highly regulated industries like the financial sector, have a need for much more rigourous testing than, unregulated but fiercely competitive markets, where being first to market is important/vital.

Fortunately for me, I work in retail which largely follows the latter of the above two scenarios.  I therefore have the option to put through unverified code if the scenario warrants it.  My success depends on the quality of those judgement calls.

I openely admit it, and make no apology for it, I have put untested code into production, and I'll do it again.  But I hope I don't have to do it very often...

Leave a Comment

Please register or log in to leave a comment.