The Body in the Trunk

When one considers that the primary purpose of a modern Source Control system is to allow branches and subsequent merges, with all the freedom that allows to development teams, it seems odd to see it suggested that it is best practice to avoid the very features that distinguish a source control system from a Wiki.

Ross Snyder, describing ‘Continuous Delivery’ (CD) at Etsy, boasts of a simple deployment process that is rapid to the point that they can do 25 deployments per day. They treat “failure to deploy” as a “P1” incident, the same level of severity as the whole site going down! In a conventional source control architecture, one can imagine that to support this would require a carefully constructed network of branches, one per feature or “user story”, merging the changes back to “master” when complete and tested, in order exert fine-grained over which features go live when, and so avoid delayed deployments. The downside, of course, is that without strict procedures and a very well-drilled team, it can becomes a recipe for “merge hell”.

So how do CD proponents handle all this branching, and more importantly the potentially-messy merging processes, that allow for such relentless deployment? The answer appears to be that they don’t. The CD orthodoxy is in favor of doing all of the development work ‘to the trunk’, thereby providing a ‘single, canonical version of code’. In this approach, as described in the Continuous Delivery book, a much better answer (than branching and merging) is to “develop new features incrementally and to commit them to the trunk in version control on a regular and frequent basis. This keeps the software working and integrated at all times.”

Ross’s presentation makes brief reference to “putting changes behind config flags”. The idea, then, seems to be that the application and database sit in the VCS, side by side in the trunk, where all developers work. No branching. They get around the delayed deployments issue by using ‘feature flags’ within the application code. They wrap every feature in code that prevents it from running until enabled via configuration. It means the team can deploy all features, even incomplete ones, only enabling each feature for the user as it becomes ready.

The idea of using toggles and flags to prevent the features from running is not new. Experience tells us that the problem with them is that they require a lot of discipline. The cause of the Knight Capital debacle, for example, is believed to misuse of what was essentially a ‘toggle’ system, where an archaic module for buying stocks was ‘toggled out’, forgotten about, and the toggle subsequently reused it for something else. It is a great temptation for developers to try to circumvent a system with bureaucratic overhead without being conscious of potential repercussions.

Of course, merges can be painful but if your source control system gives you “merge hell”, is it better to fix it, or move to another, rather than bend your development methodology to the inadequacies of your current system?