Is Continuous Release for you?

Continuous Release is the practice of releasing code to end-users in a fully automated fashion, releasing every commit that passes checks with no batching. It is the logical extension of Continuous Integration (CI) – where all commits by a team are to a central branch, with no batching – and Continuous Deployment (CD) – where all commits are deployed to one or more environments in a fully automated fashion with no batching. Continuous Release (CR) is an interesting one, because while CI and CD are purely technical concerns, Continuous Release is primarily a business concern.

The main value proposition of CR is that new features are delivered to end-users as soon as it is possible to do so. One of the benefits of Agile methodologies is the focus on regularly releasing, whether this is after every sprint or on some other kind of schedule, and CR takes this one step further. There are many successful businesses that use CR to release new features many times a day, and this can be hundreds or even thousands of tiny releases. Releasing so frequently means that there is less risk in each deployment – but not necessarily that there is no risk, as the quality of the software is not automatically improved by this approach. The emphasis in CR is in having automated tests that are trusted to the extent that whenever they pass, software is ready to go (with no manual intervention), and on having monitoring and alerts set up such that if a bad release goes out to end users it can be swiftly identified and resolved. The good news is that a fix to an issue can also be released very quickly, or the deployment could be rolled back – most prefer to fix forwards.

Successful use of CI, CD and especially CR is dependent on meeting pre-requisites – without these in place trying to use these practices will usually do more harm than good. The main pre-requisite is having good enough quality control in place that detects and prevents the release of faults, and having an effective way of rectifying faults when they do arise. This is largely psychological as well – a team who are going to use CR need to be pragmatic and mature enough to understand that a 100% perfect quality control system is unlikely to exist, and certainly not for a price that the business can justify paying. There is a balance to be struck in having the most effective guards in place without imposing prohibitive cost. In a business with a blame problem, or a culture that cannot accept, recover from, or be honest about mistakes, trying to implement CR is politically risky – the odds are that sooner or later defects will leak to end-users, and the business and engineering teams need to react in the right way when this happens.

For all three practices to work well, perhaps one of the most important pre-requisites is the speed of the feedback loop. I firmly believe that all teams should be aware of and trying to feedback loop anyway for many reasons (the main one being simply to deliver value more effectively), but in CR in particular, the time taken for a single line of code to go from one end of the pipeline (a developers’ machine) to the other (deployed to production) is critical. Measuring this time doesn’t have to be precise; a rough estimate will do. For example, you might have a build that takes 2 minutes, unit tests that take 1 minute, integration tests that take 10 minutes, a deployment that takes 5 minutes and an automation suite that takes 30 minutes – add these times together, along with any other delays, and you’ll get a rough number. Assuming you find a bug in production that has to be fixed immediately, and the fix is already known and takes no time to implement, this number tells you the quickest possible time it will take for the fix to take effect. If this time is unacceptably long (it is a business decision, probably with assistance from engineers to determine this), then effort should be invested as a priority to reduce this number. Having a fast feedback cycle will pay dividends very quickly though, as every single future feature will be easier to deliver as a result. Slow feedback cycles are one of the most clinical killers of projects I know of!

Once you have fully understood that it is not viable to both eliminate 100% of errors and have a fast feedback loop, this is the point to start making decisions. The decision though is almost entirely a business one, rather than technical. In a business that has a high volume of low value interactions, being able to release very often and fix forwards when there is a problem is normally quite palatable: some interactions will be bad, but if they are lower value, then customers’ tolerance will be higher, and the cost of maintaining reputation when these problems arise is lower. However, on the opposite end of the scale if your business has a very low volume of very high value interactions, then the risk of reputation from any bad interaction is a major factor in deciding whether Continuous Release is appropriate for the business. In such environments, there may be more value from a more rigorous quality control and release process, to minimize the risk of negative customer experiences, than in fast release of features. This is especially true in businesses that are heavily regulated, where the cost of non-compliance may be devastating.

In businesses where the balance is towards confidence in every release, a good solution can be to practice continuous “release” to a staging environment – perhaps a beta version that some customers use on the understanding that it may be less stable, but more likely an environment that is used internally only. This allows the engineering team the benefits of a well-exercised build and deployment pipeline without introducing business risk. Release of software is then a manually gated operation. Another time where this approach can work well is if the business’s marketing strategy favors highly publicized releases. For example, there may be a major event or conference where a new version will be publicly launched, and the value in building that launch could be destroyed by continuously releasing features ahead of time. Equally, for some businesses versions form part of the pricing model, where companies might pay for upgrades – and again, in this instance CR goes against the business strategy.

It is important for us as engineers to always remember that our role exists to serve a business strategy, and technical concerns must never be allowed to override these business values. There will be times when we can add value by influencing these strategy – such as when technological advances may change the terrain – but as engineers, we must always try to understand the business values, and show these the respect they deserve.