Using marker branches to aid continuous integration

For a team whose workflow is based on continuous integration – that is, frequent pushes to and pulls from a shared integration branch – it is really important to have a healthy integration branch. A team who want to use this workflow should all buy into exercising a degree of discipline in preventing the integration branch from becoming broken, which usually means respecting a definition of “done”, or at least of “good enough to share”. If for whatever reason this doesn’t work, the consequence is that the team can become blocked by a bad commit, as one by one they pull the affected commit into their working branch, which causes disruption and reduces the team’s effectiveness considerably. Normally, every developer’s core work flow should at a minimum be:

  • Get latest from the integration branch
  • Run all the tests to make sure work starts from a known-good point
  • Make changes as needed, adding and amending tests as appropriate around the changes
  • Run all the tests again to make sure there are no issues being introduced
  • Push changes back to the integration branch

In a team I was helping recently, one of their major issues was a test suite that took far too long to run locally, making this workflow practically impossible. Some of the team believed they were able to identify which subset of tests they needed to work – in practice this simply wasn’t true, as evidenced by a pass rate of as low as 30% green builds. This was causing a lot of disruption, as no-one in the team was able to know with any confidence whether they were basing their changes off an all-green starting point. At the root of these problems were deeper issues around working practices that would need addressing, as well as the technical and prioritization problem of building a test suite that was fit for purpose – neither of which would be quick or easy to implement.

As a quick way to improve the situation, I introduced “marker branches”, to show the last commits that had passed various stages in the build pipeline. We were using TeamCity as a build server, which now offers this functionality out of the box with a little configuration on git repositories. The idea is very simple – the most recent commit on the integration branch was available to people who wanted it, but it would take a while for the tests to run, and so there is a degree of risk associated with merging this into a local working copy. However, the last commit that the tests had successfully completed on would be shown by a system-managed branch. We used 2 – a “BuildVerified” and “TestsVerified” branch to show commits that had passed the initial build and full test runs respectively. A developer could then choose to merge these known-good commits into their working branch with confidence.