The 5 pillars of Continuous Integration (CI)

Sriram S
Technology, Engineering Productivity, Programming, Opinion
Page content

In a blog series about Continuous Integration and Continuous Delivery (CD), I described the process of CI and CD at a high level. In this blog, I will talk about the 5 pillars of Continuous Integration and the tools associated with these 5 pillars. I will explain how the interaction of these tools along with an Agile mindset will help you embark on a journey to continuously deliver value to your customers.

Definition and common concepts

Let’s start with the definition of CI. My favorite definition of CI comes from this Infoworld article.

Continuous integration is a coding philosophy and set of practices that drive development teams to implement small changes and check in code to version control repositories frequently. Because most modern applications require developing code in different platforms and tools, the team needs a mechanism to integrate and validate its changes.

There are many articles that cover the components or pillars of CI. Here is a quick summary of these articles.

  • In this article, Atlassian describes 4 pillars of CI. These are source control version management, automated testing, build automation, and automated deployment. For me deployments fall under the gamut of CD, so I will consider 3 out of these four pillars as valid for CI

  • This DZone article enlists the following three pillars of CI: a version control repository, a CI server and an automated build process. This is broadly similar to what Atlassian has also defined in their article. For me an automated build process leverages a CI server, so there is a strong inter-dependency.

  • Michael Bright details 3 pillars for CI. These are Automated testing, Architecture principles, and Culture. In practical terms, automated testing a good fit as a pillar for CI. Architecture principles and Culture are important concepts whether you follow CI or not. Nevertheless, I strongly urge you to read Michael’s blog for a holistic perspective.

What do I consider as pillars of CI?

The picture below shows my take on the pillars of a well-organised CI system. There are 5 pillars and these pillars are connected via a central entity which I will explain a little later. Let us understand these 5 pillars a bit better.

  • Work Management - this refers to tools like Jira, Slack etc. Modern Agile software development uses product backlogs in Jira. I have grouped Slack in this bucket because it is one of the commonly used systems for communicating status changes in the CI process and beyond.
  • Source Code - the source code versioning system is dominated by Git and its implementations like Github, Bitbucket etc. The branching methodology like Trunk-based development or Gitflow is implemented using the source code control system.
  • Build System - This is the system where source code is converted into artifacts for testing and deployment. Jenkins is the front runner in this pillar. But tools like CircleCI, Teamcity are also popular.
  • Test Infrastructure - This is usually a home grown system to run automated tests. Tools like Selenium and internal test automation libraries are used to create automated tests. While the tests may be triggered as a step inside Jenkins pipeline, the test infra is where the actual tests are executed and the results are recorded.
  • Artifact Repository - This is the repository where the packaged source code or binaries are stored. JFrog Artifactory and Sonatype Nexus are commonly used commercial repositories. A docker registry is usually part of this toolset.

What is the connection between these pillars?

Every software development team has its own methodology, style, and process which can collectively be called its engineering culture. This culture is dependent on the effective inter-operability between the 5 pillars I have mentioned earlier. This interaction between the 5 pillars is facilitated by the central pillar in the picture above.

This central, connecting pillar is often realized using automation and tools to power the entire CI experience. It is now common to have teams called “engineering productivity” or “DevOps” or “tools” team. Here are a few examples of how this automation and the associated toolchain enables strong connections between the 5 pillars and creates a smooth CI and developer experience.

  • Automatic branch creation - when a bug is found, the first step is to create a Jira ticket and the ticket is then assigned to a developer. Once the developer is ready to work on it, she will change the status to say “In Progress” in Jira. The “engineering productivity toolset” can use such fields in Jira to create an appropriate branch in Git. Effectively we have connected the “Work Management” and “Source Code” pillars and reduced the steps needed to be taken by a developer. Another example is when code is merged into Git, the comments can include special fields that can automatically update Jira status. This integration is built into Jira and Github.

  • Release notes generation - when a build system like Jenkins creates a software build, it can also use Jira and Git to create a list of changes that went into a particular build. It can further generate release notes that describe all the changes that went into a particular build.

  • Tagging builds - The engineering productivity toolchain also integrates the “Artifact Repo” and “Test Infra” pillars. Once a set of automated tests are executed, the status of the tests (passed or failed) can be used to tag the artifacts in the Artifact repository. These tags can be used to make decisions whether to run further tests or even deploy the artifacts to staging or production etc.

Conclusion

While I have only listed a few examples of how these pillars can be connected, the possibilities are immense. A lot depends on the processes you follow within your software development teams. The “engineering productivity toolchain” can integrate the pillars and help streamline the software development process. There are many open source and pre-packaged integrations available for this toolchain but investing resources in this area will truly boost the productivity of your developers.