32 Continuous Delivery Is Enough With Continuous Integration

32 Continuous Delivery: Is Continuous Integration Enough? #

Hello, I’m Zheng Ye.

In the previous two lessons, I talked to you about automating the development process, packaging our programs into release packages, and automating the deployment process, deploying the release packages using various tools.

With these foundations in place, we can consider packaging and deploying the program to the environment after each development iteration. Automatically packaging and deploying after development sounds a lot like continuous integration, right?

I have already discussed continuous integration twice in this column, discussing “Why we should do continuous integration” and “How to do continuous integration” respectively. However, the discussion about continuous integration only focused on the development stage.

With the preparation from the previous two lessons, we can further extend this process. Keen-minded you may have already noticed that the theme I am going to talk about this time is continuous delivery.

Continuous Delivery #

A book that has made the concept of continuous delivery widely known is “Continuous Delivery” by Jez Humble and Dave Farley.

I mentioned CruiseControl earlier when talking about the history of continuous integration. It is the ancestor of continuous integration servers. Due to the continuous development of continuous integration, in 2007, my former company ThoughtWorks intended to provide enterprise-level services based on CruiseControl and formed a team to build a better continuous integration server. Jez Humble was a member of this team.

Also working in this team was Qiao Liang, the translator of the Chinese version of “Continuous Delivery.” After nearly ten years since the publication of the book, he wrote “Continuous Delivery 2.0,” which included his new insights on continuous delivery.

So, what makes a better continuous integration server? At that time, my understanding was limited. I simply hoped for a better user interface and faster build speed. However, Jez Humble and his team had a vision for the product that far exceeded my imagination. They included the production environment in their considerations.

What is continuous delivery? In short, it is the ability to deploy software to the production environment at any time. What is missing between a well-packaged release and deployment to the production environment? That is the verification of the release package and its deployment to the environment.

You might think that verifying the release package is part of testing, and isn’t that already done in the continuous integration phase? Not exactly. The packages verified in the continuous integration phase often lack support from the environment.

Because the environment for continuous integration is often limited to a single machine and mainly focuses on functional verification, some tests related to the production environment are often lacking. This brings us to a point that needs attention in continuous delivery: the environment.

Generally, when building the infrastructure for continuous delivery, there are several different environments:

  • Continuous Integration (CI) Environment: Continuous integration is the prerequisite for continuous delivery. This process mainly performs basic checks to produce a deployable package.

  • Test Environment (Test): This environment is often single-machine-based and mainly responsible for functional verification. The tests run here are usually at the acceptance testing level, while faster tests such as unit tests and integration tests are typically executed in the continuous integration environment.

  • Staging Environment: This environment is usually configured the same as the production environment, including features like load balancing and clustering, but with fewer machines. It is mainly responsible for validating the deployment environment, such as identifying issues caused by concurrent operations on multiple machines.

  • Production Environment: This is the real online environment.

You can see that each environment has a different purpose. Therefore, usually all verifications are not performed together, but rather in different stages. Each stage needs to be completed successfully before moving on to the next stage. This way of organizing builds into different stages is called a build pipeline.

Once all the verifications are passed, the final stage of the build pipeline is the production release. Generally, this process is not automated. When we talk about the emphasis of continuous delivery on the ability to release software at any time, it does not mean that it should be immediately deployed. So, the final decision of whether to go live or not still lies with a person.

If the decision of whether to go live is automated, it would become another practice called continuous deployment. However, people are usually more cautious, so having a person make the final decision is considered more reliable. Therefore, continuous delivery is the current mainstream approach.

So far, we have discussed the first aspect of continuous delivery, which is verifying the release package. Next, let’s move on to another important part: deployment.

DevOps #

In the early days, people used to write their own shell scripts for deployments. But as mentioned in the previous lecture, some tools like Chef, Puppet, Ansible, etc., greatly simplified the process of script writing for deployments. The rise of these tools is closely associated with a concept called DevOps.

DevOps is a philosophy and methodology for software delivery, with the aim of enhancing the reliability of software. As the name suggests, DevOps combines the words “Development” and “Operations”.

In traditional IT companies, development and operations are often two separate positions, or even different departments. This has led to many problems, such as developers making configuration changes without notifying operations, resulting in the new code not being able to run.

DevOps advocates for the integration of development and operations, breaking down barriers. In 2009, Flickr presented a talk titled “10 Deploys Per Day”, which had a significant impact on the industry and sparked the DevOps movement. DevOps has brought about significant changes in the industry and requires continuous improvement in areas such as culture, processes, and tools.

But in our daily work as programmers, the most direct impact is reflected in various tools. Chef, Puppet, Ansible, and other tools have all gained popularity after that.

In the previous lecture, I explained the roles of these configuration management tools in the operations system. They provide a framework. But for the industry as a whole, these tools have brought about deployment standards.

In the past, writing shell scripts for deployments was a diverse skillset. What you learned at Company A couldn’t be used at Company B. In fact, many people thought that deployment should be specific to a particular scenario, and scripts had to be rewritten for each new machine. This situation was similar to the time before Spring appeared when almost every company was writing their own frameworks.

The emergence of Spring changed everything, making your Java skills applicable across the industry. Similarly, these configuration tools in the operations system play the same role. They have even brought about a new concept: Infrastructure as code. This turns managing and configuring computers into writing code.

Once it becomes code, it can run anywhere, be version controlled, and the highly hero-dependent machine configuration work finally becomes accessible to everyone. This was unthinkable in the past.

These tools use declarative interfaces, moving from describing how things are done in Shell to describing what needs to be done. This level of abstraction allows developers or system administrators to focus more on thinking about how the machine should be configured.

If these configuration management tools still require a specific machine for deployment, in the context of continuous delivery, they can only play a minor role in the deployment environment. This is where Docker completely changes the game.

As mentioned in the previous lecture, Docker is like a machine. One great thing about Docker is that it is a machine that can be described in code. The Docker configuration file describes the desired state of the machine, and then an image is generated and deployed on a specific machine.

Since we are describing the state of the machine, we can use the configuration tools mentioned earlier in the Docker configuration file. This simplifies our configuration work. Furthermore, since we are discussing continuous delivery, we can also deploy our release package to the final image using configuration tools. As a result, the final generated image includes our own application image.

You may already know what I’m going to say. Combined with these tools, our final deliverable goes from being a release package to a Docker image.

In development, Docker plays the role of an intermediary layer between our application and the actual machine. For the application, Docker is the machine, but for the machine, it is just an image that can be deployed. It unifies the various deployment differences of diverse applications, making the deployment itself simpler.

So far, I have introduced the most fundamental aspects of continuous delivery, giving you a basic understanding of the framework of continuous delivery. Of course, if you pay attention to this field, you will find that it has already surpassed the level of being just a practice, and there are more organizational and cultural aspects involved.

When Jez Humble wrote “Continuous Delivery,” he already envisioned such a complete system. However, due to the limitations of the environment at that time, the automation described in the book was more general, unlike today where there is more comprehensive tool support.

It is a pity that although he had such a high understanding of continuous delivery at that time, and his team created a pioneering continuous delivery tool, due to product promotion strategies, this tool did not become mainstream, even though it was later open-sourced. (If you want to know more about this tool, you can click on the link to GoCD).

Summary #

Let’s summarize today’s content. Building upon the previous two lectures, after preparing the infrastructure for packaging and deployment, we have incorporated the deployment process into the continuous integration mindset, which is called continuous delivery.

Continuous delivery is the ability to deploy software to a production environment at any time. There are two key points to enable software deployment to a production environment: validating the release package and deployment.

Validating the release package involves not only functional validation but also integration with the environment. Therefore, multiple environments are usually used for validation, with each environment being a separate stage. If a stage fails, it cannot proceed to the next stage. This method of organizing builds into different stages is called a build pipeline.

An important concept related to deployment is DevOps, which combines development and operations. DevOps encompasses many aspects, but its most direct impact on programmers is the development of various tools, which in turn drives the development of another concept: Infrastructure as Code. Thanks to the development of these tools, today’s definition of delivery is no longer just a release package, but a deployable image.

If there is one thing you should remember from today’s content, please remember: Consider deployment as part of development.

Finally, I would like to invite you to share your understanding of continuous delivery. Feel free to write down your thoughts in the comments.

Thank you for reading. If you found this article helpful, please feel free to share it with your friends.