23 Continuous Delivery Platform, Ten Essential Features for the Modern Pipeline ( Above)

23 Continuous Delivery Platform, Ten Essential Features for the Modern Pipeline (Above) #

Hello, I’m Shi Xuefeng.

As the culmination of DevOps engineering practice and the “ideal state” of software delivery, continuous delivery plays a crucial role in implementing DevOps within an organization. Every company I have encountered is constructing their own assembly line platform. Therefore, it is evident that the assembly line is the core practice of continuous delivery and the most direct manifestation of continuous delivery in practice.

So, how can we build a modern assembly line platform? What characteristics should this platform possess?

Based on my own experience in building and implementing assembly line platforms within companies, as well as the design principles of various companies in the industry, I have extracted and summarized the top ten features of modern assembly line design.

In the next two sections, I will break down these features and the underlying concepts behind them, as well as how to implement these concepts in platform design. I have compiled these ten features into the image below. Today, I will introduce the first five features.

Feature 1: Building a Platform instead of a Capability Center #

Compared to other DevOps platforms, a pipeline platform has a very typical feature, which is that it is the only platform that covers the complete end-to-end software delivery process. Because of this, the pipeline platform carries the capabilities for various aspects of the software delivery process, such as continuous integration, automated testing, deployment, and even manual approval.

So, would it be sufficient to directly integrate all the necessary capabilities into the pipeline platform?

This idea seems good, but it is not practical in an enterprise. Each step of software delivery is a highly specialized task. For example, establishing a good automated testing capability requires long-term investment from a team with professional skills.

Moreover, integrating all capabilities into the pipeline platform would make it bloated. Also, it would be difficult to form a team large enough to implement such an idea.

Furthermore, the construction of an enterprise’s DevOps platform is not something that can be accomplished in just a day or two. Each enterprise has a range of existing platforms that have become an integral part of the team’s daily software delivery operations. Starting from scratch would not only disrupt the team’s habits and affect short-term efficiency, but also result in significant additional costs for duplicated construction, which is not conducive to the rapid implementation of a pipeline platform.

So, since this approach is not feasible, how should we position the pipeline platform in a more reasonable way? In my opinion, the correct approach is to separate the continuous delivery pipeline platform from the vertical business platforms and define the boundaries between them.

The so-called vertical business platforms refer to single specialized domain platforms, such as automated testing platforms, code quality platforms, and operations and release platforms. These platforms are the ones that software delivery teams frequently interact with in their daily work.

The pipeline platform should focus only on process orchestration, process visualization, and providing underlying reusable basic capabilities. Examples of these basic capabilities include running resource pools, user permission controls, and task orchestration and scheduling processes.

The vertical business platforms, on the other hand, should focus on building specialized capabilities, handling logic for core business processes, and managing detailed data for specific steps. These platforms can serve external users independently or provide platform capabilities to the pipeline platform via plugins.

In this way, we can quickly reuse existing capabilities and achieve construction at the lowest cost. As the capabilities expand, the delivery processes supported by the pipeline platform will become highly flexible.

To borrow a quote from “Continuous Delivery 2.0,” the pipeline platform only serves as the scheduler, executor, and recorder of tasks, and does not need to invade the internals of the vertical business platforms.

The benefits of this design are apparent.

From the perspective of the pipeline platform, by integrating and reusing existing vertical business capabilities, it can quickly expand its range of capabilities and meet the needs of different users.

From the perspective of the vertical business platforms, they can continuously develop their technical depth, refine and master every aspect of their capabilities, which helps the enterprise accumulate core competitiveness. Additionally, the pipeline can direct more users to the platform and expose the vertical business platforms to a wider range of user scenarios.

Furthermore, during the execution process, the pipeline carries a large amount of information about the software development process, such as which requirements are included in a task and what changes are made. This information can be promptly communicated to the vertical business platforms. Armed with this process information, the vertical business platforms can greatly enhance operational efficiency through precise testing and other means. The core idea here is to build a good ecosystem for an internal enterprise DevOps platform.

Many well-known software designs in the industry embody this approach. For example, the Jenkins plugin center and GitHub Marketplace reflect the idea of building an ecosystem based on platforms.

I placed this feature as the first to introduce because it directly determines the positioning and subsequent design concepts of the pipeline platform. I will elaborate on how to design the platform for rapid integration of capabilities in the eighth feature.

Feature 2: Orchestratable and Visualizable #

In modern software development, it has become commonplace to have multiple technology stacks coexist.

For the simplest example, in a frontend-backend separated project, the frontend technology stack and the backend technology stack are obviously different. For a software architecture following the microservices style, each module should have the capability of continuous delivery.

Therefore, the traditional standardized software build and release path is no longer able to meet the needs of diverse development models. In this regard, the ability to orchestrate the pipeline process becomes essential as the process carrier for software delivery.

The so-called ability to orchestrate the process means that users can define each step of the software delivery process and the order in which each step is executed. In simple terms, it means “I am in charge of my module, I decide which delivery steps to add”.

However, many existing “pipeline” platforms still use a few “hard-coded” fixed stages, such as build, test, and release, to the extent that even if some technology stacks do not require compilation, this step cannot be skipped.

I have seen a company before that placed the action of generating a version tag in the stage of online inspection. After I asked, I found out that there was nowhere else to put this step, so it could only be temporarily placed here. You see, as a result, the appearance of the entire delivery process might not match the actual situation, which is obviously not the expected result of visualization.

To achieve pipeline orchestration, the front-end of the platform needs to provide a visual interface to facilitate user-defined pipeline processes. A typical way is to define the pipeline process as several stages, and each stage is executed in order. In each stage, steps can be added as needed, and these steps can be executed in parallel or in serial.

The front-end saves the orchestration result in a standardized format (generally in the form of JSON) and passes it to the back-end for processing. The back-end workflow engine needs to translate and process the content orchestrated by the user and pass it to the executor for interpretation and execution.

You can refer to the following diagram for an illustration of pipeline orchestration. In the actual running process, you can click on each step to view the corresponding execution result, logs, and status information.

At first glance, this mainly tests the front-end development capabilities of the platform. But in fact, the prerequisite for orchestration is that the system provides orchestratable objects, which are generally called atoms.

The so-called atom is a component that can complete a specific independent task. These components should have certain generality and should be as business-independent as possible.

For example, the action of downloading code is actually quite similar for both frontend and backend projects. The core is to complete the action of pulling code from the version control system using several parameters. Therefore, this is very suitable as an atom.

The design of atoms is the essence of the pipeline platform, because atoms embody the platform’s generality, reusability, and independence.

Taking Jenkins, which we are familiar with, as an example, an atom is a code snippet in the pipeline. By encapsulating features, the implementation is hidden inside the function implementation, and the calling method is exposed externally. Users only need to know how to use it and do not need to care about the internal implementation.

It is not difficult to implement an atom on your own. Just add a section of Groovy code in Jenkins. The sample code is as follows:

// sample_atom_entrance.groovy
def Sample_Atom(Map map) {
    new SampleAtom(this).callExecution(map)
}


// src/com/sample/atoms/SampleAtom.groovy
class SampleAtom extends AbstractAtom {


    SampleAtom(steps) {
        super(steps)
    }


    @Override
    def execute() {
        // Override execute function from AbstractAtom
        useAtom()
    }
  
    private def useAtom(){
        steps.echo "RUNNING SAMPLE ATOM FUNCTION..."
    }

Feature 3: Pipeline as Code #

In recent years, the concept of “something as code” has become deeply ingrained in people’s minds. In the field of application configuration, we have Configuration as Code, and in the realm of servers, we have Infrastructure as Code… when it comes to the design and implementation of pipelines, we also need to achieve Pipeline as Code, which means that the pipeline itself is represented in code.

For example, the introduction of Jenkinsfile in Jenkins 2.0 is a typical implementation of this. Similarly, GitlabCI provided by Gitlab uses a code-based approach and a descriptive language to showcase the business logic and execution of pipelines.

The benefits of code-based pipelines are self-evident: leveraging the powerful functionality of version control systems, pipeline code can be treated just like business code, enabling simple tracking of changes made to each pipeline.

During the execution of a pipeline, if there are changes to the pipeline configuration, these changes should also be reflected in the change log of the pipeline run. You can even include pipeline and environment change records in the release notes of a version. In the event of an exception, this information greatly enhances the speed of problem diagnosis.

Of course, if you only want to achieve change tracing for pipelines, there are other ways to do it. For instance, you can store pipeline configurations in a backend database and record the version information of the database each time a pipeline task is executed.

In fact, the benefits of pipeline as code go far beyond this. It greatly simplifies the configuration cost of pipelines and, like atomicity, is another pillar in modern pipeline construction.

Let me share an example of pipeline as code with you. In this example, you can see that the entire software delivery process is described in a very clear manner. Even if you are not an expert in pipelines, you can understand and use it.

image: maven:latest

stages:
  - build
  - test
  - run

variables:
  MAVEN_CLI_OPTS: "--batch-mode"
  GITLAB_BASE_URL: "https://gitlab.com"
  DEP_PROJECT_ID: 8873767

build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS compile

test:
  stage: test
  script:
    - mvn $MAVEN_CLI_OPTS test

run:
  stage: run
  script:
    - mvn $MAVEN_CLI_OPTS package
    - mvn $MAVEN_CLI_OPTS exec:java -Dexec.mainClass="com.example.app.A

Note: The example provided showcases how an entire software delivery process can be described using a code-based pipeline.

Feature 4: Pipeline Instantiation #

As a modeling of the software delivery process, pipelines are very similar to classes and instances in object-oriented languages. A class can initialize multiple objects, each with its own memory space and independent existence, and pipelines should also have this capability.

First, pipelines need to support parameterized execution.

By inputting different parameters, you can control the running result of the pipeline and even control the execution process of the pipeline.

For example, a pipeline should meet the needs of different branches, so it is necessary to extract the branches as parameters and manually or automatically obtain them during runtime.

Considering this scenario, in the platform design, you can define a format for parameters in advance. The standard format defined here is to start with “#” followed by the parameter name. By defining such parameters in the pipeline template, a business can reuse existing pipelines quickly without the need for rearrangement, only by modifying the runtime parameters.

Second, each execution of the pipeline can be understood as an instantiation process.

Each instance is based on the pipeline configuration at the execution time and generates a snapshot. This snapshot will not change with the change in the pipeline configuration. If you want to retrigger this task, you need to run it based on the snapshot at that time, thus achieving the requirement of historical backtracking.

Lastly, pipelines need to support concurrent execution capability.

This means that pipelines can be triggered multiple times, generating multiple running instances. This tests not only the scheduling and queuing capabilities of the pipeline, but also the management capabilities of persisting data.

Because each execution requires an independent workspace. To speed up the pipeline execution, static data needs to be mounted in the workspace, such as code cache and build cache. Some pipeline platforms do not support concurrency because they have not solved this problem well.

Feature 5: Principle of Limited Support #

The design goal of a pipeline should be to meet the needs of the majority of common scenarios and provide a certain level of customization and extensibility, rather than trying to satisfy all requirements.

When designing pipeline functionality, we often find ourselves in a dilemma: we want to abstract a general model that caters to all business scenarios, but we also encounter various special requirements in practice. It’s like trying to catch fish in a net, there will always be some that slip through, so we make the net bigger. For a platform, it eventually becomes very complex.

For example, let’s take the most common use case of building Android applications. Currently, the majority of enterprises use the Gradle tool, and there are only two common commands:

gradle clean
gradle assembleRelease / gradle assembleDebug

However, in actual business scenarios, Application A might require Node.js and need to install NPM, Application B might require Git-lfs for large files and need to execute installation commands, and Application C might have even more complex requirements, such as configuring patch mode and full build mode based on options.

If we try to satisfy everyone’s requirements within a framework, the configuration and logic will become very complex. It will be difficult for both development implementation and user usage.

Taking the native Xcode build step in Jenkins as an example, it provides 53 parameter options, meeting the needs of the majority of scenarios, but it also gets lost in a sea of parameters.

Therefore, pipeline design should offer limited possibilities, rather than exhaustively covering all variable factors.

When designing parameter interfaces, we need to follow the principle of “Occam’s Razor”, which means “entities should not be multiplied unnecessarily”. If a user wants to add a variable parameter to an atomic task, our first consideration should be whether this feature is something that 90% of users will use. If not, it should not be easily included in the atomic task design.

You may ask, how can we meet the diverse demands of users then? In fact, it’s quite simple. You can provide some common atomic capabilities within the platform, such as the ability to execute custom scripts, call HTTP interfaces, and define custom atomic tasks, etc. As long as these capabilities are provided, users’ diverse requirements can be satisfied.

Summary #

In this lecture, I introduced you to the first five characteristics of modern pipeline design, including building platforms instead of capability centers, being orchestratable and visualizable, pipeline-as-code, pipeline instantiation, and principle of limited support. In the next lecture, I will continue to introduce the remaining five characteristics. Stay tuned.

Reflection Question #

Does your company use assembly lines? What are some essential features that you believe assembly lines should have?

Feel free to share your thoughts and answers in the comment section below. Let’s discuss and learn together. If you find this article helpful, please feel free to share it with your friends.