24 Quick Feedback: Why is your company always doing continuous integration poorly? #

Hello, I’m Zheng Ye.

In the “Start with the End in Mind” module, we left a huge tail behind. In the article “Continuous Integration: Integration is a Part of Coding,” we explained the importance of integration, especially continuous integration, from the perspective of “start with the end in mind.”

But how to do continuous integration well is the content that many people are truly concerned about. Today, let’s talk about how to do continuous integration well.

Since we are planning to discuss continuous integration, let’s take a moment to think about your first impression of continuous integration.

Continuous integration? Jenkins? Yes, the first impression many people have of continuous integration is the continuous integration server, also known as the CI server. In the past, it was CruiseControl, and today it has been replaced by Jenkins.

Because of this, many people have come to understand the CI server as continuous integration. I have encountered teams like this before, where they would put everything on the CI server: compiling, code checking, running unit tests, and even gathering test coverage statistics.

Perhaps you may question, what’s wrong with that?

In the matter of software development, we don’t measure things in right or wrong. I can only say that this approach is feasible, but it is not the best practice. I hope you can think about whether there is a better approach than this.

To answer this question, we need to go back to the essence of continuous integration. The birth of continuous integration is the result of people trying to shorten the integration cycle. Why shorten the cycle? Because we want to get feedback as soon as possible to know if our work is effective.

Therefore, to do continuous integration well, we need to adhere to the essence of continuous integration: obtaining feedback quickly.

From this, we can deduce the key point of continuous integration. Just remember one sentence, quick feedback.

Quick feedback can be divided into two parts: quick and feedback. This also brings up two important goals of continuous integration: how to get feedback quickly and what kind of feedback is effective.

Getting Quick Feedback #

Let’s go back to the example we discussed earlier, where we put various checks on the CI server to see if there are any issues with our code. This is an effective form of feedback, but is it fast enough? While it is much better than not having continuous integration, we need to ask ourselves if we can get feedback even faster.

Obviously, there is room for improvement in terms of speed. Running these checks on our local development machines would be faster than running them on a CI server. In other words, performing the same operations in our local environment is faster than doing it on a CI server.

Why is this the case? Let’s take a look at what the actions are for each developer when the checks are done on the CI server.

We write our code, then we need to submit it and wait for the CI server to run the checks. Afterwards, we use the CI monitor to see the results. If everything is fine, we move on to the next task. If there are errors, we fix them and repeat the process.

Now let’s look at the actions involved when running the checks locally. We run the build script, and if everything is correct, we can choose to submit the code or continue with the next task. If there are failures, we can fix them immediately.

Comparing these two scenarios, when running the checks locally, we don’t need to submit anything, wait for the CI server to start running, or go somewhere else to check the results. So this operation is much faster compared to submitting the code to a server.

Furthermore, there is another key point here: our actions are continuous. Once the check results show an error, we can immediately enter the fix phase. As developers, we understand the importance of continuous actions. It’s like when we’re playing a game and we don’t feel the passage of time, this state is called “flow”.

On the other hand, submitting the code and waiting for the CI server to complete the checks interrupts your flow.

If you are interested in the concept of flow, you can read Mihaly Csikszentmihalyi’s book “Flow”, where he introduces the concept of flow.

So far, we have only been discussing the effect of continuous integration on individual developers for the sake of simplicity. Now, let’s get closer to the real world and introduce another important factor: team collaboration.

Imagine if your team were running the checks on the CI server. You are excited to finish writing a piece of code and are ready to submit it, but then your colleague next to you submits their code first, forcing you to wait. If you’re unlucky and your colleague’s checks fail, they will have to fix it, which means you’ll have to wait even longer.

This might not be a big deal for a small issue, but if it’s a major issue, it might take a long time to fix. At that point, you have no other choice but to wait. This is a significant waste of time.

At this point, we need to “interject” an important discipline in continuous integration: you can only submit code when the CI server is green. You cannot submit when checks are still running or when there are errors. The reason is simple: if multiple people submit code at this time and the checks fail, whose fault is it?

On the contrary, if only one person submits code at a time, the responsibility is clear. If the team is small, it’s relatively easy to follow this discipline, by simply checking before submitting or giving a heads up.

If the team is slightly larger, you can use something as a token, where only the person with the token can submit. If someone violates the discipline and submits code while the CI server is still running, it’s simple: they are responsible for the fix. They shouldn’t have violated the discipline.

Okay, now you understand the key point I’m trying to make: all checks should not only be performed on the CI server. So what should we do? The answer is imminent: perform the checks in your local development environment.

To effectively implement continuous integration, a critical step is to make use of a local build script, ensuring that various checks can be performed in the local environment.

Once you have a build script, your actions on the CI server become simple: just call that script. In other words, the actions of running checks locally and on the CI server are consistent.

As for what should be included in the build script, let’s set that topic aside for now and discuss it later in the “Automation” module.

In the “Task Decomposition” module, I discussed with you the importance of small actions in work. The faster we complete small actions, the quicker we receive feedback. Therefore, only by persistently performing small actions can we shorten the feedback cycle.

Now let’s combine this principle with continuous integration. Our workflow would look like this:

After completing a task, run the build script locally. If there are issues, fix them. If everything is fine, you can sync your code. If there are no ongoing services on the CI server, you can submit your code.

The most troublesome action during code submission is actually merging the code. However, because we are working on small tasks, the amount of code changes is not large, so even if there is a need for merging, it won’t be a large amount of code. The mental and work time required will be significantly reduced. In this way, our development efficiency can be truly improved.

When a team truly implements continuous integration, you will find that the duration of local checks will increase with time. There are many reasons for this, such as increasing amount of code or tests. In short, if the duration of checks becomes longer, it will affect the speed of integration.

At this point, following the principle of quick feedback, we must find a solution. For example, some teams perform distributed testing or categorize the tests, as we mentioned in the testing pyramid. They run unit tests and integration tests locally, and leave more complex system tests to be run on the CI server.

In simple terms, our goal is to get feedback quickly.

Obtaining Effective Feedback #

After discussing “Fast,” let’s now focus on the second key aspect of good continuous integration: feedback, specifically, obtaining effective feedback.

Why do we need feedback? The reason is simple; we need to know if what we’re doing is correct or not. You may ask, based on what was discussed earlier, if the scripts executed locally and on the CI server are the same, and I’ve passed locally, do I still need to care about the feedback from the CI server?

The answer is yes, because other issues can still arise. For example, the most simple scenario is that you forgot to commit a file.

Alright, now that we understand the importance of paying attention to the feedback from the CI server, the next question is how does it provide feedback to us?

Let’s start with a common mistake. Some teams use email as a means of feedback for continuous integration. What’s wrong with using email? It’s clear that email is not an instant feedback tool.

I don’t know how many people consider their email client as an everyday tool, but based on my personal habits, I only check my email a few times a day, and that’s considered good. If we rely on email as a feedback tool, it’s likely that errors will go unnoticed for a long time.

We have been emphasizing the importance of speed. What we need is instant feedback. Once email becomes part of the continuous integration chain, it cannot be fast, no matter what.

So, what can you do? In various discussions, I have actually already hinted at the answer: a continuous integration monitor, also known as a CI monitor.

(Image Source: CI monitor example - projectmonitor)

The principle of a CI monitor is simple. After the CI server completes the build, it exposes the results through an API. Early on, there were RSS and ATOM formats, and later JSON. Once the results are obtained, they can be presented in different ways. There are many CI monitor software options available, some visually presenting the results, while others provide desktop notifications.

Now, we can finally discuss the main point of this section: how to effectively present the feedback?

The answer is simple: make it attention-grabbing.

For example, many teams use a large screen to display the results of continuous integration. This way, everyone can see the results, and even if you happen to overlook something, someone else will remind you if an error occurs.

There are also some sensory-stimulating approaches. For instance, some people use red and green lights where a blinking red light indicates a test failure. Some even incorporate voice notifications, with loudspeakers shouting, “Test failed, please fix it quickly.” I’ve seen a more exaggerated approach in a video, where someone uses a toy gun and shoots at the person responsible for the error.

As a smart programmer, you should be able to think of even more interesting ways to present the feedback.

Why do this? The key here is that to do continuous integration well, the entire team needs to pay attention to it.

These attention-grabbing methods aim to increase the visibility of continuous integration. Without them, even if the technical aspects of continuous integration are executed skillfully, it won’t be effective if people are not paying attention to it.

So, as you can see, the feedback from continuous integration, especially after an error occurs, is one of the most noticeable aspects of the practice. Its purpose is to draw attention.

Here, let’s insert a discipline for continuous integration: Once the CI server detects an error, it must be fixed immediately. The reason is simple. If you don’t fix it, others cannot submit their work, and the progress of many people will halt, interrupting the workflow of the entire team. It affects the work of the whole team.

If you cannot fix it quickly, revert your commit. The most critical reason is that if the team does not prioritize fixing the issue for an extended period, continuous integration loses its significance. People will give up on it, and continuous integration will become ineffective for your project.

Summary #

Continuous integration is an important practice in software development, and the key to successful continuous integration lies in quick feedback. There are two goals here: how to obtain feedback quickly and what kind of feedback is effective.

To achieve quick feedback, we need to do things that can be done locally on our local machines. We should also speed up the pace of code development by making small, incremental commits. What constitutes effective feedback? It is feedback that is immediate and attention-grabbing. There are many continuous integration tools available to help us achieve effective feedback.

To excel in continuous integration, there are also some disciplines to follow:

• Code can only be committed when the CI server is in a green state.
• Once the CI server detects an error, it should be fixed immediately.

If you can only remember one thing from today’s content, please remember: the key to successful continuous integration lies in quick feedback.

Finally, I would like to ask you to share if your team practices continuous integration. What difficulties have you encountered? Feel free to leave a comment and share with us.

Thank you for reading. If you found this article helpful, please consider sharing it with your friends.