11 Branching Strategies, The Key Element for Efficient Development Cooperation #

Hello, I’m Shi Xuefeng. Today, let’s talk about branching strategy.

In the previous lecture, I repeatedly emphasized the idea of including everything in version control. In fact, modern version control systems not only record versions and change logs but also have a very important function, which is branch management.

Modern software development emphasizes efficiency and quality, which largely depends on collaboration between multiple teams. For some large-scale software, it is not uncommon to have collaboration among teams of hundreds of people. If the software architecture is not well divided, it is very likely that hundreds of people will be working in the same code repository. In this case, branch management becomes an indispensable feature.

On the one hand, branches can isolate the changes made by different developers, providing them with a relatively independent space to complete their own development tasks. On the other hand, the entire team also needs to complete tasks such as code submission, review, integration, and testing according to the software release schedule.

Therefore, if there is a soul in a multi-person software collaboration project, I believe that soul is the branching strategy. It can be said that the branching strategy is the compass for software collaboration and release patterns. Choosing a branching strategy that aligns with the DevOps development mode will also be helpful for the implementation of DevOps.

Today, I will break down some common branching strategies for you, helping you understand the core processes, advantages, disadvantages, as well as applicable scenarios and examples of these strategies.

Mainline Development, Branch-Based Release #

Image source: What is your branching model?

In this branching strategy, the development team shares a mainline branch where all the code is directly committed. The mainline branch serves as a comprehensive collection of the code. Before software version release, a short branch based on the mainline branch is created specifically for the release purpose.

You need to pay attention to two key phrases in this sentence:

1. For the purpose of release: the purpose of this branch is not to develop new features, but to review and accept existing features and release them externally after meeting certain quality standards. Generally, new features are not submitted based on this branch, only bug fixes are integrated. Therefore, there are strict permission controls for this release branch. After all, no one wants unverified and unreliable features to be introduced into the release branch.
2. Short branch: this release branch does not typically exist for a long time. Once it has passed regression testing and met the release criteria, it can be directly released externally. At this point, the historical mission of this branch is fulfilled. Unless urgent issues are discovered after the release and need to be fixed, modifications will continue to be made and synchronized back to the mainline branch. Therefore, during the time when the mainline and release branches coexist, all changes made in the release branch need to be synchronized back to the main branch. This is also why we do not want this branch to exist for a long time, as it would result in linear cumulative repetitive work.

This branching strategy is very common for software projects driven by version release rhythms, such as client products or smart hardware products that require upgrades on customer terminals, like smartphones or smart TVs.

Many years ago, when LeTV (now known as LeEco) first launched its Super TV, they had a slogan called “weekly updates”. It’s worth mentioning that smart TV products were generally updated only once every few months back then.

Actually, if you understand branching strategies, you will find that there was nothing particularly special behind the “weekly updates” slogan. At that time, the team I belonged to was responsible for the branching strategy of the smart TV product line, which used the mainline development and branch-based release model. A release branch based on the mainline branch was created two weeks in advance, and then regression testing was conducted on the release branch. In the first week, a preview version was released to adventurous users for trial. Based on user feedback and issues collected from the backend, further adjustments were made and eventually, a stable version was released. I will share with you the branching strategy diagram from back then for your reference.

This model has three advantages:

1. For the development team, there is only one mainline branch and no need to switch between multiple branches.
2. After the release branch is created, the mainline branch remains in an integratable state, allowing the development pace to remain relatively stable.
3. Release branches are generally named based on version numbers, which are clear and easy to understand. When a problem occurs in a specific version online, the fix can be made in the corresponding branch.

However, this model also has disadvantages and challenges:

1. It demands high quality for the mainline branch: if there is a problem with the mainline branch, it will block the work of the entire development team. For a team of hundreds of people with thousands of submissions per day, if there are no constraints on submissions, the frequency of such incidents will be very high.
2. It requires high collaboration rhythm: if the features on the mainline branch are not merged in a timely manner, but the business side insists on releasing these features in a specific version, it will lead to difficulties with the release branch. Sometimes, it is even necessary to allow unfinished features to continue development on the release branch, which poses significant challenges to the quality and stability of the release branch.
3. During the coexistence of the mainline and release branches, there may be unsynchronized submissions: for example, if a fix is made in the release branch for an issue but is not synchronized back to the mainline branch, the same issue may reoccur in the next version. The more issues found during testing, the higher the probability of this situation occurring, not to mention the situation with multiple versions coexisting.

The solutions to these problems include the following:

1. Establish requirements for code submissions and do not allow code that does not meet quality standards to be merged into the mainline branch.
2. Adopt a version train approach to accelerate the iteration speed of versions. Features that cannot keep up with a specific version are carried over to the next version. Additionally, feature switches and hotfixes can be used to break the fixed rhythm of version releases and allow more flexible external releases.
3. Use automation tools to scan and compare the differences between the mainline and release branches and establish rules. For example, hotfixes must be submitted to both the mainline and release branches simultaneously, or after the release branch goes live, someone is responsible for synchronizing the changes back to the mainline branch in reverse.

Branch Development, Trunk Deployment #

Image source: https://paulhammant.com/2013/12/04/what_is_your_branching_model/

When a developer receives a task, they will create a feature branch based on the trunk. After completing the feature development on the feature branch and verifying its functionality, they will initiate a merge request or pull request to merge the changes into the trunk. Once the review is approved, the changes will be merged into the trunk and undergo regression testing. The popular GitHub model in the open-source community follows this approach.

Based on the characteristics and the team’s situation, it can be further divided into two scenarios:

• Each feature branch is named after a feature or requirement number, and only one feature development is completed on this branch.
• A long-lived feature branch is created based on a development module, and collaborative development is conducted on this branch.

The difference between the two lies in the lifespan of the feature branch. The longer a branch exists, the greater the difference from the trunk, and the more conflicts there will be during merging. Therefore, for the long-lived branch strategy, either the module is well-defined and no one else is working on it, or frequent synchronization with the trunk is maintained. The short-lived branch strategy is more suitable as the granularity of the requirements decreases.

There are two advantages to this approach:

1. Branch development is relatively independent, without interference from parallel development. At the same time, only after a feature is developed and accepted, it will be merged into the trunk, which protects the quality of the trunk branch.
2. With the popularity of feature branches in this model, branches naturally become the carriers of features. All the code related to a specific feature can be stored on a feature branch, which provides a new possibility for a release based on feature granularity. In other words, if you want to release a specific feature, you can directly merge the feature branch into the release branch. This allows a feature to be easily added or removed, rather than being mixed with a large amount of code.

Regarding the method of releasing feature branches, I can provide you with a reference for you to understand. However, I would like to remind you that although feature-based releases may seem good, they have three prerequisites: first, the features need to be small enough; second, there should be a robust testing environment to support flexible feature combination validation requirements; third, there should be an automated feature management tool.

Of course, the branch development and trunk deployment approach also has its disadvantages and challenges:

1. It requires the team to have good capability of splitting features. If a feature is too large, it will result in a large number of parallel development branches, lengthening the integration cycle and increasing potential conflicts. Also, long-lived branches will cause significant differences from the mainline. Therefore, the granularity of the features and the lifespan of the branches are critical factors. As a general rule, the lifespan of a branch should not exceed one week.
2. There is a high requirement for naming conventions of feature branches. With a large number of feature branches created, the entire code repository can become messy. Faced with a large number of branches, it is hard to determine which ones are still active and which ones are no longer used. Therefore, it is best if the branch creation process can be automated and integrated with a change management system.
3. The atomicity and integrity of feature branches require that all changes related to a feature should be committed to one branch instead of being scattered everywhere. Also, the commits on feature branches should be as clear as possible, with atomic commits being a typical example.

The team I previously worked with used this branch strategy. Once, I had a heated argument with the development lead about the details of branch strategy execution. The core issue was whether the code on feature branches should be refactored when merging them back into the trunk.

Anyone with development experience knows that it is rare for someone to get the code right with just one commit, as there are always various issues leading to messy commits on feature branches.

When merging into the trunk, there is an opportunity to reorganize the commit history to ensure code atomicity, which Git is very powerful at. If you are proficient in using the git rebase command, you can quickly merge and split commits, organizing each commit into meaningful atomic commits before merging them into the trunk. Alternatively, you can squash the changes on the feature branch into one commit. However, the cost of doing this is constantly rewriting the history of the feature branch, which brings additional workload to the development team. I will share with you some common commands.

For example, if the current feature branch is feature1 and the main branch is master, you can execute the following command to reorganize the commit history:

git checkout feature1 && git fetch origin && git rebase -i origin/master

The most common operations include: - p: pick a commit; - r: update the commit message; - e: edit the commit, allowing you to split a commit into multiple ones; - s: squash commits, merging multiple commits into one; - f: similar to squash, but discards the original commit message and uses the merge commit message; - there are other options available during the interactive interface of git rebase, such as adjusting the order of commits, integrating feature functionality with related bug fixes, etc.

I need to remind you that the branch strategy represents the behavioral guidelines of the development team, and each team needs to develop a suitable model through practice.

Trunk-based Development and Release #

Image source: https://paulhammant.com/2013/12/04/what_is_your_branching_model/

Today I will introduce to you the third type of branching strategy, which is trunk-based development and release. Martial arts masters often discover that the path to greatness is simple, and the same goes for branching strategies. So, the third branching strategy can be understood simply as having no strategy. The team only has one branch, and developers’ code changes are directly integrated into this main branch. At the same time, software releases are also based on this main branch.

For continuous delivery, the ideal situation is that each commit goes through a series of automated environments and is deployed to the production environment. This strategy brings us closer to that goal.

As you can imagine, if we want the main branch to be in a releasable state at any time, the quality of the code in each commit must be very high.

In some companies that pursue engineering excellence, submitting a line of code is a Herculean task. This is because there is a series of automated acceptance measures and a strict code review mechanism in place to ensure that your commit does not break the main branch. Of course, even with these measures, problems are still inevitable. So what should we do then? Here, I will introduce Facebook’s branching strategy evolution case to you.

Facebook initially adopted the trunk-based development and branch-based release strategy, releasing twice a day. However, as the pressure of business development increased, the team had higher requirements for release frequency, and this branching strategy could no longer meet the needs of multiple releases per day. Therefore, they began to change the branching strategy from trunk-based development and branch-based release to trunk-based development and release.

To ensure the quality of the main branch, automated acceptance measures are essential. Therefore, each code commit triggers a complete process including compilation and building, unit testing, code scanning, and automated testing. After the code is integrated into the main branch, it will be released on demand. First, it will be released to the internal environment, where only Facebook employees can see this version. If any issues are found, they will be fixed immediately. If there are no issues, the release will be further opened to 2% of online production users, while automatically detecting feedback data from the online environment. Only when everything is confirmed to be normal will it be released to all users.

Finally, through the integration of branching and release strategies, the injection of automated quality acceptance and online data feedback capabilities, the release frequency has been increased from fixed twice a day to multiple times a day, and even achieved on-demand release. The latest branching strategy at Facebook is shown in the following illustration:

Image source: https://engineering.fb.com/web/rapid-release-at-massive-scale/

Seeing this, you may ask, “Among these three typical strategies, which one is the best? How should I choose?” Indeed, this question also troubles many companies.

Indeed, software projects of different types, scales, and industries may adopt different branching strategies. At the same time, factors such as release frequency, software architecture, infrastructure capabilities, and personnel skill levels also restrict the effectiveness of branching strategies.

Therefore, it is difficult to say that there is a universal branching strategy that can meet the needs of all scenarios. However, some branching strategy principles are more suitable for scenarios that require rapid iterative releases and are more in line with the development trends of DevOps. So, personally, I would recommend the trunk-based development combined with feature branches pattern, where the team shares a development trunk, feature development is based on the trunk but on separate feature branches, and rapid development, acceptance, and merging back to the trunk are done. At the same time, different quality gates and automated acceptance capabilities are established for feature branches and the main trunk.

The benefits of this approach are that it accelerates code integration frequency and keeps features relatively independent and clear, while maintaining a certain level of code quality in the main branch. However, during execution, you need to adhere to the following principles:

1. The team shares one main branch.
2. The lifespan of feature branches should be as short as possible, preferably not exceeding 3 days.
3. Merge code into the main branch once a day. If a feature branch exists for more than 1 day, synchronize it with the main branch code daily.
4. Use feature toggles cautiously to maintain clean code and a clear history.
5. The fewer parallel branches, the better. If possible, adopt the trunk-based release.

Regarding the last principle, what you need to consider is whether a release branch is needed, which mainly depends on the project’s release model. For projects that release based on versions, such as apps, smart hardware systems, and core systems that require extensive external system integration, you can create release branches according to a fixed release rhythm. For applications with a faster release rhythm and relatively independent architecture after system architecture decomposition, you can directly adopt the trunk-based release pattern and combine it with secure release strategies to control the overall release quality.

The illustration for this branch release strategy is shown below:

Summary #

Today, I introduced three branch strategies to you. I suggest that you refer to the branch strategy diagram I shared with you and try to understand it. In addition, I also introduced branch strategies suitable for the DevOps mode and some usage principles. Do you remember what I said at the beginning? Branch strategy is the compass for development collaboration and release patterns. The changes in branch strategy require a significant adjustment in the habits and rhythm of the entire development team. Finding a branch strategy that suits the current team is the most important.

Thought Question #

What branch strategy does your current team use? Do you think there are any problems or room for improvement with the current branch strategy? Have you ever experienced any adjustments to the branch strategy? If so, did you encounter any challenges during this process? Do you have any insights to share?

Feel free to write down your thoughts and answers in the comment section. Let’s discuss together and learn from each other. If you find this article helpful, please feel free to share it with your friends.