08 Research and Development Process Design How to Design a Go Project Development Process

08 Research and Development Process Design How to Design a Go Project Development Process #

Hello, I’m Kong Lingfei. Today we’re going to talk about how to design a development process.

In Go project development, we not only need to complete the development of product features, but also ensure that the entire process is efficient and the code is of high quality. This requires a well-designed development process.

An unreasonable development process can bring about many problems, such as:

Chaotic code management. Issues arise when merging code, such as merging incorrectly, losing code, or encountering code conflicts.
Low development efficiency. Compiling, testing, and static code checking all rely on manual operations, resulting in low efficiency. Moreover, developers may even skip testing, static code checking, and other steps due to the lack of a standardized process.
Low release efficiency. Lengthy release cycles and frequent production issues caused by irregular releases.

Therefore, it is essential to design a reasonable development process for Go projects in order to improve development efficiency and reduce software maintenance costs. Different development processes may vary depending on the project, team, and development mode, but they still share some similarities.

So how do we go about designing a development process? This is likely the first question that comes to mind when you see the title “design.” When you see these two words, you might immediately think that I will guide you through a series of methodologies to help you design a process. However, the reality is that project development processes can differ based on the team, project, and requirements, making it difficult to generalize a methodology for designing a development process.

Therefore, in this lecture, I will introduce a well-designed and relatively standardized development process that is already established in the industry to guide you on how to design a development process. By studying this process, you will not only gain an understanding of the general flow of project development, but also be able to optimize and customize it to meet your own process requirements.

What points should be paid attention to when designing the R&D process? #

Before looking at specific development processes, let’s think about what a good process should look like.

Although we just mentioned that the development processes for different teams, projects, and requirements will not remain unchanged, in order to maximize development efficiency, these different processes will follow the following principles.

High release efficiency: The development process should improve release efficiency by reducing release time and the amount of manual work required.
High release quality: The development process should improve release quality by ensuring that the released code is thoroughly tested and completely free of faults caused by human factors.
Fast iteration speed: The entire development process should support rapid iteration. The faster the product iteration speed, the stronger the product’s competitiveness, and the better able it is to seize opportunities in the Internet age.
Clarity: The roles, tools, methods, and processes in the entire development process should be clear, which can enhance the executability of the process.
Reasonable process: The development process is ultimately used by personnel involved in product, development, testing, and operations. Therefore, the entire process design should not be counterintuitive and should be able to be accepted and executed by various participants.
Flexible scalability: The development process should be flexible and scalable, able to be flexible and adaptable to various scenarios.
Input and output: Each stage of the development process should have clear inputs and outputs, which indicate the completion of the previous stage and the start of the next stage.

With these points in mind, we have grasped the key factors to design and optimize the development process. Next, we can learn a relatively standard industry development process together. During this learning process, you will also better understand my experience and suggestions for various processes.

What does the relatively standard R&D process in the industry look like? #

A project goes through many stages from project initiation to completion. The industry standard divides the R&D process into six stages, namely the requirements stage, design stage, development stage, testing stage, release stage, and operation stage. Among them, developers need to be involved in four stages: design stage, development stage, testing stage, and release stage. The following diagram shows the relatively standard process in the industry:

At the end of each stage, there needs to be a final deliverable, which can be a document, code, or deployment component, etc. This deliverable serves as both the milestone for the completion of the current stage and the input for the next stage. Therefore, the stages are not isolated but closely connected as a whole. Each stage is further divided into many steps, and different participants are responsible for completing these tasks. During the process, multiple rounds of discussions and revisions may be required, leading to the final draft.

There is an important point we need to pay attention to: the R&D process is also a kind of standard that cannot rely solely on developers’ self-consciousness to comply with. In order to ensure that project participants comply with the standards as much as possible, it is necessary to use tools and systems to enforce strict constraints on them. Therefore, after designing the complete R&D process, it is important to carefully consider where automation can be implemented and where tools and systems can ensure the enforcement of standards. These automation tools will be discussed in detail in Lecture 16.

Next, let’s take a closer look at each stage of the R&D process and the specific contents of each stage.

Requirements Stage #

The requirements stage is the stage where an abstract product idea is concretized into an implementable product. In this stage, product personnel discuss product ideas, conduct market research, and analyze requirements to produce a comprehensive requirements document. Finally, the product personnel organize a review of the requirements, and if it passes the review, it will move on to the design stage.

In general, developers do not need to participate in the requirements stage. However, I still recommend that you actively participate in the discussion of product requirements. Although we are developers, our perspective and contribution to the team can extend beyond the development field.

One point to note here is that if your team has testers, they should also be involved in this stage to gain an understanding of the product design, which will be helpful for writing test cases and conducting functional testing in the testing stage.

The deliverable of the requirements stage is a detailed requirements document that has passed the review.

Design Stage #

The design stage is a very important stage in the entire product development process, and it includes various contents. You can take a look at this table:

Each design item here should be discussed and refined repeatedly to reach a consensus within the team. This ensures that the design is reasonable and reduces the likelihood of rework. I would like to remind you that both the technical solution and the implementation should be carefully discussed and unanimously approved. Otherwise, if there are design flaws in the technical solution, you will bear most of the responsibility in the end.

For backend developers, sufficient research should be done before designing technical solutions. A technical solution should not only survey excellent implementations in the industry but also understand how similar technologies are implemented by competitors. Only in this way can we ensure that our technology is implemented in the best possible way.

In addition, some design items can be carried out in parallel in this stage to shorten the duration of the design stage. For example, product design and technical design can be carried out simultaneously. Moreover, if your team has testers, it is best to involve them in the development stage as well, which will benefit the subsequent testing. The output of this phase is a series of design documents, which will guide the entire development process that follows.

Development Phase #

As the name suggests, the development phase is the main battlefield for developers, and it may also be the longest phase. During this phase, developers code and implement the product requirements based on the technical design documents.

The development phase is the core phase of the entire project and involves various tasks. The specific steps for each Go project may vary. I have summarized the common steps in the development phase in the following diagram to give you an overall understanding.

Let’s take a closer look at the steps presented in this diagram. The development phase can be divided into two parts: “Development” and “Build”. Let’s start with development.

Firstly, we need to establish a Git workflow specification that all developers should follow. The most commonly used workflows are Git Flow workflow or Forking workflow.

To improve development efficiency, more and more developers are using code generation to generate some code. Therefore, before actual compilation, it may be necessary to generate code, such as generating .pb.go files, API documentation, test cases, error codes, etc. My suggestion is to consider how to generate code automatically as much as possible in project development. This not only improves development efficiency but also reduces errors.

For an open-source project, we may also need to check if newly added files have copyright information. In addition, depending on the project, there may be other different steps in the development phase. At the end of the process, static code checks, unit tests, and compilation are usually performed. After compilation, we can start the service and perform self-testing.

After self-testing, we can follow Git Flow workflow to push the development branch to the code hosting platform for code review. After the code review is approved, we can merge the code into the develop branch.

Next is the build phase. It is best to leverage a CI/CD platform for automation to improve the build efficiency.

The code merged into the develop branch also needs to undergo code scanning, unit testing, and compilation and packaging. Finally, we need to perform archiving, which is to upload the compiled binary file or Docker image to an artifact or image repository.

I have just taken you through the common steps in the development phase. As you can see, there are many steps in the entire development phase, and they are all frequently performed operations. So how to improve efficiency? Here, I recommend two methods:

Centralize and manage the steps in the development phase using a Makefile.
Automate the steps in the build phase using a CI/CD platform.

You also need to pay special attention to this point: Ensure that the code is thoroughly tested before finally merging it into the master branch. This requires us to rely on the webhook capabilities provided by the code management platform to trigger CI/CD jobs, scan and test the code upon submission, and finally compile and package it, with the successful execution of the entire job as a prerequisite for merging the code.

The outputs of the development phase are the source code that meets the requirements, development documentation, and the compiled archive files.

Testing Phase #

The testing phase is managed by the test engineer (also known as the quality engineer). The main process of this phase is as follows: the test engineer creates a test plan and writes test cases based on the requirements document, and then collaborates with the development team to review the test plan and test cases. Once the review is approved, the test engineer will proceed with testing the service according to the test plan and test cases.

To improve the overall development efficiency, the creation of the test plan and the writing of test cases can be done in parallel with the development phase.

When delivering to the testing phase, the developers should provide a self-test report, self-test cases, and installation and deployment documentation. Here, I want to emphasize that during the testing phase, in order to ensure that the project is released on time without blocking the testing process, the developers should prioritize the resolution of bugs reported by the testing team, especially the blocking bugs. To minimize unnecessary communication and troubleshooting, the installation and deployment documentation should be as detailed and accurate as possible.

In addition, you can also keep track of the testing progress and understand the difficulties encountered by the testing team. In actual work, some testing team members may not be good at or may not be able to timely communicate the difficulties they encounter. Sometimes, an issue that can be resolved by the development team in one minute may take the testing team several hours or even longer to resolve.

Of course, it is almost impossible for test cases to cover the entire code branch, so for some challenging and hidden tests, the developers need to strengthen their own testing.

Finally, after the testing of a major feature is completed, it’s a good idea to treat the testing team to a meal to strengthen relationships and improve collaboration for future projects.

The deliverables of the testing phase include the source code that meets the product requirements and release conditions, as well as the compiled archive files.

Release Phase #

The release phase mainly involves deploying the software to production. To ensure the efficiency and quality of the release, we need to follow a certain release process, as shown in the following diagram:

The release phase can be further divided into three sub-phases: code release, release approval, and service release. Next, I will provide you with a detailed introduction to these three sub-phases. Let’s start with code release.

First, the developers need to merge the tested code into the main branch, usually the master branch, and generate a version number. Then, they need to tag the latest commit with the version number. After that, the code can be pushed to the code hosting platform, and the CI process is triggered. The CI process typically includes code scanning, unit testing, compilation, and finally, publishing the build artifacts to the artifact repository. Any additional features can be added to the CI process as needed.

Next, we move on to the release approval phase. First, resource allocation is required. The resource allocation process may take a long time, so it is best to initiate the allocation request as early as possible, even during the testing phase. During the resource allocation phase, various resources such as servers, MySQL, Redis, Kafka, etc., can be requested.

Resource allocation is usually a process where developers submit demands to the operations team, who then prepares the requested resources by the specified time. If physical machines are requested, the allocation process may take a long time. However, nowadays, more and more projects choose containerized deployment, which can greatly shorten the resource allocation period. If deploying business applications on serverless container platforms such as Tencent Cloud Elastic Container, resources can be provisioned in seconds. Therefore, I also recommend giving priority to containerized deployment.

Before the release, a release plan needs to be created, which should provide detailed descriptions of the changes in this release, such as the scope of the changes, release plan, test results, verification and rollback plan, etc. Here, you need to pay attention to thoroughly review the impact points of the changes when creating the release plan. For example, whether there are any incompatible changes, whether configuration or database changes are required, etc. Any omissions may cause production failures and negatively impact product reputation and user experience.

Next, a release ticket needs to be created. The release plan can be attached to the release ticket, and other release contents can be filled in accordance with the team’s requirements. The release plan needs to be aligned with relevant stakeholders to clarify responsibilities and define the process. The release ticket is ultimately submitted to the approver (usually the technical leader) for approval. Only after approval, the deployment can proceed. Finally, we can enter the service release phase and deploy the service to the live network. During the formal deployment, the application needs to be deployed to a pre-production environment. In the pre-production environment, product personnel, testing personnel, and development personnel will each validate the product. Among them, product personnel mainly verify the smoothness of the product’s functions, while development and testing personnel mainly verify if there are any bugs in the product. Only after the pre-production environment validation is successful can the product be officially released to the live network.

Here, I strongly recommend writing some automated test cases and conducting a thorough regression test on the live service after it is released on the live network. Through this automated test, the live functions can be verified at minimal cost and in the quickest way, thus ensuring the quality of the release.

In addition, we also need to pay attention to the fact that there may be multiple regions in the live network, and after each region is released, a live network validation needs to be conducted.

The output of the release phase is the formally launched software.

Operational Phase #

The final stage of the development process is the operational phase, which is mainly divided into product operations and operations and maintenance (O&M).

Product Operations: By carrying out a series of operational activities such as offline technical salons, online free courses, improving keyword rankings, or publishing technical promotional articles, the overall product awareness can be enhanced, the number of users can be increased, and the monthly and daily active users can be improved.
Operations and Maintenance: Responsible for operations and maintenance engineers, the core goal is to ensure the stable operation of the system and to promptly detect and repair any abnormality. The long-term goal is to improve the overall system architecture through technical means or processes, reduce manpower input, improve operational efficiency, and enhance the robustness and recoverability of the system.

From the above, it can be seen that operations and maintenance belong to the technical category, while product operations belong to the product category. Don’t confuse the two. To deepen your understanding and memory, I have summarized these contents in the following figure.

In the operations phase, the main responsibility of development personnel is to assist in solving live network bugs and optimize deployment architecture. Of course, development personnel may also need to cooperate with operations personnel to develop operational interfaces for their use.

We have now completed learning this set of relatively standard development processes in the industry. In the learning process, you must have also noticed that the entire development process involves many roles, with different roles participating in different stages and responsible for different tasks. Here, I will provide you with some additional information on these core roles and their responsibilities.

I have included this additional content in a diagram and a table below. These roles and responsibilities are relatively easy to understand, and you don’t need to memorize them. Just have a general impression for now.

The specific responsibilities are shown in the table below.

Summary #

When developing Go projects, it is important to master the project development process. Understanding the development process will make project development more transparent and facilitate the formulation of detailed work tasks.

So how do we design the project development process? You can design it according to your needs. When designing it, there are some points that you must pay attention to, such as supporting high release efficiency and release quality, supporting rapid iteration, and ensuring that the process is reasonable and scalable, etc.

If you don’t want to design it yourself, that’s also possible. In this lecture, I introduced a relatively general and standard development process, which can be directly used as the design for your own project development process if suitable.

This development process consists of six stages: requirement stage, design stage, development stage, testing stage, release stage, and operation stage. I have summarized these stages and the key points of each stage in the following diagram.

Exercise #

Recall the specific work content of the development phase. If you feel that these work contents are not enough to meet the needs of the development phase, what else do you need to add?
Consider and research which tools can help implement the entire process and automate tasks in the process. See how they can improve our development efficiency.

The development process will vary depending on the team, project, and requirements. If you have a better process plan, feel free to communicate and discuss with me in the comments. See you in the next lesson!