09 Research and Development Process Design How to Manage the Lifecycle of an Application

09 Research and Development Process Design How to Manage the Lifecycle of an Application #

Hello, I’m Kong Lingfei. Today, let’s talk about how to manage the lifecycle of an application.

In the previous lesson, we introduced a relatively standard development process, which can ensure that we efficiently develop an excellent Go project. In this lesson, let’s take a look at how to manage our Go project, in other words, how to manage the lifecycle of an application.

So, how do we understand the management of the lifecycle of an application? It actually means using good tools or methods to manage the application throughout its entire lifecycle, in order to improve the efficiency and quality of development.

How can we design an excellent means of managing the lifecycle of an application? Just like the approach of “designing” a development process, you can design it yourself or adopt excellent management methods that have been accumulated in the industry. Similarly, I recommend that you adopt existing best practices, because it is too difficult to reinvent the wheel or create a good one.

Therefore, in this lesson, we will learn together about the excellent management methods accumulated by the industry at different times, as well as my experience and suggestions on these management methods, to help you choose the most suitable one.

What are the application lifecycle management technologies? #

So, what are the application lifecycle management technologies?

Here, let me give you an overall introduction first, so you can have a general idea. Later, we can discuss each of them in detail. We can understand application lifecycle management technologies from two dimensions.

The first dimension is the evolution dimension. Initially, application lifecycle management was mainly managed through development models, and over time, waterfall model, iterative model, and agile model appeared in chronological order. Then, to address some issues in the development models, another management technology emerged, which is CI/CD technology. With the maturity of CI/CD technology, another more advanced management technology called DevOps came into being.

The second dimension is the category of management technologies. Application lifecycle management technologies can be divided into two categories:

Development models, which ensure the efficiency of the entire development process.
DevOps, which mainly improves software release efficiency and quality by coordinating collaboration between different departments. DevOps includes various technologies, primarily CI/CD and various Ops, such as AIOps, ChatOps, GitOps, NoOps, etc. Among them, CI/CD technology improves software release efficiency and quality, while Ops technology enhances software operation and maintenance efficiency.

Although these application lifecycle management technologies are different, they support and interact with each other. The development models focus on the development process, CI/CD in the DevOps technology focuses on the process, and Ops focuses on the practice.

To help you understand, I have summarized the following diagram for your reference.

Although there are many different management technologies involved in these two dimensions, they can be categorized into only a few types. Therefore, to explain these technologies clearly and logically, I will start with the evolution dimension, which is the order of: development models (waterfall model -> iterative model -> agile model) -> CI/CD -> DevOps.

You may wonder, since it is an evolution, these technologies must have advantages and disadvantages. How should I choose? Should I choose the technologies that appeared later?

To address your question, here, for the choice of development models and DevOps technologies, I will give you my advice in advance: For development models, I recommend choosing the agile model because it is better suited for rapid iteration in the Internet era. For DevOps, it is important to prioritize the implementation of CI/CD technology, and then explore the implementation of ChatOps technology. If conditions permit, actively explore AIOps and GitOps.

Next, let’s discuss these application lifecycle management methods in detail, starting with the development models that focus on the development process.

Development Models #

There are three main development models, in the order of evolution: waterfall model -> iterative model -> agile model, let’s take a look at each one.

Waterfall Model #

In the early stages, the widely used software development model was the waterfall model. Systems like RHEL and Fedora are examples of systems that use the waterfall model.

The waterfall model advances the development process according to pre-planned stages. For example, development tasks are executed in a sequential order of requirement phase, design phase, development phase, testing phase, release phase, and operation phase. Each stage must be perfectly completed before entering the next stage, with deliverables exchanged through documents. The entire process is shown in the following diagram.

The biggest advantage of the waterfall model is its simplicity. It strictly advances the development process according to stages, with clear procedures, making it suitable for project-based application delivery.

However, it also has its disadvantages, the most prominent being:

It is only delivered to the customer in the final stage of the project. If the customer discovers problems after delivery, it can be very difficult and costly to make changes.
The development cycle is relatively long and it is difficult to meet the demands of the internet era for rapid product iteration.

To address these two issues, the iterative development model was created.

Iterative Model #

The iterative model is a development process that is completely opposite to the waterfall model: development tasks are divided into a series of iterations, with each iteration representing a complete process from design to implementation. It does not require every task of each stage to be done perfectly, but rather construct the main functions first and then continuously improve based on customer feedback.

Iterative development can help improve products and control progress. Its flexibility greatly enhances the ability to adapt to changes in requirements, overcoming the characteristics of high risk, difficulty in change, and low reusability.

However, the problem with the iterative model is that it focuses more on the development process and pays less attention to accelerating and optimizing the project development process from a project management perspective. The next model to be discussed, the agile model, addresses this drawback.

Agile Model #

The agile model divides a large requirement into multiple small iterations that can be completed in stages, with each iteration delivering usable software. Throughout the development process, the software should always be in a usable state.

The representative development model in the agile model is the Scrum development model. There is a lot of information about the Scrum development model on the internet, you can take a look.

In the agile model, a large requirement will be divided into many small iterations. This means that there will be many processes of development, build, test, release, and deployment in the development process. This high frequency of operations can bring a heavy workload to the developers, operators, and testers, reducing work efficiency. To solve this problem, CI/CD (Continuous Integration/Continuous Deployment) technology was born.

CI/CD: Automated Build and Deployment of Applications #

CI/CD technology uses automation to quickly perform tasks such as code checking, testing, building, and deployment, thus improving development efficiency and addressing the drawbacks of agile mode.

CI/CD consists of three core concepts:

CI: Continuous Integration
CD: Continuous Delivery
CD: Continuous Deployment

CI is easy to understand, but many developers have trouble distinguishing between the two CDs. Here, I will explain these three core concepts in detail.

First, let’s talk about Continuous Integration. Its meaning is to frequently (multiple times a day) merge developers’ code into the main branch. Its process is as follows: after the developer completes code development and pushes it to the git repository, the CI tool immediately scans, tests, and builds the code, and provides feedback to the developer. Once the CI process is successful, the code is merged into the main branch.

The CI process exposes issues in the application software during the development stage, which gives developers more confidence when delivering code. Because the CI process has a lot of content and is executed frequently, it requires automation tools to support it.

Next is Continuous Delivery, which refers to a software approach that allows reliable releases of software in shorter cycles.

Building on top of Continuous Integration, Continuous Delivery automatically deploys the built artifacts in the target environment. The target environment can be a test environment, staging environment, or production environment.

Generally speaking, Continuous Deployment can automatically deploy services to a test environment or staging environment. However, since there is a certain level of risk in deploying to the production environment, manual operation is required. The benefit of manual operation is that it allows relevant personnel to assess the risk of the release and ensure its correctness.

Finally, there’s Continuous Deployment. Building on top of Continuous Delivery, Continuous Deployment automatically deploys thoroughly tested code to the production environment without the need for any human review. Continuous Deployment emphasizes automated deployment and represents the highest stage of delivery.

We can use the following diagram to understand the relationship between Continuous Integration, Continuous Delivery, and Continuous Deployment.

Continuous Integration, Continuous Delivery, and Continuous Deployment emphasize continuity, which means supporting frequent integration, delivery, and deployment. This cannot be achieved without the support of automation tools. Without these tools, CI/CD is not implementable. The core point of Continuous Integration lies in code, the core point of Continuous Delivery lies in deliverable artifacts, and the core point of Continuous Deployment lies in automated deployment.

DevOps: Integrated Development and Operations #

The maturity of CI/CD technology has accelerated the maturity and implementation of DevOps, a technology for managing the application lifecycle.

DevOps, a combination of Development and Operations, is a collective term for a set of processes, methods, and systems used to facilitate communication, collaboration, and integration between development (application/software engineering), technical operations, and quality assurance (QA) departments. The collaboration of these three departments can improve software quality and enable rapid software releases. The figure below illustrates this:

To achieve DevOps, support from tools or processes is needed. CI/CD can adequately support the software development model of DevOps. Without automated tools and processes for CI/CD, DevOps would be meaningless. CI/CD makes DevOps feasible.

Are you feeling a bit confused at this point? You may wonder, what is the difference between DevOps and CI/CD? In fact, this is also a question that puzzles many developers. Here, we can understand it like this: DevOps != CI/CD. DevOps is a collective term for a set of processes, methods, and systems, while CI/CD is just a technique for software build and release.

DevOps technology has been around for a while, but it hasn’t been well implemented because there were no good tools to realize the concept of DevOps. However, with the emergence and maturity of containers and CI/CD technology, DevOps has become easier to implement. In other words, in recent years, more and more people have adopted DevOps practices to improve development efficiency.

With the advancement of technology, many operations (Ops) practices have emerged to achieve highly automated operations and management. Now, let’s take a look at the four Ops practices in DevOps: AIOps, ChatOps, GitOps, and NoOps.

AIOps: Intelligent Operations #

In 2016, Gartner proposed a new generation of IT operations that utilizes AI technology, known as AIOps (Intelligent Operations). AIOps intelligently operates IT systems through AI techniques, which collect massive amounts of operational data and use machine learning algorithms to intelligently locate and fix faults.

In other words, AIOps enhances IT operations automation with intelligence, thereby reducing manpower costs.

As IT infrastructure scales up and becomes more complex, along with exponential growth in the scale and number of enterprise applications, traditional manual/automated operations can no longer cope with the increasingly heavy operational workload. AIOps provides a solution. Many teams in major companies like Tencent and Alibaba have been trying and using AIOps, enjoying the benefits it brings. For example, fault alerts are more sensitive and accurate, and common failures can be automatically fixed without the need for operations personnel to intervene.

ChatOps: Getting Things Done by Simply Chatting #

With the rise of enterprise communication tools like WeChat Work and DingTalk, a new concept called ChatOps has emerged in recent years.

In simple terms, ChatOps is the process of sending a command to a ChatBot in a chat tool, which then performs predefined operations. These operations can involve executing certain tools or calling specific interfaces and returning the execution results.

What are the advantages of this new intelligent working method? It allows team members to connect with various auxiliary tools through a ChatBot, completing work in a communication-driven manner. ChatOps can eliminate information silos between people, tools, and tools, thus improving collaboration experience and work efficiency.

The workflow of ChatOps is illustrated in the figure below:

Development, operations, and testing personnel trigger tasks through the @chat window’s Bot. The backend of the Bot interfaces with different systems through API calls and other methods to complete various tasks, such as continuous integration, testing, and deployment. The Bot can be our own development or an open-source one. Currently, there are many popular bots to choose from, including Hubot, Lita, Errbot, StackStorm, etc. Using ChatOps can bring several benefits:

User-friendly and Convenient: All operations are done in the same chat interface, sending commands to the bot in a chat format. This eliminates the need for tedious operations like opening different systems and executing different actions, making it more user-friendly and convenient.
Transparent Information: All colleagues in the same chat interface can see the commands and results sent by others, eliminating communication barriers and making teamwork smoother with a traceable work history.
Mobile-friendly: Commands can be sent to the bot and tasks can be executed on mobile devices, making mobile office work possible.
Building DevOps Culture: By conversing with the bot, the understanding and usage cost of project development for team members can be reduced, making DevOps easier to implement and promote.

GitOps: A Cloud-Native Continuous Delivery Model #

GitOps is a way of continuous delivery. Its core idea is to store the declarative infrastructure (YAML) and application code in a Git version control system. Using Git as the core of the delivery pipeline, developers can submit pull requests and use Git to accelerate and simplify Kubernetes application deployment and operations tasks.

With tools like Git, developers can focus on functional development rather than software operations, thereby improving software development efficiency and iteration speed.

Using GitOps has many advantages, and the most important one is that when changes are made to the code using Git, GitOps can automatically apply these changes to the infrastructure of the application. Because the entire process is automated, deployment time is reduced. And because Git code is traceable, our deployed applications can roll back stably and reproducibly.

We can understand GitOps from the concepts and processes, which have three key concepts:

Declarative Container Orchestration: Use Kubernetes YAML resource definition files to define how the application is deployed.
Immutable Infrastructure: Each component of the infrastructure can be automatically deployed, and once deployed, the component cannot be changed. If a change is needed, a new component needs to be redeployed. For example, a Pod in Kubernetes is an immutable infrastructure component.
Continuous Synchronization: Continuously monitor the Git repository and reflect any state changes to the Kubernetes cluster.

The workflow of GitOps is as follows:

First, after developers finish coding, they push the code to the Git repository, triggering the Continuous Integration (CI) process. The CI process compiles and builds a Docker image and pushes the image to a Docker image repository. The push action triggers a push event, which is notified to the Config Updater service through a webhook. The Config Updater service retrieves the latest push image name from the webhook request and updates the Kubernetes YAML file in the Git repository.

Then, the GitOps Deploy Operator service detects the changes in the YAML file and retrieves the modified files from the Git repository, deploying the images to the Kubernetes cluster. Both the Config Updater and Deploy Operator components need to be designed and developed by developers.

NoOps: No Operations #

NoOps means fully automated operations. In NoOps, there is no need for collaboration between developers and operations personnel. It combines microservices, low code, and serverless technologies, allowing developers to focus only on business development in the software lifecycle, while all maintenance is handled by cloud service providers.

Undoubtedly, NoOps is the ultimate form of operations. In my opinion, like DevOps, it is more of a concept that requires a lot of technology and means to support it. The development of the entire operations technology is also evolving towards NoOps. For example, GitOps and AIOps allow us to minimize operations, and Serverless technology can even eliminate the need for operations completely. I believe that in the future, NoOps will become a popular and implementable concept just like Serverless is now.

How to Choose the Right Application Lifecycle Management Technology? #

Alright, by now we have learned about the main application lifecycle management technologies. So, how do we choose the right one for practical development? In my opinion, you can consider the following aspects.

First, choose a suitable development model based on your team and project. If the project is large, the requirements change frequently, and rapid iteration is required, I suggest choosing the agile development model. Agile development is a popular choice for many large companies in the era of the internet.

Next, establish your own CI/CD process. Any code changes must be validated through the CI/CD process before merging into the master branch. I recommend setting quality gates in the CI/CD process to ensure the quality of merged code.

Next, in addition to establishing a CI/CD system, I also suggest implementing ChatOps in your work and automating tasks as much as possible. Use ChatOps to trigger the automation process. With the maturity and development of enterprise chat software such as WeChat Work and DingTalk, ChatOps has become popular and well-established.

Lastly, GitOps and AIOps can automate deployment and operations to the extreme and are worth exploring if your team has the resources.

You may wonder, how do big companies manage the application lifecycle?

Large companies generally adopt the agile development model to meet the demand for rapid iteration of applications in the internet era. For example, Tencent’s TAPD and Coding’s Scrum Agile Management are both agile development platforms. CI/CD is mandatory, ChatOps is widely used, there are many successful cases of AIOps, GitOps is still in the exploration stage, and NoOps is still in the theoretical stage.

Summary #

In this lecture, I introduced application lifecycle management techniques from a technical evolution perspective. These techniques can improve the efficiency and quality of application development.

Application lifecycle management was initially managed through development models. In the development models, I introduced the waterfall model, iterative model, and agile model along a timeline. The agile model has become more widely used as it meets the demands of rapid iteration in the internet era.

In the agile model, we need to frequently build and deploy our applications, which brings additional workload to developers. To address this issue, CI/CD technology was introduced. CI/CD automates tasks such as code checking, testing, building, and deployment, improving development efficiency and ensuring code quality to a certain extent. In addition, CI/CD technology enables the feasibility of DevOps. More and more teams are adopting DevOps to manage the lifecycle of applications.

Furthermore, I also introduced several concepts that are easily confused.

Continuous delivery and continuous deployment. Both involve continuous deployment of applications, but in continuous deployment, the entire process is automated, while in continuous delivery, manual approval is required before releasing the application to the production environment.
CI/CD and DevOps. DevOps is a collective term for a set of processes, methods, and systems, which also includes CI/CD technology. CI/CD, on the other hand, is an automated technology that supports the implementation of DevOps philosophy.

Finally, regarding how to manage the lifecycle of applications, I provided some recommendations: Choose agile model as the preferred development model as it is better suited for the demands of rapid iteration in the internet era. First prioritize the implementation of CI/CD technology in DevOps, then explore the adoption of ChatOps if possible, and actively explore AIOps and GitOps if conditions allow.

Homework Exercise #

Learn and use GitHub Actions to automatically perform static code analysis after submitting code.
Try adding an Enterprise WeChat bot that can automatically print “hello world” every day, and consider what automation tasks can be implemented using this bot.

Look forward to seeing your thoughts and answers in the comments area. See you in the next lecture!