10 Configuration Management, The Often Overlooked DevOps Engineering Practice Foundation #
Hello, I am Shi Xuefeng. Starting from today, this column officially enters the section on engineering practices. In the DevOps system, engineering practices play a very important role and are closely related to our daily work. Because of this, DevOps includes a large number of engineering practices, many of which we are familiar with, such as continuous integration, automated testing, and automated deployment, which are basically prerequisites for practicing DevOps.
However, there are some practices that are often overlooked, but this does not mean that they have been eliminated or are less important. On the contrary, they are also the foundation for DevOps to deliver value, and configuration management is one of them. Its concepts are ubiquitous in the software development process and can be said to be the foundation of the entire DevOps engineering practice. So today, let’s talk about configuration management.
After saying so much, what exactly is software configuration management?
Those familiar with operations may say, isn’t it similar to environment configuration management tools like Ansible or Saltstack? Some may also say that CMDB configuration management databases are configuration management too, right? These statements are not wrong. The concept of configuration management is widely used in the field of software development and can be said to be ubiquitous, but the concepts mentioned earlier are local definitions within specific domains.
The configuration management I want to talk about today is a macro concept that stands from the perspective of the entire software delivery lifecycle, standardizes and manages the entire development process, controls the change process, makes collaboration smoother, and ensures the integrity, consistency, and traceability of the entire delivery process.
Seeing this, I estimate that you may be overwhelmed. Indeed, the theoretical system of configuration management is very extensive. But it doesn’t matter, you only need to remember the four core concepts. These four concepts are: standardize version changes, include everything in version control, have full process traceability, and single trusted data source.
1. Standardization of Version Changes #
Version control is a very core concept in configuration management, and for software, the most core asset is the source code. Nowadays, many companies are using tools like Git and SVN to manage source code, which are actually version control systems. Versions describe the state of the software deliverables, and it can be said that versions have existed since the first line of software code was written.
Modern software development is becoming more and more complex, often requiring collaboration among multiple developers. Therefore, how to manage each developer’s versions and effectively integrate them together has become a problem. In fact, version control systems are designed to solve this problem. Just imagine, if there is no such system, all the code is stored locally, and not to mention others, even oneself will be confused about which is the latest code. Therefore, it would be chaos when everyone’s code is integrated together!
Not only that, if a serious problem occurs online and the corresponding historical version cannot be found, the only option is to directly release the latest code, which is a disaster.
Another core concept in configuration management is change. Any change made to the software can be called a change, such as a requirement, a line of code, or even an environmental configuration. Versions come from changes. For changes, the core is to record: who, at what time, made what changes, what specific contents were changed, and who approved it.
At first glance, it doesn’t seem that complicated, because modern version control systems generally have the ability to record changes. So, as long as a version control system is used, can change management be achieved?
Indeed, the appearance of version control systems has greatly simplified the cost of managing changes, at least there is no need to manually record them. However, from another perspective, using version control systems effectively also requires a set of rules and behavior standards.
For example, version control systems need to be integrated with the company’s unified authentication system, which means that anyone who wants to access the version control system needs to go through the company’s unified login authentication. At the same time, when using Git, you need to configure the local information correctly, especially the username and email information, so that complete user information can be generated when submitting. In addition, the system itself needs to add relevant verification mechanisms to avoid invalid information being submitted to the repository due to incorrect employee configurations.
The change description is generally the commit record of the version control system. A complete commit record should include at least the following aspects:
- Commit Summary: Briefly describe which functions and issues are implemented by this change.
- Commit Details: Provide detailed description of the change, including its details, how it was changed, any potential risks and outstanding issues, etc.
- Associated Requirements: Identify which change caused this commit modification, and include the upstream system number to associate the commit with the original change.
These changes should follow a standardized format and have corresponding format instructions and writing styles, such as what keywords to use, the length of each line, whether to use a comma, space, or semicolon as the separator for change numbers, etc. If each change record is written according to this standard, the cost is actually quite high, not to mention writing in English, as the English expression and presentation style are another challenge.
Let me share a great commit comment with you as an example.
switch to Flask-XML-RPC dependency
CR: PBX-2222
The Flask-XML-RPC-Re fork has Python 3 support, but it has a couple
other problems.
test suite does not pass
latest code is not tagged
uncompiled source code is not distributed via PyPI
The Flask-XML-RPC module is essentially dead upstream, but it is
packaged in EPEL 7 and Fedora. This module will get us far enough to- the
point that we can complete phase one for this project.
When we care about Python 3, we can drop XML-RPC entirely and get the
service consumers to switch to a REST API instead.
(Note, with this change, the Travis CI tests will fail for Python 3.- The
solution is to drop XML-RPC support.)
At this point, someone may ask, is it worth the effort to do this? From a local perspective, yes, it seems so. However, from another perspective, when others see your changes or review your code, if they can understand your intentions clearly through the commit records without having to call you and ask you to explain again, the time saved is much greater than the time spent writing the commit records.
So you see, a set of standardized rules and behavior habits can reduce the cost of communication in the collaboration process. Doing things right the first time is the important significance of standards and specifications.
Of course, if the standardized process relies entirely on the self-consciousness of people to guarantee, it would be unreliable. After all, people are prone to making mistakes, which may affect the execution of standards. Therefore, it is best to use automated means to ensure process standardization after the team has gradually formed a set of specifications through continuous adjustment.
This approach has two advantages: on the one hand, it can reduce the influence of human factors. If you don’t follow the standards, you will have difficulties, and it also reduces the possibility of human manipulation. For example, sometimes, out of laziness, the same change number is written for every commit, which indeed meets the requirements of standardization, but it produces a large amount of invalid data. At this time, you can add some verification mechanisms, such as only allowing the addition of changes assigned to you, or only allowing change numbers in open status, and so on. On the other hand, after standardization, many repetitive tasks can be automated, and standardized information is convenient for computer analysis and extraction, which can improve the efficiency of the process flow.
It can be said that standardization is the premise of automation, and automation is the core practice of DevOps. In this regard, it can be said that configuration management is the foundation of DevOps engineering practices.
2. Include Everything in Version Control #
If there’s one thing you should remember from today’s lecture, it is to include everything in version control. This is the golden rule of configuration management. You may ask, what kind of content should be included in version control? I will answer without hesitation: “Everything!” This includes source code, configuration files, test compilation scripts, pipeline configurations, environment configurations, database changes, and so on. Everything you can think of should have a version and be included in control.
This is because software itself is a complex system, and any change can bring problems. Therefore, full lifecycle version control gives us the ability to trace everything and quickly rollback to a specific version status, which is crucial for issue localization and resolution.
I have encountered a problem before. When an iOS application was released in a gray version, everything was normal. But when the official version was released, there was an issue with the inability to download. As the release was approaching, we had to find the cause of the issue. It can be said that everyone in the team joined the effort, and the team even started blaming each other. The developers claimed that there were no code changes, so it must be an operations problem. The operations team argued that the environment had not been modified, so it must be a development issue. The truth only emerged when it was finally discovered that one of the tools had been upgraded, and the default value of a certain parameter changed from “off” to “on”.
You see, if you include all content in version control, it is a matter of minutes to quickly compare two versions and identify the differences. In this way, resolving such issues becomes straightforward. You can simply revert all the changes if needed.
The value of including everything in version control goes beyond this. In fact, many DevOps practices are implemented based on version control. For example, regarding environment management, it is recommended to manage environments using infrastructure as code. This means describing complex environment configurations using code and including it in the version control system. In this way, any environment change can be treated just like code submission. Not only is the content of the change clear, but automation can also be easily achieved. Complex tasks can be simplified, and anyone can make environment changes.
In this way, the gap between development and operations gradually disappears, and this is the essence of DevOps. Therefore, the popular “something as code” in the industry is based on version control.
However, I need to clarify that including everything in version control is not equivalent to putting all content in Git. Sometimes, we easily confuse capabilities and tools. Git is just a popular version control system, and what is emphasized here is a capability. The tools are only carriers of this capability. For example, Git is not good at managing large files, so those files can be stored in Artifactory or other self-built platforms.
For self-built systems, there are many ways to achieve version control implementation. For example, for each change, a new set of data can be inserted, or Git, as a mature tool, can be directly reused as the backend. The only constant requirement is that regardless of the system and tools used, the capability of version control needs to be considered.
In addition, when practicing to include everything in version control, you can follow a simple principle. If you are unsure whether something should be included in version control, a simple judgment method is: If this artifact can be reproduced by other artifacts, then it can be managed as a product and does not need to be included in version control.
For example, if a software package can be generated again through source code and tools, then the code, tools, and packaging environment should be included in control, while the generated software package can be treated as a product. If the test report of the software can be automatically regenerated through a test management platform, then it can also be considered as a product. However, the prerequisite is that the test management platform can regenerate the test report for each version.
3. Full Process Traceability #
For traditional industries, the ability to trace the entire process is not an optional feature but a mandatory one. Industries such as aerospace, manufacturing, and finance have strict control over changes. Once a problem occurs, it is necessary to trace all the relevant data at that time, such as software source code, test reports, operating environments, and so on. If there is a lack of management and unable to provide evidence to prove that sufficient validation was done based on the objective situation at that time, it will face heavy fines and compensation. This is not a trivial matter. Blockchain technology, which has recently become popular, besides issuing coins, has its most typical use case as full process traceability. Therefore, technology may change rapidly, but many concepts remain unchanged for a long time.
For configuration management, in addition to traceability, there is another important value, which is documenting the association and dependency relationship. How do we understand this statement? Let me ask a question first: in your company, can you quickly identify the associated code, version, test cases, release records, defect information, user feedback, and deployment monitoring data for any given requirement? Can you identify the environment, middleware, upstream and downstream systems, services, and data that an application depends on?
If your answer is “yes”, then congratulations, your company is doing a great job. However, the majority of companies cannot achieve this. Because it not only requires linking and data integration between systems, but also involves a whole set of complete management mechanisms.
DevOps emphasizes value orientation and emphasizes shared goals within the team, which is actually the business goals. But the reality is that the dimensions of concern for business, development, testing, and operations are different. Businesses are concerned with whether the requirements have been deployed, while developers are concerned with whether the code for these requirements has been integrated, and operations are concerned with whether the version containing this code has been deployed. Therefore, if these pieces of information cannot be connected and linked, true full process traceability cannot be achieved.
Regarding this issue, my suggestion is to grasp the source and establish a mainline. The so-called source, for software development, is the most fundamental requirement. All changes originate from requirements. Therefore, the first step is to manage requirements uniformly, whether they are development requirements, testing requirements, or operational requirements.
Next, starting from the requirements, associate downstream processes, link data. This requires the support of system capabilities and rules. For example, each change must be forcibly associated with a requirement number, and different processes are defined for different requirement levels. This approach can not only reduce unnecessary approval processes, give the team a certain level of flexibility, but also achieve the goal of full process control. This is a relatively long process, but as the saying goes, “a journey of a thousand miles begins with a single step.” DevOps also needs to be built step by step.
4. Single Trusted Data Source #
Finally, I want to talk specifically about the concept of a single trusted data source. Many people do not understand what this means, so let me give you an example.
There is a popular term on the internet called “official announcement” or “官宣” in Chinese. Typically, information from an official announcement is considered reliable and highly credible. However, the problem arises when there are multiple channels for official announcements and the information they provide is inconsistent. In such cases, how do you know which one to believe? This is where the concept of a single trusted data source becomes important.
Imagine this: we put a lot of effort into building a version control capability, but if the data source itself is unreliable and lacks unified control, wouldn’t that be a waste of time? Therefore, for software development, it is crucial to have unified control:
- For code, there should be a unified version control system to avoid chaotic code management.
- For versions, there should be a unified distribution channel to prevent unauthorized releases.
- For the components on which development depends, there should be a unified source from which they can be obtained. This is not only important for security control but also crucial for maintaining information consistency within the organization.
At the same time, a single trusted data source should also cover the management of internal metadata within an organization. For example, it is common for different departments within an organization to use different names for the same application. When integrating these platforms, it becomes a challenge because the same thing is referred to differently. Another example is when the information security team maintains a list of applications, but in reality, many of these applications have been discontinued and are no longer maintained in the business systems. This not only leads to resource wastage but also poses significant security risks.
Many of these problems often arise due to a lack of unified top-level planning and design. Therefore, when establishing configuration management capabilities, it is important to pay special attention to this aspect.
Summary #
Today, I introduced you to the basic configuration management of DevOps engineering practices, as well as the four principles of configuration management, including standardized version changes, including everything in version control, traceability throughout the process, and a single trusted data source. I hope this helps you grasp the overall concept of configuration management.
Although configuration management may not seem significant, it is just like the classic saying: “The tranquil years are because someone is carrying the weight for you.” For any company, information overload is the norm, and the greatest value of configuration management is to serialize information, effectively organize, classify, record, and associate information. In addition, software development standards and orderliness are also the sources of improved collaborative efficiency. Therefore, emphasizing the importance of configuration management cannot be overstated.
Reflection Questions #
What difficulties have you encountered in terms of configuration management in your organization? And how was your configuration management system established? Have you ever experienced any interesting stories where “speaking different languages” occurred due to a lack of a single trusted data source?
Feel free to write down your thoughts and answers in the comments section. Let’s discuss together and learn from each other. If you find this article helpful, please feel free to share it with your friends.