03 DevOps Implementation, Is it Tools First or Culture First #

Hello, I am Shi Xuefeng.

When an enterprise finally embraces the DevOps mindset and is determined to start implementing it, it always faces a dilemma: should tools or culture go first?

Indeed, in the theoretical framework of DevOps, both tools and culture play significant roles. When discussing this topic with others, we often divide into two different “camps,” debating endlessly, each side having its own reasons, making it difficult to convince each other. In the world of DevOps, the question of whether tools or culture should go first is like the question of whether soy milk should be sweet or salty—there has never been a definitive answer.

However, for many people who have just started exploring DevOps, if they don’t clarify this question, the path of subsequent DevOps practices is likely to deviate. So, in any case, I’ll dry this bowl of soy milk first. Today, let’s talk about this topic.

DevOps Tools #

With the deepening of the DevOps concept, various tools named after DevOps have sprung up around us like bamboo shoots after a spring rain. Even many well-established tools have voluntarily changed their product names to DevOps to adapt to the development of the DevOps era. The most representative example is Microsoft’s collaboration platform VSTS (Visual Studio Team Services), which was officially renamed Azure DevOps in September last year. This further confirms that DevOps has become the core concept of tool platform construction.

In the previous lecture, I mentioned that efficiency and quality are the core values of DevOps, and tools and automation are the most direct means to improve efficiency. Automation can be said to be the code of conduct for DevOps.

All manual processes in the software delivery process are areas that can be optimized in the future. Even in the field of operations and maintenance, ITIL (IT Infrastructure Library) has always been the cornerstone of operations and maintenance survival, but this does not prevent the concept of automation from gradually entering the ITIL process and continuously optimizing process efficiency on a controlled basis.

In addition, because everyone recognizes the value of automation, the introduction and construction of tool platforms have become one of the key factors that impress people about DevOps.

At the same time, many open-source tools in the industry have become quite mature. Excellent companies such as Netflix, Amazon, and Etsy are continuously opening up their internal tool platforms, providing abundant reference materials and usage examples.

Whether it is simply using these tools or based on them for secondary development, the cost is no longer as high. A slightly more mature small team can develop a tool in a very short time. Taking my previous team as an example, it took just over two months from building from scratch to the landing and promotion of the first product, and compared with similar products in the industry, it is not inferior.

However, this also brings a side effect, which is the proliferation of internal tool platforms in enterprises. Many homogeneous tools stagnate after completing the process from 0 to 1, falling into a cycle of repetition, which is obviously a waste of resources.

Of course, for advocates of tool determinism, this is not a big problem because introducing tools is the best implementation path for DevOps.

Sometimes, when you ask someone, “How is your company doing with DevOps?” You may get an answer like, “All of our teams have started using Jenkins.” It sounds strange. If achieving high efficiency in software delivery were as simple as using the latest and most powerful DevOps tools, then the Fortune 500 companies would have achieved DevOps long ago.

Many companies have introduced complete agile project management tools, but they use these tools in the same way as traditional project management, and there is no significant improvement in efficiency compared to before. The same principle applies to self-developed platforms. If you simply move the offline approval process online, it can indeed improve execution efficiency to some extent, but it is far from the desired transformation for the enterprise.

In the end, tools cannot solve people’s problems. This seemingly shortcut path cannot solve the fundamental problems of an enterprise. At this time, culture needs to come to the forefront.

DevOps Culture #

Before discussing the DevOps culture, let me share a story with you.

In the 1980s, there was an automobile manufacturing company called NUMMI in California, USA. At that time, the company was a subsidiary of General Motors (GM), but due to tense labor relations, it was the least profitable company under GM. The employees would come to work and spend their days drinking alcohol and gambling, making the factory a chaotic and unproductive place with an absenteeism rate of up to 20%. GM eventually couldn’t tolerate it anymore and decided to shut down the company.

Later, Toyota, a Japanese company, wanted to form a joint venture and build a factory in the United States. They reached an agreement with GM despite the United Auto Workers’ (UAW) demand to rehire the previously laid-off employees. Surprisingly, Toyota agreed because they believed that the issues in the NUMMI factory were more related to the system rather than the people.

Toyota then sent the newly recruited employees to Japan for training. After just three months, the entire company transformed, becoming the most profitable company within the GM group within six months.

This story shows that even with the same people, their productivity can vary greatly under different cultural systems.

Similar stories are not uncommon. A group of American experts once visited Japan to study production assembly lines and discovered something interesting.

In American companies, there was always someone using a rubber hammer to knock on car doors to check if they were properly installed. However, even with this inspection process, the quality of the car doors remained poor. On the other hand, there was no such role in Japanese factories.

Curious, they asked, “How do you ensure that the car doors are problem-free?” The Japanese experts responded, “We ensure it doesn’t have problems when designing the car doors.” You see, two companies using assembly line techniques yielded different results.

Similarly, in DevOps, if we rely on a person with a hammer to ensure product quality throughout our software delivery process, and when problems occur, blame it on the lack of skilled individuals with hammers, then maybe there is something wrong with the process itself.

Returning to the concept of culture itself, a good culture not only allows processes and tools to play a greater role but, more importantly, it triggers people to think about the flaws in the current processes and tools, leading to more optimization needs and continuous improvement towards better supporting business development.

However, the internal DevOps culture in an organization is intangible, and it is challenging to quantify a team’s culture level and subsequently change the company’s culture. Aimlessly discussing culture can harm an organization because culture without practice becomes rootless. When an organization fails to see the actual benefits DevOps brings, they lose the enthusiasm and confidence to transform.

Therefore, we need to change behavior first and then change culture through behavior. The most critical aspect of changing behavior is to establish an effective mechanism. As I have always emphasized, a mechanism is something that people are willing to do, and doing so brings benefits.

Remember the case of a financial company mentioned earlier. If their boss simply shouted the slogan, “We need to complete the DevOps pilot project by the end of the year,” even if the project succeeded by the end of the year, essentially nothing would change. On the other hand, they established an internal mechanism, which included setting OKR goals, incentives for achieving key metrics, forming dedicated workgroups, introducing external consultants, and a set of objective evaluation standards. All of this ensured that the team was on the right track. And what underpins these objective standards is a unified measurement platform. Ultimately, rules need to be built into tools and guided through tools to inform practice.

In this way, when a team achieves tangible improvements through DevOps, the culture of shared responsibility and continuous improvement that DevOps advocates will naturally take root.

Therefore, you can see that the culture and tools in DevOps are two sides of the same coin. We should neither blindly adhere to a tool-centric view, rushing to purchase and build tools, nor blindly discuss culture, forming an impractical atmosphere internally.

The Three Pillars of DevOps #

A systematic understanding of tools and culture can be summarized into the three pillars of DevOps: people, process, and platform. The combination of these three pillars forms the “correct posture” for implementing DevOps. Emphasizing the importance of only one dimension is clearly one-sided.

People + Process = Culture #

Under specific processes, people form a set of behavioral guidelines that subtly influence all aspects of software delivery efficiency and quality. These guidelines, when combined, form the internal culture of the organization.

A positive culture can bridge the gaps in process and platform, promote the continuous improvement of both, and allow the same processes and platforms to produce different results in the hands of different people. Just like the classic line in “The Grandmaster”: “Real masters don’t compete with their kung fu skills, but with their thoughts.” The guiding principle for the implementation and development of DevOps is the culture of DevOps itself.

For example, in the practice of Google SRE, the applications developed need to be self-operated for a certain period of time and will only be handed over to SRE for operation after reaching certain quality standards. However, to avoid the situation of “development leaves, operation takes the blame,” they have also established a “send back” process. This means that after SRE has operated the application for a period of time, if they find that the stability of the application does not meet the standards, it will be returned to the development team for them to take responsibility for maintenance. In this way, developers will proactively ensure the quality of the applications. Additionally, during this process, SRE also provides technical and platform support, thereby fostering a culture of shared responsibility and quality-driven mindset.

Similarly, some companies have mechanisms for “online security points.” Within a certain limit, teams are allowed to make mistakes without being held accountable. This motivates teams to proactively complete delivery activities without being overly cautious about making mistakes. Through changes in processes and behaviors, the team’s culture also gradually improves.

From this perspective, although it is difficult to directly change culture, we can define the expected behaviors under the desired culture and change everyone’s behavior through process improvement. This allows the culture to take root, sprout, and thrive.

Process + Platform = Tools #

Standardization of internal processes within an organization is a prerequisite for automation. Imagine if there were no set rules and every task required human intervention for analysis and decision-making. Results would inevitably be influenced by human factors, making it difficult to achieve automation.

The greatest significance of a platform is to support the standardization of internal processes within an organization. When these standardized processes are solidified within the platform, everyone can communicate according to a set of rules, significantly improving communication efficiency. Every process solidified on a platform is actually a tool that can be used to solve practical problems. Many people can’t distinguish between tools and platforms. It seems that as long as a tool is introduced or developed, it can be called a platform. Because of this, there are platforms everywhere within enterprises.

In fact, besides factors such as user base, recognition, and support from higher-ups, platforms also have three significant characteristics.

1. Attraction Effect: Platforms continuously absorb small and medium-sized tools to gradually become a collection of capabilities.
2. Economies of Scale: The cost of platforms does not increase linearly with the expansion of users and can achieve scalability.
3. Lego Effect: Platforms have basic and general sharing capabilities and can quickly build new business implementations.

In simple terms, a platform is like a stage, and tools perform on it. The platform provides a venue for publicity, attracts users, and also provides props and data analysis for the performance. The preferences of the audience may differ, but when the platform brings together various performances, it can meet the needs of most people. If the platform focuses on the performance aspect, it will be difficult to maintain the quality of the “stage” and will soon go out of business. Similarly, if those who perform are constantly thinking about building a platform, the quality of the performance itself will be difficult to improve continuously. Therefore, whether to build a platform or a tool has nothing to do with good or bad, only with choice.

Platform + People = Training Empowerment #

The platform is the carrier of standardized processes. On the one hand, it can standardize and constrain employees’ behavior. On the other hand, through platform empowerment, everyone can achieve the same results through the same operations. In this way, the handover between different domains and experts is replaced by the platform. When something no longer depends on individuals, waiting waste is greatly reduced, and the platform becomes a collection of capabilities within the organization.

However, at the same time, when we define the expected goals and provide platform tools, training people becomes crucial because only in this way can we maximize the effectiveness of the platform tools. What’s even more important is that through user feedback and usage validation, a lot of room for improvement can be discovered, further promoting the improvement of platform capabilities, and driving the overall evolution of the organization.

So you see, culture, tools, and training, as the three key focuses of DevOps development, reflect the attention to organizational processes, platforms, and people, which are interdependent and indispensable.

Finally, I would like to share an example about the first bank of the United States. When they first implemented DevOps, they used outsourcing, and even a small modification required complex change processes, taking several days. Later, they decided to adopt a " Open Source First " strategy and strictly reviewed their original commercial procurement process. In addition, they built their own platform based on open source tools, and conducted cross-domain role training internally. The delivery efficiency was greatly improved, achieving multiple online deployments per day instead of once a day.

Summary #

We have come to the end of today’s column. In this talk, I discussed the practical value of tools and culture in DevOps, as well as the potential problems and challenges. Finally, I derived the three pillars of DevOps, which are people, processes, and platforms. These three pillars are indispensable. Only through the organic integration of people, processes, and platforms, and the joint promotion in the fields of culture, tools, and personnel training and empowerment, can we achieve the true implementation of DevOps.

Reflective Question #

Finally, I would like to leave you with a thought-provoking question: Which aspects of your company’s culture do you find particularly appealing? And how do these cultural aspects contribute to the implementation of DevOps?

Feel free to share your thoughts and answers in the comments section. Let’s discuss and learn together. If you found this article helpful, please consider sharing it with your friends.