06 The Road to Transformation, Common Paths and Problems in Enterprise Implementation of DevOps #

Hello, I am Shi Xuefeng. Today, I will talk to you about the common paths and challenges for implementing DevOps in enterprises.

Due to various reasons, I have been directly or indirectly involved in the DevOps transformation process of some enterprises, and I have also talked to many DevOps leaders in these enterprises about their transformation stories. The transformation process of these enterprises was not smooth sailing. When introducing DevOps at the beginning, they also faced many common problems. For example, the business of the enterprise was too busy to have the time and energy to invest in the transformation work, or the internal systems of the enterprise had become very large after several generations of construction, to the extent that no one dared to make changes easily.

However, even with these problems, I have always believed that the path to DevOps transformation should be traceable. Many of the problems faced by enterprises are not unique, and it can even be said that many companies have gone through these steps. Therefore, at the beginning of the transformation, if you can refer to and borrow a common path, and prepare in advance for the possible problems you may encounter, the transformation process of the enterprise will be much smoother.

Two Tracks #

In fact, there is nothing particularly special about DevOps in terms of enterprise transformation. Just like the earlier Agile transformation, there are two feasible tracks for promoting a new model within an enterprise: one is bottom-up, and the other is top-down.

Bottom-up #

In this track, the introduction and practice of DevOps within the company originate from a small department or team. They may be the early advocates and practitioners of DevOps, and they start trying to use the DevOps model to solve problems within their own team as well as in the interaction process with upstream and downstream teams. Due to the small size of the team and the relative ease of mobilizing internal resources, this track is relatively easy to achieve results at a local level.

However, the core of DevOps lies in the collaboration between teams, and improving just one small team internally does not truly represent a DevOps transformation. But as mentioned earlier, if the enterprise is too large to change all at once, it does require some enlightened individuals to drive the process. If you find yourself in such a team, my advice to you is to adopt the “butterfly effect” principle, which means first establishing connections within your own team and with upstream and downstream teams that have strong dependence on your team’s business scope. On one hand, continuously expand your team’s capabilities, and on the other hand, gradually blur the boundaries of the upstream and downstream teams, gradually building a DevOps community.

Of course, if you want to maximize the effectiveness of the DevOps transformation, you must find ways to make senior management aware of the effectiveness of the local improvements and get their approval for such attempts, ultimately achieving horizontal expansion and gradually spreading it within the enterprise.

Top-down

Do you remember the financial company I mentioned in Lesson 2 of this column, which defines DevOps as a vision OKR indicator? This is a typical top-down approach, where the company’s senior management, based on their understanding of industry trends and the current state of the teams, issues task goals through administrative commands to drive the DevOps transformation. In this track, the company’s leadership has the willingness to promote the DevOps transformation and allocate resources, and each team has clear goals.

So, is everything going well in this case? Not necessarily. There is a saying within the company: as long as there is a goal, it can be achieved. Because it is difficult for company leaders to grasp every detail comprehensively, teams, in order to achieve the upper-level goals, always come up with perspectives or data to prove that the goals have been achieved. Such a DevOps transformation may actually be harmful to the company’s business and teams.

For example, once I had a conversation with the person in charge of DevOps transformation in a company. I asked, “What is your lead time?” He replied, “One week.” I thought, that’s pretty good. So I further asked, “How is this lead time calculated?” He replied, “We calculate it from the start of development to the completion of functional testing.” I thought, this seems to be a problem. So, I asked again, “What about the time from when the business raises a requirement to when it is released?” He replied, “Oh, that takes about two months.” You see, no wonder the business is constantly complaining, as it takes two months to release a requirement. But if you only look at the one-week development duration, it seems pretty good, right?

Therefore, having an objective and effective set of metrics becomes very important. I will spend two lessons discussing this in detail.

By now, you may have noticed that regardless of which track an enterprise adopts for its DevOps transformation, seeking the recognition and support of top management is a must-have. Without the support of management, the path to DevOps transformation will be full of difficulties. Because no matter the era, change has always been a game for the brave. For a mature company, the organizational structure, team culture, engineering capabilities, and collaborative spirit are the results of long-term accumulation, not something that can be established overnight.

In addition to this, transforming work also requires continuous resource investment, which must be driven by high-level management within the company. Only then can consensus be achieved and the transformation be quickly implemented. If you happen to work in a company that has such a high-level leader with a forward-looking perspective, then congratulations, you have obtained a crucial resource for the DevOps transformation on the road.

I had a leader like this in my previous company. He was always very concerned about improving the internal development efficiency. When I heard that he was going to invest a large amount of resources to accelerate the construction of DevOps capabilities, I enthusiastically painted a beautiful picture. But at the time, he said a meaningful sentence: “Once you start working on this, you will find that it is not easy.” Later, in the process of implementing DevOps, this sentence was repeatedly confirmed.

General Path #

So, as you can see, management support is just one of the factors that drives the transformation to DevOps. In the actual process, there are many techniques needed to help you avoid pitfalls. To assist you in this journey, I have summarized and extracted a general path, which I will now share with you.

Step 1: Find the Right Pilot Project

A pilot project is the business object within the organization in which DevOps practices are initially introduced and improvement efforts are implemented. It can be said that a suitable project is crucial for accumulating DevOps practice experience within the organization. In my opinion, a suitable project should have the following characteristics:

• Aligned with core business. DevOps should be driven by business value. For the core business, management attention is usually high, and various business metrics are relatively well-established. If the improvement results can be reflected through core business metrics, it will be more convincing. At the same time, resource allocation for the core business will be ensured in the long term. After all, you don’t want your DevOps transformation to be abandoned halfway due to business adjustments, right?
• Lean towards agile business. Agile business demands and changes are generally more frequent, making it easier to verify the effects of DevOps transformation. If a business primarily seeks stability and is in the maintenance phase with low demand and willingness for changes, it may not be a good choice for DevOps. For example, I once learned from a communication with a military enterprise that they only release updates twice a year. In this case, do you think it is necessary to implement DevOps?
• Priority on improvement willingness. If teams within the company have an extremely high opinion of themselves and look down upon DevOps, considering their current processes to be perfect, then putting in effort to emphasize the value of DevOps may result in only partial success. Conversely, teams with mediocre performance often have a strong desire for improvement and are more willing to cooperate with transformation efforts. At this time, the team’s energy can be focused on the work itself, without wasting time on repeated communication.

Step 2: Identify Team Pain Points

Once you have found the right team and reached a consensus, the next step is to identify the team’s pain points. The so-called pain points are the things that currently have the most impact on team efficiency and are also the things that can generate the greatest benefits after improvement.

I don’t know if you have read “The Goal,” the classic book by management guru Eliyahu M. Goldratt. The most important theory he proposed in this book is the Theory of Constraints. I will explain this theory in detail later in the content, but for now, just remember the “barrel principle,” which states that the capacity of a barrel is limited by the height of the shortest stave.

As for how to find pain points, I have explained it in detail in the previous lesson. You can consider conducting a value stream analysis activity within the pilot team, and I believe you will discover many surprising findings. If you don’t remember exactly how to do it, you can go back to Lesson 5 to review.

Step 3: Establish Early Success Rapidly

So, you have found the right team and identified a bunch of improvement items. Do you feel optimistic and ready to roll up your sleeves and get to work? Wait a moment! At this stage, remember not to spread yourself too thin or stretch the battle lines too long. This is actually the most typical trap in the early stages of DevOps transformation.

First of all, in the early stages of transformation, resource allocation is limited and cannot support a large number of parallel tasks. Secondly, because trust relationships among team members have not been fully established, those so-called best practices are prone to meet resistance. If they are blindly implemented, it is likely to cause a lot of friction, which will affect the effectiveness of the improvements. Finally, the patience of management is not as abundant as imagined. If they don’t see results for a long time, it is easy to affect the subsequent allocation of resources. So, the most crucial thing is to identify an area for improvement and define a goal. For example, if the process of environment application and preparation takes too much time, you can define a metric like optimizing the environment preparation time by 50%. This way, the team’s goal will be more clear, making it easier to break down and refine tasks, and visible results can be achieved in a few weeks.

Step 4: Quick Showcase and Continuous Improvement

After achieving phase milestones, it is important to report to management in a timely manner and conduct internal team summaries. This will not only increase the confidence of management and the team, gradually increasing resource allocation, but also help identify problems in the improvement process, foster a culture of continuous learning within the team, and motivate team members, thereby improving the team’s culture indirectly.

Of course, cases like this have great value within an organization. If the transformation can be expanded quickly, the impact will go beyond just the small team and spread to the company level, making it even more significant.

These four steps outlined above cover the general path of enterprise DevOps transformation. However, even when following this path completely, it is difficult to proceed smoothly. Under this path, some foreseeable problems exist, the most typical one being the DevOps transformation’s J-curve, which was also a key finding in the 2018 DevOps State of the Industry Report.

At the beginning of the transformation, the team needs to quickly identify major issues and provide solutions. At this stage, the overall team’s performance level is relatively low, but this can be quickly improved by introducing practices and tools, increasing automation capabilities and skills, and helping the team achieve initial success.

However, as delivery capabilities improve, problems with quality and technical debt become apparent. For example, due to a large amount of manual regression testing, the team struggles to shorten the testing cycle, resulting in delivery bottlenecks. Architectural issues lead to more integration problems due to technical debt, and tightly coupled components make small changes have widespread effects.

At this point, the team is faced with a choice: should they continue to progress or stagnate? Continuing to progress means that the team needs to allocate additional resources to strengthen the core automation abilities through research and development, such as optimizing build times, increasing test coverage, etc. This requires long-term investment and may even result in a temporary decline in delivery capabilities.

At the same time, human factors related to the organization’s inherent processes and boundaries also constrain further improvement in enterprise efficiency. Building confidence for the team to reduce review and approval processes also depends on the establishment of a robust quality assurance system. If the team postpones DevOps improvement work due to business pressures, it means that DevOps is difficult to truly take root and demonstrate its value - many DevOps projects die this way.

Now, you may wonder, who should be responsible for these issues? In other words, does an enterprise undergoing DevOps transformation need to establish a dedicated team? If so, what should the team’s composition be like?

Regarding these questions, my recommendation is that in the initial stages of transformation, it is necessary to establish a dedicated transformation working group. This team should be made up of key stakeholders from the relevant teams, DevOps experts, and external consultants, generally consisting of domain experts or senior members. They will be like the “brains” of DevOps implementation, mainly responsible for formulating DevOps transformation project plans, identifying improvement objectives, designing technical solutions, and revamping processes.

In addition to the core team, the management team and tooling team are also crucial. I have attached a diagram illustrating the composition of a transformation working group for your reference. Of course, DevOps advocates for a multi-skilled, cross-functional team, so it is also important to select team members who possess these capabilities.

Summary #

Today, I introduced to you the common trajectories of enterprise DevOps transformation, which are bottom-up and top-down approaches. Regardless of which trajectory is adopted, seeking support from the management is crucial. Next, we discussed the general path of DevOps transformation together. It is important to note that any change will not be smooth sailing, and the same goes for enterprise DevOps transformation. After experiencing early success, it is easy for us to fall into the “J-curve” trap. If we cannot overcome the difficulties, it is easy for the transformation to be abandoned halfway and go back to square one. Finally, we discussed whether a dedicated DevOps transformation team is necessary. When an enterprise is just starting to explore DevOps, such a team is necessary for quickly getting started and building team confidence.

However, as the poet Lu You wrote in his poem “Reading on a Winter Night for My Son Yiu”, “Knowledge acquired from books may seem shallow, true understanding requires putting it into practice.” Having heard many methods and paths of implementing DevOps, but still not being able to truly enjoy its immense benefits, the missing piece might be the confidence and motivation to take action first.

Thought Question #

Have you encountered any problems when implementing DevOps in your organization? How did you solve these problems? Have you made any detours?

Feel free to write down your thoughts and answers in the comments, let’s discuss and learn together. If you find this article helpful, please feel free to share it with your friends.