26 Platform Product Development, How to Achieve a Thousand-Scale Product in Three Months #
Hello, I am Shi Xuefeng.
Although we have discussed a lot about the idea of platform construction before, many people may not have had the opportunity to experience the complete process of a platform from concept to development, and then to promotion and implementation.
How long does it take to develop a DevOps product that can be used by a thousand people? You might say it takes six months or even longer, and that’s what I used to think.
However, in 2018, when we started constructing the pipeline platform, our boss waved his hand and demanded to see results within three months. I was almost shocked.
Because we were really starting from scratch: there wasn’t a single prototype, not a single line of code, and the temporary team was scattered in Beijing and Shanghai. The team members had never really communicated before. Can this work?
Today, I want to share with you this true story. Let’s review this “rapid march” process together and see what we did right, what we did wrong, what practical knowledge we can take away, and what pitfalls you must strive to avoid.
In fact, as a non-professional DevOps product manager, you will eventually face this kind of challenge, but you have to believe that as long as you start doing it, nothing is impossible.
Project Kickoff #
Going back in time a year ago, what was the situation with the team I was a part of? The team composition was as follows: two backend developers in Beijing, one and a half frontend developer in Shanghai, along with one infrastructure engineer and one assembly line development engineer, plus half of a versatile product manager, which is me, making a total of six people.
The project conception started in mid-November, and the kickoff meeting took place at the beginning of December. At that time, apart from me, no one else had a clear idea of what exactly we were going to do. What should we do in such a situation?
Those who have played games should understand how important it is to start well. So, for this kickoff meeting, I did a lot of preparation work in advance, which included a version 0.2 prototype of the product. It’s not really a prototype but rather a draft, absolutely crude.
The kickoff meeting of the project was the first meeting between the project team members and the future product, and leaving a positive impression was very important. Therefore, from the first moment, I was fully engaged.
Firstly, I sent out a warm invitation to the meeting. In the invitation, I carefully explained why we should do this, why now, and why it’s not feasible without doing it.
During the official meeting, I reiterated the importance and urgency of the project and demonstrated the first version of the system prototype (yes, the same crude prototype I mentioned earlier).
Even so, the three-month deadline made everyone very anxious. To alleviate the tension and demonstrate the feasibility of the project, I did two things:
- Set up a system demo with a few simple pages.
- Since we used the core technology of another open-source product, I gave a simple demonstration of this technology.
Although I myself was quite worried about this plan, I still wanted to tell everyone that this was not an impossible task, and I tried my best to help everyone build confidence.
During the project kickoff meeting, the team reached two critical conclusions: one was the selection of the system solution, and the other was the establishment of a collaboration mechanism.
Firstly, due to the tight schedule and heavy workload, we decided to use a frontend-backend separation development model that is more conducive to collaboration. Later on, it proved to be a very wise choice. This not only greatly improved development efficiency but also significantly reduced the cost of future migration to the mobile end. When developing mobile end products, most of the backend interfaces can be directly used.
In terms of technology framework, since everyone reached a consensus on the frontend-backend separation model, we used Python+Django+Vue. You may ask, why not use the Java-based Spring series? Because I believe that for internal systems, these typical frameworks are more than sufficient, and the key is to choose the one you are familiar with and easy to start with. From this perspective, Python obviously has a unique advantage. Even if you had only written some scripts before, it wouldn’t be difficult to start with Python.
In terms of project collaboration, I will mention it specifically later on. Since team members are dispersed in Beijing and Shanghai and are not familiar with or trust each other, establishing a fixed communication mechanism is very important.
At least in the early stages of the project, we had two conference calls every week:
- One was for the entire team. On the one hand, it synchronized the latest progress of the project, and on the other hand, it gave everyone a sense of urgency, making everyone feel like “everyone else is executing the plan, so I can’t fall behind”.
- The other one was for the backbone members across different regions. This was mainly to build closer connections and to confirm the progress of some core issues. Not involving the entire team was also to avoid wasting too much time of the project members.
Lastly, the project still had some technical risks, so we needed to start research in advance. The main risk of our project was in frontend interaction.
This was a scenario that had never been implemented before, with a lot of user interface layout operations involved. Therefore, we designated a colleague and instructed them to focus solely on the technical challenges.
It turned out that colleagues who could handle tough situations were all very reliable and capable, regardless of their age, even fresh out of school.
To sum up, let me highlight a few key things to focus on in the project kickoff stage:
- Clearly define the project goals and build the team’s confidence.
- Communicate the development model and select the technology framework based on rapid development and easy learning.
- Establish communication channels and maintain high-frequency contact.
- Identify the technical risks of the project and start specialized research in advance.
Development Strategy #
Every aspect of human society requires more and more people to work towards the same goal, and software development is no exception. So, how do we do it?
First, we containerize the development environment.
For developers who are new to a technology stack, setting up a complete and runnable environment locally is always a challenge. Even if you follow the documentation step by step, there will always be something missing. In addition, if any of the various middleware dependencies of the project have even a slight version inconsistency, it can take a long time to find the problem once it arises.
In that case, why not start with a standardized environment? This is where container technology comes in handy. The main backend developers take on the task of setting up and debugging the environment locally, and then containerize the environment configuration. This way, when new team members join the project, they can have a fully functional local development environment ready in just a few minutes. Even if there is a need to upgrade environment components in the future, such as the Django framework version, it is very simple - just push an image and restart the local environment.
Next is choosing the branching strategy. Although DevOps advocates for trunk-based development, we still chose the “three-branch” strategy because we have set up three environments.
The test environment corresponds to the dev branch as the main development line. All new features are developed in feature branches and tested locally. Once the self-test is successful, an MR (Merge Request) is created to merge the changes into the dev branch and deploy them to the test environment for acceptance testing, which is usually done by the stakeholders who raised the requirements.
Then, twice a week, we merge from dev to the master branch. The master branch corresponds to the pre-production environment, ensuring consistency with the production environment. Data is also regularly synchronized. Only after the final acceptance testing in the pre-production environment, are we ready to deploy to the production environment. This is done by merging the master branch into the release branch. The diagram below illustrates this branching strategy:
Why do we adopt the “three-branch” strategy with three environments?
The main reason is that our team is in the early stages of forming, and there are often inconsistencies in frontend and backend configurations. Moreover, our project involves not only frontend and backend development, but also core atomic business development and infrastructure maintenance. Any inconsistency in the pace of any party can lead to problems.
Furthermore, it is a common problem in internal platform development that there is no dedicated testing. This is understandable; there are only a few people in our “startup team”, where can we get testing resources? So, basically, we rely on development and product teams to ensure quality.
However, testing is a professional trade after all, and this inevitably leads to various issues. Combined with the fact that product requirements themselves are not always clear and often subject to change, having an additional environment provides an extra layer of security.
But it must be acknowledged that this strategy is not the optimal solution; it is merely a feasible solution based on the current situation. Once the team has settled in and matured, we can simplify it to one branch and one environment. However, the premise is that only with rapid iteration and quick deployment can we fully leverage the advantages of the two environments.
Use what you build to build what you use.
This is our consistent philosophy. Since it is a DevOps platform, the team must also have a DevOps mindset. Therefore, as a full-function team, we excel in developing and operating our own projects.
At the same time, we do not use a standardized deployment process provided by the company, but rather built our own standardized deployment process and solidified it in our tools. Every member of the team is able to complete the deployment action.
This way, we no longer rely on specific individuals, ensuring maximum flexibility. Even in the case of large promotions or network restrictions, normal development activities will not be blocked.
Development Collaboration Process #
Just achieving the above points is not enough for the team to operate efficiently because the most important development collaboration process is missing.
As the project leader, I have spent a lot of effort optimizing the development collaboration process and establishing development collaboration standards. After everything runs smoothly, I found that the initial investment was worth it.
At the tool level, we use Jira. For small teams, Jira’s functionality is excellent enough to meet the needs of most scenarios. However, the drawback of Jira is that it has a slightly higher learning curve for both usage and configuration. Therefore, there needs to be a team member who is familiar with Jira in order to make use of this method effectively.
In Jira, we adopt the Lean Kanban plus Iteration approach, with a basic iteration cycle of two weeks to maintain the rhythm of development delivery. This development workflow is well suited to our branch strategy and multi-environment deployment.
Requirements are unified into Backlog management. When an iteration begins, the requirements are moved to the To-Do state, and developers choose tasks to start development, entering the In Progress state. When development is complete, it also means that the functionality has been deployed to the testing environment. At this point, the functionality can wait for acceptance testing. Only after acceptance testing is passed, will it be released to the pre-production environment. After a second round of acceptance testing, it will finally be released to the users.
The development process is not complicated. You can take a look at the two versions of the process diagrams below.
Image version:
Text version:
After defining the development workflow, the next step is to clarify the principles and standards. For a newly formed team, rules are the best means to eliminate disagreements and misunderstandings, so these rules must be clear and easy to understand. For example, within our team, we have a “3-2-1” principle:
3: Create three elements of a task
- Detailed problem description and explanation
- Clear acceptance criteria
- Specific assignee and iteration schedule
2: Handle two elements of a task
- During development, code changes must be associated with a Jira task number
- After development is completed, Jira comments must be added to explain the changes and scope of impact
1: Resolve one element of a task
- The reporter of the issue is responsible for verifying and closing the task
Of course, team rules are much more than just these few points. You need to create your own internal team rules and emphasize these rules repeatedly to help everyone develop good habits. In doing so, you will find that the efficiency of development and the self-organization of the team will gradually become a reality.
In addition, do not overestimate people’s proactiveness and expect everyone to follow the rules conscientiously. Therefore, regular and timely reminders are very necessary. For example, add daily scheduled email notifications to inform people about which requirements need acceptance and which can be released. Try to make sure that everyone knows where to get the latest information.
Furthermore, during weekly meetings, emphasize the compliance with rules, and even communicate on a daily basis as needed. Only by maintaining short and frequent communication can the desired effect be achieved.
Product Operation Strategy #
The concept of “good wine needs no bush” is somewhat outdated when it comes to product operation strategy. In order for a product to succeed, the team not only needs to do well, but also needs to be good at operation and promotion, which is often a weak spot for technical teams.
Development teams mostly only know how to implement functions, but are not aware of how to operate a product. Because of this, teams often have a hard time getting real feedback from users, and sometimes even develop brilliant features that users are not aware of. Product development becomes a self-indulgence, which certainly does not align with the original intention of product design.
Considering these, we have adopted some measures in platform operation. I would like to remind you that many things are not as difficult as they seem, but rather depend on whether you have the desire and the persistence to do them.
For example, you can establish an internal user communication group and choose some active seed users to try out the product in the early stages. Those users who show particular interest, willingness to try new things, and continuously provide suggestions are super users. These users are the “sparks” in each team and at the early stages of the project, you must identify these users.
In addition, release notes should be published for each release and notified to all staff through email and internal communication groups. This serves two purposes: it can promote new features and, more importantly, refresh the sense of existence. You want users to know that this product is in a process of rapid iteration, and that there is always something new each time that can attract them or prompt them to actively raise their own questions.
In user groups, it is important to respond promptly to user questions. You can establish an OnCall mechanism within the team, where team members take turns each week to address frontline user issues. This ensures timely problem resolution and allows developers who are distant from users to truly hear their voices. This way, during requirement planning meetings and iteration review meetings, developers will be more actively engaged in discussions.
The aforementioned methods are relatively conventional. In our product operation, there are two methods that have been particularly effective, and I also recommend them to you.
Platform operation is similar to advertising: the more you advertise in places with high traffic and attention, the better the effect. Every company usually has similar homepages, such as internal technology homepages, technology forums, and daily office OA systems, which actually have channels and entrances for promotion. What you need to do is find these entrances and contact the people responsible for these channels. Our product once achieved dominance on popular sites, and the promotional effect was very significant, resulting in a sharp increase in user numbers.
Another method may seem a bit clever, but it is very suitable for technical teams: promote the product through technical sharing channels.
I believe that every team has regular technical sharing channels, such as technical public accounts. You can extract the core technical points and design ideas of the platform, prepare a sharing topic, and share it on the widest possible range of internal technical sharing channels.
Often, simply promoting a product does not gain much enthusiasm from people. However, if you talk about new technologies and combine them with the realization of products, it will be much more attractive to technical personnel. Therefore, taking a different approach to operation is also a good way to increase product visibility. I have summarized the product operation channels and methods I previously summarized into a mind map, which I will also share with you.
Team Culture Building #
Lastly, I’d like to briefly discuss team culture building with you. After all, no matter what tools, processes, or goals we have, they ultimately depend on people to accomplish them. Neglecting to focus on people is equivalent to putting the cart before the horse, and it’s not something a mature team manager should do. I want to share two points with you.
1. Let professionals do their job
Never underestimate professionalism. Every internal function in an organization requires the support of professional skills, and these professional skills are the result of quantitative changes leading to qualitative changes.
Let me give you a simple example. Do you remember the prototype draft I mentioned earlier for version 0.2? In reality, by version 0.3, as the frontend engineer put it, “the prototype looks better than the actual system.” Why is that? Did I, a “half-baked” product manager, suddenly become enlightened?
Clearly not. The answer is actually quite simple: I sought out a professional product manager as an external resource and asked him to help me improve the prototype for two days. For professionals, these tasks are a breeze.
By finding professionals to handle these tasks, not only can you quickly bridge the gap, but you can also acquire a lot of ready-made experience for later use. This is something that cannot be achieved through solitary hard work.
This principle applies not only to the product side but also to the technical field. We must be willing to admit our ignorance and seek help from others; otherwise, in the end, we will be the ones paying the cost of lost time and opportunities, which outweigh the benefits.
2. Focus on the big picture and selectively ignore details
During the collaboration process, conflicts may arise within the team over certain details. If team members are allowed to endlessly argue about these details, it will eventually affect the trust among the team. In such cases, it is necessary to guide the team to focus on the bigger picture and temporarily defer detailed discussions in order to ensure the efficiency of team collaboration.
For example, there may be a debate about whether to handle a certain business logic on the frontend or backend, with no significant difference in the end result. However, frontend colleagues may insist that logic processing should be handled by the backend to reduce coupling between the frontend and business. This argument is not wrong. On the other hand, backend colleagues may have their own ideas, such as whether or not to cooperate with returning the specific status codes requested by the frontend instead of throwing out the original HTTP status codes.
Such issues do not necessarily have a right or wrong answer, but if we get caught up in them, they cannot be resolved in just a few words.
At such times, someone needs to make a decision, choose a more conventional approach, and reserve space for further discussion. Additionally, in order to promote cooperation among multiple locations, it may be necessary for our side to make some sacrifices in order to facilitate smooth progress. This way, you will find that some irreconcilable issues become insignificant in the process of continuous project success and team integration.
Summary #
In this lecture, I have shared my summary of how to develop a product based on my experience in past projects.
In fact, as I mentioned in “The Skills Required for DevOps Engineers,” soft skills such as communication, collaboration, empathy, and continuous improvement play a significant role in promoting rapid iteration and development of a product. As a non-professional product manager, I am gradually accumulating my own product wisdom, and I hope to discuss more about it in the future.
You may still be wondering if we achieved the goal of reaching a thousand users. Well, I want to tell you that sometimes real life can be even more exciting than a story.
On the second-to-last day of our goal, the platform only had 997 users. I complained about this number with my colleagues, and they suggested bringing in a few more users. However, I said, “Let’s leave it as it is.” Can you guess what happened? By the end of that Friday, when I checked the platform again, there were exactly 1000 registered users. My first reaction at that moment was to believe that when we wholeheartedly dedicate ourselves and our passion to the development process of a product, wonderful things will naturally happen.
Reflection Questions #
What part of this lecture left the deepest impression on you? Do you have any other insights that can contribute to the rapid development and implementation of products?
Please write down your thoughts and answers in the comments section. Let’s discuss and learn together. If you found this article helpful, feel free to share it with your friends.