Conclusion_Practice is the Only Criterion to Test Truth #

Hello, I’m Bowen.

Over the past few years, I have been involved with A/B testing, and the more time I spend studying it, the deeper I understand its meaning. Therefore, when designing the course syllabus, I chose “Practice is the Sole Criterion for Testing Truth” as the title for this concluding lecture without hesitation. This statement is simple but represents my true experience of working with A/B testing for so many years.

Firstly, I want to talk to you about why I repeatedly emphasize the importance of practice in the course.

A/B testing itself is a tool that leans towards experience and methodology. Even after mastering the statistical principles and standardized procedures that I discuss in the course, it does not guarantee that you will feel at ease and proficient in practice. After all, to truly understand the experiences and methods you have heard, you need to repeatedly refine them in real business scenarios.

Even some of the common misconceptions and hidden pitfalls that I have mentioned, as well as other pitfalls you may encounter, are likely to be encountered in practice due to the vast differences in business environments. However, there is no need to be afraid because it is precisely through the struggle with these “detours” that you can grasp the techniques and tricks of A/B testing and improve your business skills.

A/B testing brings not only continuous improvement to companies and teams but also cultivates an experimental mindset. When a member of a team proposes an idea, is it just a whim or a truly reliable one? Can it be effectively implemented? We can completely test this idea through A/B testing in practical scenarios and finally reach convincing conclusions. As the saying goes, “Bold conjecture, careful verification, continuous adjustment, and rapid iteration.”

“Practice” is not an easy task. It also requires initiative, patience, and courage when actually doing it.

You can actively participate in projects related to A/B testing. Whether you are personally conducting the tests or observing the entire process, it is all part of the learning and accumulation. Take the initiative to propose business requirements and try them out. As long as you practice, you will definitely gain something.

Be patient with “failure.” I put quotation marks around “failure” because you might think that if the changes in A/B testing are not implemented in the product or business, it counts as a “failure.” However, let me tell you that according to the practical data, more than half of the changes in A/B testing are ultimately not implemented. While conducting tests, we certainly hope for significant results, but in reality, the probability of not achieving significance is quite high. This is the gap between expectations and reality. So, does this mean it’s a failure?

In the field of A/B testing, it is not the case. Every “failed” test is valuable experience for us. Through these tests, you may discover and improve problems in testing settings, engineering implementation, and other aspects. Even if the test results are not significant, it doesn’t matter because it can help us eliminate different ideas, reduce potential losses to the business, and quickly iterate to the next idea.

Dare to put forward your own ideas. I can’t remember how many times I have disagreed with colleagues and leaders, and A/B testing has become our magic weapon for solving problems. Over time, it has also helped us create an atmosphere of experimentation within the team. More and more people dare to propose different ideas and opinions instead of being subject to the decisions of management.

You see, once the plain wisdom contained in “Practice is the Sole Criterion for Testing Truth” is combined with our current life and work, it becomes more vivid and rich in understanding. This is also why I have a passion for history because the wisdom of our predecessors always remains relevant.

However, at the end of today’s course, I want to share more of my own experience in learning A/B testing, and discuss my study insights, hoping to bring you inspiration and encouragement.

The first insight is to build your own knowledge framework, which can make your learning more efficient.

Let’s use myself as an example. In fact, I am not a data professional with a formal education background, so in order to establish a foothold in this unfamiliar field and specialize in my craft, I have to put in more effort. Let me give you a small example. In order to understand difficult statistical concepts such as the Central Limit Theorem, P-value, and Power, I have studied various versions of statistics textbooks more than 20 times. To comprehensively master knowledge in the field of data science, I have spent my spare time studying nearly 10 online courses related to data science. This systematic learning has provided me with a solid foundation for practical application. When encountering other problems in practice, I know how to search for information and find solutions, instead of feeling clueless.

So, this is also the original intention for me to create this course. Based on my years of experience, I have systematized the experience and methods in the field of A/B testing, helping people who want to learn this field build a knowledge framework, allowing you to achieve nonlinear breakthroughs in a short period of time.

The second insight I want to share is that learning should have a purpose, and the knowledge learned should be applied to practice in a timely manner. During school, most of our learning is for the sake of learning. However, after getting a job, the purposeful learning can achieve multiple benefits.

As I mentioned earlier, I have studied statistics textbooks more than 20 times. This is just the relatively systematic number of times. If we count the actual times I reviewed the content, it would be far more than a hundred times. Because I have an average memory, I tend to forget purely knowledge-based things once I don’t use them frequently. So, for me, statistics textbooks are like the “Modern Chinese Dictionary” I used in elementary school when studying Chinese. I look things up when I need them, and it becomes a muscle memory.

Therefore, if you complete this course and feel that you haven’t fully digested some of the content, it’s okay. I hope you can treat it as a reference book for your work in A/B testing. Whenever you encounter difficult problems, you can come back to review and learn. If you have questions, feel free to leave a message, and I will reply to your questions from time to time.

I also want to share an insight from my experience of writing columns: “Textual output is an important criterion for evaluating the quality of input.”

How do we test the knowledge we have enriched and perfected through practice? Textual output is a good way. This has also been an important lesson I learned from maintaining a consistent output in my columns over the past few months.

Over the years, I have often worked on projects or given lectures with students. However, writing is different from speaking. When you have a clear understanding of a problem in your mind, you can explain it to others, but when it comes to putting it into writing, you have to consider every detail: whether the entire text is logical, whether the wording is accurate enough, whether the examples are appropriate, and so on. This is a great exercise for textual expression and professional logic.

Writing columns is different from my usual article writing as well. In writing columns, I need to remove academic language and refine the expression, striving to explain a problem in the simplest and most straightforward language, while also considering the readers’ reading habits, among other things. So, what I constantly refine is not only the textual content but also the careful consideration of the problem. Of course, this also tests my energy and willpower.

I have had an intimate relationship with A/B testing for over 7 years. For me, it is not just a method of growth in my work. After deeply understanding its subtleties, the “experimental mindset” it represents has become an important philosophy and principle in my life.

When encountering confusion and uncertainty in life, or when facing anxiety in the face of the unknown, I will not be timid or shrink back, but bravely try. Considering the possible results, I’m willing to make mistakes. Because without personally experiencing it, I may never know whether something is good or bad for me.

Just like in the past year 2020, which was destined to be a special year, the sudden outbreak of the pandemic disrupted the rhythm of many people’s learning, life, and work. In the swing between survival and challenges, stability and uncertainty, it’s better to put the thoughts and ideas in our minds into practice, thus breaking through the barriers in our hearts.

This is also the last thing I want to inspire you with: “Only by keeping moving forward can we encounter more surprises on the road of life.”

Finally, I have prepared a survey questionnaire for you. The questionnaire is short, and I hope you can take two minutes to fill it out. I am very much looking forward to hearing your feedback and thoughts on this course.