00 Preface_Use good AB testing to master this learning #

Hello, I am Bowen. Welcome to learn A/B testing with me.

You may not be very familiar with me, so let me introduce myself first. Currently, I work in a major internet company in the United States and hold the position of a senior data scientist. Over the past 7 years, I have been working on A/B testing, machine learning modeling, and big data analysis.

In my experience with A/B testing, I have been involved in the entire process, from designing and implementing tests to analyzing test results and providing business guidance. Gradually, I have taken the lead in the field of A/B testing within my team, and have developed data products related to A/B testing. I have also collaborated with engineering teams to improve and enhance our internal A/B testing platform. Through continuous A/B testing, we have brought millions of new users to our company’s new businesses. With years of experience, I now also provide lectures on A/B testing and consulting sessions to data analysis, marketing, and product teams. I share with them the best practices and pitfalls to avoid in A/B testing.

In my many years of data analysis practice, I have come to realize that A/B testing is the most practical and effective way to drive continuous business growth.

However, I have also found that among various data analysis methods, A/B testing is the method most prone to mistakes.

The reason behind this is that A/B testing is a highly practical method, and it is often not covered in traditional academic courses. You may have learned its theoretical foundation, hypothesis testing, in your statistics class, but it is still too theoretical and you may not know how to apply it. The difficulty of A/B testing lies in the fact that if you only have the theoretical foundation without practical experience, the varied business scenarios may present numerous potential pitfalls. Only by combining both theoretical foundation and practical experience can you obtain reliable test results.

Therefore, I am eager to systematically organize and summarize the knowledge and experience I have gained from working at mature tech companies in Silicon Valley and share it with you. In this course you are about to learn, I will first help you establish a framework for conducting A/B testing. This will enable you to navigate through different business scenarios and flexibly apply the framework to achieve your goals.

However, before delving into the specific learning methods, I would like to talk to you about what problems A/B testing can actually help us solve.

Why is it necessary to learn A/B testing for sustained business growth? #

In the era of big data, every company talks about data-driven product and business rapid iteration, which is certainly not wrong. However, many people believe that being data-driven means conducting some data analysis and generating reports, without actually incorporating data into the company’s business decision-making process.

This is a serious misconception.

Years of professional experience have taught me that to determine whether a company or team is truly data-driven, one must look at whether A/B testing is included in its decision-making process.

Why is this the case? Let’s first understand the decision-making process, which is the process of product/business iteration.

As you can see, the process of product/business iteration is roughly divided into 3 steps:

1. Concrete business problems give rise to the idea of iteration. For example, after a business problem arises, the team will propose specific iteration plans.
2. The team verifies the feasibility and effectiveness of the plan.
3. After the verification is completed, the specific iteration plan is implemented.

It is clear that once the verification step is completed, iteration begins. Therefore, conducting thorough and accurate verification is crucial.

This is easy to understand as well. Just think, if an iteration idea has just emerged, and it is implemented without verification, it will be difficult to achieve the expected results, and may even have negative effects.

This is similar to a newly developed drug being directly launched into the market to treat patients without undergoing clinical trials. The risk is very high, as it may not only fail to cure the patients, but may also cause serious side effects. When we think about it this way, do you understand the importance of verification?

And A/B testing is the best solution to ensure the success of this critical step. Because it not only allows us to clearly understand whether the product/iteration plan is effective and to what extent, but also enables us to quickly abandon ideas that do not meet expectations with solid evidence when the results are unsatisfactory.

This not only greatly saves the company’s costs, but also accelerates the speed of iteration. If after spending a lot of time and resources to implement an idea, the expected results are not achieved, it will be a waste.

Therefore, only by incorporating A/B testing into the decision-making process, making business and product decisions based on trustworthy test results rather than so-called experience, can we truly achieve data-driven decision-making.

In fact, this is also a problem that all companies face: business growth is never achieved in one step, so how can we maintain sustained business growth? A/B testing is truly effective in improving business and product iteration, and can continually bring revenue and user growth.

Whether it is the “FLAG” companies in Silicon Valley or the “BAT” companies in China, they conduct thousands of online A/B tests every year, involving millions of users (in fact, most users are involved without their knowledge). Even some startup companies or traditional enterprises like Walmart and American Airlines optimize their business through small-scale A/B tests.

Taking Bing search as an example, A/B testing helps them discover dozens of methods to increase revenue every month. The revenue from each search can be increased by 10% to 25% per year. The improvements brought by these A/B tests and other efforts to improve user satisfaction are the main reasons for the profit growth of Bing search and the increase of its market share in the United States from 8% in 2009 to 23% in 2017.

Now, you may be curious about what specific business problems these companies use A/B testing to solve. You will understand when you look at the summary table I have prepared for you below.

Because of the significant role of A/B testing in product iteration, algorithm optimization, and marketing, more and more companies are starting to use A/B testing, which leads to a growing demand for talent in this field. Whether it is a technical data scientist, data analyst, marketing analyst, product manager, or growth hacker, they all need to master and apply A/B testing in their work. And from my experience as an interviewer for many years, A/B testing is also an important topic in interviews for these positions.

Having reached this point, you may already be eager to learn A/B testing. But let’s not rush it. I have found that many people are both familiar and unfamiliar with A/B testing.

When I say familiar, it’s because the basic concept of A/B testing is easy to understand—it refers to controlled variable experiments in science. When I say unfamiliar, it’s because A/B testing involves various business scenarios, different data, and multiple tedious steps in the implementation process. There are also many misconceptions.

Understanding the principles of A/B testing is simple, but using it effectively is difficult #

So why is that? Let’s look at a few real-life examples.

I often collaborate with marketing and product teams on A/B testing. They usually come up with ideas for A/B tests, such as improving the push notification effectiveness of a certain app, hoping to increase the click-through rate of the notifications by changing different factors.

Initially, many of their ideas are not suitable for A/B testing. For example, compared to the control group, they would think of changing both the title and content of the push notification, or changing both the content and timing of the push notification, and so on. This violates the principle of control variable experiments where only one factor should be different between the experimental group and the control group. When we simultaneously change multiple factors, even if we obtain significant test results, we cannot determine which factor caused the effect.

This is due to a lack of solid foundation. It is a very serious problem because without a clear understanding of the principles, it is easy to adopt incorrect methods in experimental design and analysis of experimental results.

You may ask, if I have a good grasp of the theoretical foundation, does that mean I will have no problem conducting A/B tests?

Of course not. A/B testing is a highly practical method. You may have learned its theoretical foundation in a statistics class, but how do you apply it in actual business scenarios? That is the difficulty of learning A/B testing.

Theoretical knowledge is static, but the application scenarios and related data of A/B testing are constantly changing. When implementing A/B tests, you will encounter various data issues or engineering bugs. If you accidentally overlook even a small detail, various traps await you, and the accuracy of the experimental results will be compromised, rendering all previous efforts futile.

Let me share another example with you.

There is a platform dedicated to testing app push notifications. One type of process involves comparing whether sending a push notification has an effect. The control group does not receive a push notification, while the experimental group does receive a push notification.

Before sending it out officially, the platform performs a filtering step to exclude users who are not eligible for receiving push notifications, such as underage users or users with outdated devices that do not support push notifications, and so on. However, since only the experimental group receives the push notification, the platform only implements the filtering mechanism for the experimental group:

However, upon careful consideration, this process introduces two differences between the experimental group and the control group: the presence or absence of the push notification and the presence or absence of filtering. The first difference is intentional in the experimental design, but the second difference is purely a result of the process, introducing bias that can lead to inaccurate experimental results.

The correct process is shown in the following diagram. Even if the control group does not ultimately receive the push notification, they should still undergo the same filtering process as the experimental group to ensure the accuracy of the experiment:

As you can see, such a small problem can cause the entire A/B test to fail.

How is this course designed? #

So, in order to help you quickly and solidly grasp the craft of A/B testing, I have combined my industry experience and organized a best learning path for A/B testing for you, covering statistical principles, basic processes, and advanced practical skills.

The first module is “Statistics”.

To do A/B testing well, learning statistical principles is definitely a must. Statistical knowledge can be complex, but you don’t need to master everything for A/B testing. Therefore, I have selected statistical theories that are closely related to A/B testing, mainly explaining the theoretical foundation of A/B testing - hypothesis testing, as well as the statistical properties of A/B test metrics, so that you can learn theory knowledge in a targeted manner and truly establish a solid theoretical foundation for A/B testing.

Even if you don’t have a strong background in statistics, you can quickly grasp the statistical foundation of A/B testing in this module without worrying.

The second module is “Basics”.

In this module, I have organized several key steps for conducting A/B testing, including setting goals and hypotheses, determining metrics, selecting experimental units, calculating sample sizes, and analyzing test results. While explaining the process, I will also explain the underlying principles, helping you apply them in practical scenarios.

The third module is “Advanced”.

To take your A/B testing skills to the next level, mastering the key process steps is not enough. You also need to be able to identify potential pitfalls in actual business scenarios and have corresponding solutions.

In addition, you should know that A/B testing is not a panacea, so I will specifically dedicate one lesson to explain the scope of A/B testing and alternative methods.

If you are preparing for A/B testing-related interviews, don’t worry either. I will spend two lessons helping you grasp common interview topics and how to respond to them.

Lastly, I will guide you through practical exercises to create a practical sample size calculator to solve the problems of inconsistent and limited utility in online tools.

A/B testing is not difficult because it does not require you to master highly advanced computer algorithms or advanced mathematics. Understanding basic statistical knowledge is sufficient. However, doing A/B testing well is certainly challenging. Its difficulty lies in the fact that if you do not follow a scientific process, various situations and problems may occur during practical implementation.

Therefore, I also hope that you can practice while learning this course, learn from experience, and gradually make A/B testing a core competency in your work.

Lastly, today is the beginning. You can write in the comment section about your expectations for this course or your study plan. Let us witness each other’s growth together!