Guide_What Does a Scientifically Rigorous AB Testing Process Look Like? #

Hello, I am Bowei.

In the previous two lessons, we put a lot of effort into learning the theoretical prerequisites for A/B testing. This is also to help you solidify your theoretical foundation. However, unless you have a background in statistics, I would recommend that you continuously review the content of the statistics section while learning the practical aspects to combine theory with practice. If necessary, you can also extend the statistical concepts and theories I discussed in the statistics section by referring to relevant statistical textbooks to deepen your understanding.

After completing the study of statistical theory, we will now start designing and implementing A/B testing. However, before summarizing the process of A/B testing, I would like to briefly introduce the preparation work for conducting A/B testing in practice, which mainly includes two parts: data and testing platform.

On the one hand, we need to have data, including various user behaviors in our products and business, performance of marketing advertisements, etc., in order to construct metrics. Because A/B testing is an analytical method based on data, just as “no rice, no cooking,” without data, we cannot compare who is better or worse through A/B testing.

Generally speaking, as long as the company’s data infrastructure is well established and the tracking is properly implemented, basic common metrics can be satisfied.

If the metrics for the A/B test we want to conduct are relatively new or special, or if the database is not comprehensive and there is no existing data available to calculate the corresponding metrics, we can discuss with the data team to see if we can find alternative methods of calculating metrics based on existing data.

If no similar substitute metrics can be found, we need to negotiate with the data engineering team to see if this data can be constructed. This may require new tracking points or obtaining data from third parties.

On the other hand, we need a suitable testing platform to help us implement A/B testing. It can be a platform built by the company’s internal engineering team or a platform provided by a third party. For these platforms, we need to familiarize ourselves with them before conducting A/B testing so that we can set up and implement new A/B tests on the platform.

Of course, when conducting A/B testing, the database and testing platform need to be integrated through APIs or other means, so that the A/B tests set up and implemented on the testing platform can calculate the corresponding metrics through data.

The above preparations are not necessary for every A/B test but are more like the infrastructure of A/B testing, which needs to be done when conducting the first A/B test. The process described below is what we need to go through for every A/B test, and I have summarized them in one image for you to refer to.

The above is a standardized process for conducting A/B tests. You can see that A/B testing is highly practical, but it can be divided into several steps. In this course, I will focus on explaining the five most important parts: setting goals and hypotheses, determining metrics, selecting experimental units, estimating sample size, and analyzing test results.

In the entire process, except for the specific details of implementing randomization and the implementation of the tests, I will explain each step one by one. You may ask why I cannot explain all the steps at once.

In fact, I will focus on explaining the basic principles of A/B testing, the specific process in practice, and the common problems and solutions encountered in practice. These are more experiential and methodological content. Regardless of which company you are in or which industry you are in, and no matter which platform you use to implement A/B testing, these experiences and methodologies are universally applicable, and after learning them, you can apply them in practice.

As for the different randomization algorithms for randomization and the platforms used for implementing tests, these details are more about engineering implementation. Different companies and platforms may differ greatly when implementing A/B testing. For example, large companies generally develop their own internal testing platforms, while medium and small-sized companies use third-party testing platforms.

Therefore, in these initial lessons, I also hope that while you are studying, you can also connect it with your work. If you have previously conducted A/B testing in your work but feel that the process is not well-systematized, you can compare your usual A/B testing with the process covered in the basic section to see what areas need improvement. At the same time, by learning the basics, you will understand why these processes exist and what principles they are based on, allowing you to have a deeper understanding of the processes and apply them more effectively.

If you have not yet conducted A/B testing, that’s okay. I will provide in-depth explanations using actual cases. If you have the opportunity, you can try to conduct your first A/B test after completing the course!

Finally, I want to clarify one point. The prerequisite for A/B testing is data, which involves a company’s data architecture and tracking strategy. This is more about engineering and database construction issues, not the focus of our A/B testing. Therefore, in the upcoming lessons, I will assume that we are already able to track the data needed for A/B testing. As for how to track this data and the implementation details of tracking, we will not discuss them here.

Okay, now that you understand this, let’s start the journey of A/B testing!