28 Big Data and Ai How to Test Efficiently

28 Big Data and AI How to Test Efficiently #

Testing, as a crucial part of continuous delivery, aims to discover quality issues in the delivery process. With the accelerated speed of Internet iteration, many products are released every two weeks or even every week, leaving less time for testing.

So how can we ensure the quality of products in such a short time? How can we test efficiently? Our development mode is constantly changing, and the positioning of testing is also undergoing changes. What will the future of testing look like?

The Evolution Process of Testing #

“Leave professional matters to professional personnel.” The division of labor in various fields of society has become increasingly refined, with design, development, product, and testing teams working together to complete a product. However, with the development of technology, this division of labor is not static. Based on the organizational adjustments made by major company departments in recent years, the roles of testing and development have been continuously merging.

What changes have occurred in the responsibilities of testing in the development of the Internet? What has been the evolution process for mobile testing?

1. The Golden Age of Testing

At the beginning of the mobile Internet era, it was still in the traditional software development stage. The product requirements were handed over to the development team, and after implementation, the product was handed over to the testing team, who then delivered the final product to the users. We can call this stage the “Golden Age of Testing”.

As a part of the development process, testing acted as a barrier for delivering a quality product to users. In this stage, there were two important assessment criteria for testing: the number of bugs discovered in each version and the number of bugs missed in production.

In the “Golden Age of Testing”, many bugs may require multiple reworkings, making it difficult to speed up the product delivery process. The main problems in this stage are:

  • Personnel conflicts: The testing team’s KPI is to discover as many bugs from the development team as possible, which can easily lead to conflicts. Especially when incidents occur in production, developers blame the testing team for not being effective, while the testing team blames the developers for their incompetence.

  • Low efficiency: Although basic automation tools like Monkey were introduced, most testing methods still relied on manual execution. Despite continuously increasing the ratio of testing to development personnel, there were still many problems throughout the process, making it difficult to shorten the product delivery cycle.

2. The Efficiency Era of Testing

With the intensification of competition in the Internet industry, delivery speed has become a core competence of products. Both domestic and foreign large companies have adapted to this trend by transforming the responsibilities of their testing teams into efficiency services.

Testing is not only responsible for delivering quality but also needs to consider the quality and efficiency of the product. As I mentioned before, the “211” efficiency goals, which include a 2-week delivery cycle, a 1-week development cycle, and a 1-hour release duration, require testing personnel to directly take responsibility for this goal and consider how to quickly and efficiently deliver products of high quality.

In the story of the three brothers treating illnesses, the elder brother treats the illness before any symptoms appear. Since no one knows he can remove the cause of the illness in advance, his name cannot spread. The middle brother treats the illness when it first occurs. People generally believe he can only treat minor illnesses, so his reputation is restricted to the local area. On the other hand, the doctor Bian Que treats serious illnesses. He uses needles to drain blood from the meridians and applies medicine to the skin. People believe he has the most advanced medical skills, and his reputation spreads throughout the world.

以上是扁鹊讲述的三兄弟治病的故事，故事中谈到长兄治病是在疾病尚未发作之前就治疗，由于一般人不知道他事先就能铲除疾病的根源，所以他的名气无法传出去。中兄治病是在疾病刚开始时治疗，人们认为他只能治疗轻微的小病，所以他的名气仅限于乡里。而扁鹊治病是在疾病严重时，通过经脉穿针放血，在皮肤上涂药，人们都以为他的医术最高明，因此名气扩散到了全国各地。 From this story, Bian Que believes that his eldest brother has the highest medical skills and can achieve “treating illnesses before they occur”. Regarding the topic of testing that we are discussing today, many companies have started proposing “shifting testing to the left”. This means that they want testing to be involved in the delivery process at an earlier stage, not just to find problems, but to consider how to prevent problems from occurring.

Testing needs to be deeply involved in every process from product design to release, and it needs to " understand technology better than the product and understand the business better than development “. In order to adapt to this change, major companies have started breaking up large testing teams in their organizational structure and integrating them into various business development teams. The purpose of the integration of production, research, and testing is to unify the goals and directions of the teams, so that everyone can work together to improve product efficiency. This way, all members of the team become a loving family and eliminate the opposition between production, research, and testing.

However, as mentioned in the article “How to Measure Development Efficiency”, “in the early stages of product iteration, the team focuses on design and coding, introducing defects, but does not integrate and verify immediately. Defects are hidden in the system until the later stages of the project when the defects accumulate and burst out”.

Although continuous delivery can to some extent reduce the submission peaks, it still cannot avoid frequent “rushed” submissions. In the era of testing efficiency, how to improve testing efficiency is still an urgent problem to be solved.

At this stage, my answer is the tooling and platformization of continuous integration. From requirement initiation to branch management, code review, code inspection, and test release, the testing team is responsible for controlling various tools or platforms.

Since the entire continuous delivery process involves platform tools at various stages, I will only focus on two platforms related to testing.

  • Testing platform. Competitive testing, weak network testing, boot testing, UI testing, etc., the entire testing process has introduced a large number of automation tools. Major companies have also improved or created many useful tools. For example, Tencent’s New Monkey tool can greatly enhance the intelligence and coverage of monkey testing. In addition to well-known automation frameworks such as Espresso, UIAutomator, Appium, and Robotium, open-source testing frameworks like Alibaba’s Macaca and Facebook’s Screenshot Tests for Android are worth learning. Of course, some companies have packaged their testing platforms as services available to the public, such as Tencent’s WeTest and Huawei’s Developer Services.

Below is a simple comparison of these commonly used tools for your reference.

  • Experience platform. In the era of testing efficiency, automation cannot completely replace manual testing. At different stages of product delivery, we need to increase the involvement of team members both inside and outside the team. Whether it’s daily daily builds, centralized experiences during the final version stage, internal employee experiences during testing, or external crowd testing platforms during the gray period, the goal is to involve as many team members as possible in the product experience and enhance their recognition of the product. Of course, major companies also have their own experience platforms, such as Tencent’s RDM and Ant Group’s Partner APP.

The era of testing efficiency is the current stage that most companies are in. As far as I know, many engineering efficiency teams in companies have evolved from testing teams. With the transformation of the role of the testing team and the improvement of technical depth, a large number of experienced testing developers will emerge, and there will be more excellent testers moving towards development or product positions.

However, I have noticed some differences between domestic and international testing practices. For example, foreign countries highly advocate for test cases written by developers, but it is very difficult to implement this in China. This is mainly because business iteration is faster in China, and developers don’t even have time to complete their development tasks, let alone write test cases, which may take more time than developing requirements.

The Testing Era in the Age of Intelligence #

“Everyone can be a tester.” Although we have done a great job in stability, compatibility, and performance testing, we have not yet reached the level of manual testing in some automated testing scenarios, especially in UI testing.

Take UI testing as an example. Due to the increasingly shorter iteration cycles and frequent UI changes, both developers and business testers are not very enthusiastic about writing test scripts and cases. Because the cost of writing and maintaining test scripts is relatively high and their reusability is low, UI testing is often time-consuming and laborious, and the efficiency is often not as good as manual testing.

So, how do we solve these problems and move testing from the era of efficiency to the next stage in the age of intelligence?

1. The Application of AI in Testing

AI technology has already demonstrated its great capabilities in human-machine battles, such as in the games of Go and StarCraft. What sparks can it generate in the field of testing?

Let’s first look at the challenges we encounter in UI automation testing:

  • Coverage. How many scenarios can be covered by automated testing? How to handle various interference, such as login and network exceptions.

  • Efficiency. Can we improve the efficiency of writing test cases or reduce the cost of writing them, so that everyone can write test cases?

  • Accuracy. How to improve the stability of the entire automated testing process? Do we still need a large amount of manual intervention and analysis for the testing process and final results?

NetEase’s open-source tool Airtest and iQiyi’s AIon have attempted to use AI technology to solve the problem of test case writing efficiency and threshold.

These tools mainly utilize image recognition and OCR technology. Taking iQiyi’s AIon as an example, its entire process includes:

  • Image Processing. Firstly, it recognizes the scene, such as whether there is a pop-up dialogue or a login page on the current interface, and then divides the screenshot images. The difficulty lies in the OCR recognition of text and the classification of layouts. For example, how to classify different segmented images and know the corresponding content of each image.

  • Result Judgment. How to determine whether the result of this UI test conforms to expectations and to what extent the similarity is achieved. Because UI testing may encounter many different situations, we need to improve the accuracy of testing and reduce manual intervention as much as possible.

In Alon’s reference article, there is also mentioned an application scenario called UI2Code, which is to generate corresponding code by using image recognition on an application screenshot or a UI design. It is actually the desired outcome that Pixel to App hopes to achieve, and I believe it is also a dream for many developers. It would allow us to completely free ourselves from UI development and generate the final UI code directly from design drafts.

Airtest and Alon are focused on improving the efficiency of UI automation testing, but they still require manual test case writing. As for stability testing, although I mentioned Tencent’s New Monkey earlier, it still has many flaws:

  • Coverage of scenarios: Although it is an improved version of Monkey, it still cannot cover all user scenarios, and many of its execution processes differ from those of real users.

  • Lack of intelligence: You can think of it as non-human. Monkey’s operations are completely different from those of humans.

So, are there any more intelligent solutions? Facebook’s Sapienz attempts to use our developed applications just like real users, by collecting the operation paths of real users to train testing behavior. And after a crash is found during testing, Sapienz automatically associates and locates the code, improving the efficiency of problem solving.

Although AI testing cannot fully replace manual testing at present, as technology matures, I believe its transformation in the testing field will be revolutionary. Of course, there is still a lot of work to be done to achieve this goal, which also means that there are many opportunities for technological innovation.

2. Application of Big Data in Testing

Nowadays, more and more businesses are operating in a data-driven manner, and the objects of testing are changing from simple code to data and algorithms. With businesses becoming increasingly complex and the amount of data growing larger, how can we timely discover quality and business problems in our products?

Some domestic companies have proposed a “real-time quality” system based on big data, hoping to achieve real-time data acquisition to complete business data validation and quality risk perception.

The data here mainly includes two aspects: quality and business.

  • Quality: Quality issues such as crashes, startup speed, lag, and network errors. We hope to achieve real-time monitoring at the minute level and support more granular dimension analysis. For example, we can view network issues based on dimensions such as country, city, and version.

  • Business: For the core business of the product, we need to implement data collection, analysis, and validation to build the capability for data monitoring and tracking.

For the “real-time quality” testing system based on big data, the key lies in ensuring the real-time and accuracy of the data, which I will discuss in detail with you in the section on “data evaluation”.

In addition to quality and business data, big companies also track and analyze user feedback and public sentiment. Almost all major companies have their own systems which crawl user feedback, app stores, Weibo, news, and other sources to monitor public sentiment about their products. You can refer to How Alipay Builds a Public Sentiment Analysis System for Mobile Products.

Summary #

During the process of building an APM system and optimizing continuous delivery, I have come into contact with many test engineers and have paid attention to the current state of testing in major companies. As for the career development of testing, my personal advice is: no matter how testing evolves and how testing efficiency improves, test personnel need to learn to constantly change, change themselves, and change the entire development process. We cannot just stick to our own little field; we need to try many things that developers used to do, such as performance, stability, security, and other aspects of work. To put it more bluntly, if you cannot update your technical skills in a timely manner and do not go deeper into the underlying layers, in the intelligent era of testing, the first to be eliminated will be traditional functional testers.

In this era of change, there are also many independent test engineers around me who have been promoted to experts through platform opportunities, and some outstanding testers have shifted to product or development roles. Therefore, it is still necessary to improve one’s own abilities and seize opportunities.

“Learning is like rowing upstream, not to advance is to retreat,” this is also true for developers. Now that platform tools and frameworks are becoming more mature, many beginners can create cool interfaces with a bunch of open-source tools. If we don’t progress, especially in unfavorable environments, we will also be easily eliminated.

Today, I have shared some of my insights and experiences on efficient testing. If there are any experts in the field of testing among the students, I would love to hear your thoughts on this industry and its development, and share your views on efficient testing.

Homework #

What stage is the testing in your company currently? Do you have any questions about testing? Feel free to leave a message to discuss with me and other classmates.

Today’s homework is to watch the presentations on AI application in the testing field by NetEase, iQiyi, and Facebook. Write down your thoughts and experiences in the comments.

Feel free to click “Please ask a friend to read”, share today’s content with your friends, and invite them to learn together. Lastly, don’t forget to submit today’s homework in the comments section. I have also prepared a generous “learning encouragement package” for students who have completed the homework seriously. I look forward to learning and progressing together with you.