33 How to do acceptance testing well? #

Hello, I’m Zheng Ye.

After the explanations in the previous three lessons, I believe you have gained a relatively complete understanding of what an automated project should be like: programmers write good programs, execute inspections using build scripts, submit the code, create a release image on the server, deploy it to various environments for inspection, and once the inspection is done, it can be released online at any time.

In the previous content, we only mentioned the inspection, but how do we do the inspection? This is where testing comes into play.

In the “Task Breakdown” module, I introduced the concept of developer testing to you in detail, but the testing explained in that part basically stays within the realm of unit testing and integration testing. We did not discuss in detail how to test the entire application.

Today, let’s talk about the topic of application testing: acceptance testing.

Acceptance Testing #

Acceptance Testing is a test to confirm whether an application meets the design specifications. This type of testing is often done from the perspective of the users, to see if the entire application meets the business requirements.

From its name, acceptance testing should be the responsibility of business personnel, but the most they can do is accept the application. It is unlikely for them to perform detailed testing.

Therefore, acceptance testing is often done by the technical team themselves, and in many companies, it is the responsibility of the testing personnel.

Today, many testing teams have the capability for automation. Therefore, automation of acceptance testing is a key consideration. Today, our focus is on how to do automated acceptance testing well.

In fact, acceptance testing should have been the earliest form of automated testing people thought of. Even before unit testing became popular, people began exploring automated acceptance testing. Many teams even built their own frameworks, but these frameworks are not the testing frameworks we understand today. They were frameworks specifically built for testing an application.

For example, I have seen people building a complete testing framework for communication software, and even creating their own language. The role of the testing personnel was to use this specific language to configure and run the system, and see if it met their expectations.

Compared to that, their approach was already quite mature. But for most teams, the reality is that they simply wrap the application access in a simple way, and then writing tests means writing code to call this wrapper.

What has gradually brought acceptance testing out of the chaos of everyone doing their own thing is the concept of Behavior Driven Development (BDD), which many people are familiar with.

Behavior-Driven Development #

In Behavior-Driven Development (BDD), “behavior” refers to business behavior. BDD aims to facilitate collaboration between business personnel and development teams, which means that if you want to do BDD, you should use business language to describe it.

This is quite different from our traditional understanding of system testing, which typically takes the perspective of the development team and describes interactions between systems and external systems using computer terminology.

BDD allows us to change our perspective and use business language for system testing, making it a higher-level abstraction.

BDD was introduced by Dan North in 2003. Besides proposing the concept, Dan North also created the first BDD framework, JBehave, to put his ideas into practice. He later rewrote it in Ruby as RBehave, and this project was eventually merged into RSpec.

Today, the most popular BDD framework is probably Cucumber, whose author is one of the authors of RSpec, Aslak Hellesøy.

Cucumber has evolved from being a Ruby BDD framework to a BDD testing framework that supports many different programming languages, such as Java, JavaScript, PHP, and more.

The most obvious feature of BDD frameworks is the language system they provide to describe the behavior of our applications. Here’s an example that describes a trading scenario where the application needs to determine whether to issue an alert based on the trading result. Take a look:

Scenario: trader is not alerted below threshold
 
Given a stock of symbol STK1 and a threshold of 10.0
When the stock is traded at 5.0
Then the alert status should be OFF

Scenario: trader is alerted above threshold
 
Given a stock of symbol STK1 and a threshold of 10.0
When the stock is traded at 11.0
Then the alert status should be ON

What we’re focusing on in this example is the structure of the description: “Given… When… Then”. This structure corresponds to the execution steps in this test case. “Given” represents an assumption or precondition, “When” represents a specific operation, and “Then” corresponds to the result that this test case must verify.

Do you remember the test structure we discussed? Preparing, executing, asserting, and cleaning. It just happens to correspond to “Given… When… Then”. “Given” corresponds to the precondition, “When” corresponds to the execution, and “Then” corresponds to the assertion. As for cleaning, it involves releasing resources and is more related to implementation rather than being meaningful at the business level.

After understanding the format, we also need to pay attention to the content. You can see that the behavior described here is narrated from a business perspective, and “Given,” “When,” and “Then” are independent and can be freely combined. In other words, once the basic framework is established, we can use these building blocks to write new test cases, even without the involvement of technical personnel.

However, these descriptions are all from a business perspective and do not contain any implementation details. So where does the implementation go?

We still need to define a glue layer that connects the test cases with the implementation. In Cucumber’s terminology, this is called step definition. Here’s an example you can refer to:

public class TraderSteps implements En {
    private Stock stock;

    public TraderSteps() {
        Given("^a stock of symbol {string} and a threshold of {double}", (String symbol, double threshold) -> {
            stock = new Stock(symbol, threshold);
        });

        When("^the stock is traded at {double}$", (double price) -> {
            stock.tradeAt(price);
        });

        Then("the alert status should be {string}", (String status) -> {
            assertThat(stock.getStatus().name()).isEqualTo(status);
        })
    }
}

Writing Good Acceptance Test Cases #

Now that you have a basic understanding of the BDD framework, the next question is how to effectively use the BDD framework. Let’s use a simple example: if we were to write a test case for a login functionality, how would you write it?

One way to write it is as follows. For the sake of convenience, I have translated it into a Chinese description format, but please note that Cucumber itself supports localization, so you can write the test cases using your preferred language:

Given Zhang San is a registered user with the username and password as zhangsan and zspassword respectively
When Zhang San enters zhangsan in the username field and zspassword in the password field
And clicks on the login button
Then Zhang San will be logged in successfully

What do you think of this test case? You may think it’s good if you’re looking at it from a programmer’s perspective. However, as I mentioned before, BDD requires us to look at it from a business perspective, and this example is completely from an implementation perspective.

If the login process changes and the user is automatically logged in after entering the username and password without the need to click, would this test case need to be modified? Let’s try to describe it from a different perspective:

Given Zhang San is a registered user with the username and password as zhangsan and zspassword respectively
When Zhang San logs in with the username zhangsan and password zspassword
Then Zhang San will be logged in successfully

This is a description that looks at it from a business perspective. Unless there are changes in the business requirements and we no longer need to log in with a username and password, this test case doesn’t need to be changed. Even if the specific implementation changes, only the step definitions need to be adjusted.

Therefore, the key to writing good BDD test cases lies in describing them from a business perspective.

When defining step definitions for acceptance test cases, there is another point that people often overlook: the business test model. Many people’s initial instinct is, what does a test need a model for? Do you remember the attributes we discussed for good testing? One of them is Professionalism. Just like writing good code, a good model is essential for writing good tests.

An example that can be referenced in this regard is the Page Object, which is commonly used in Web testing. It encapsulates the access to web pages, so even when writing step definitions, you should not manipulate HTML elements directly in the code, but rather access different page objects.

Using the login example mentioned earlier, we might define a page object like this:

public class LoginPage {
    public boolean login(String name, String password) {
      ...
    }
}

With this, you don’t need to worry about how to locate the input fields in the step definitions, which improves the abstraction level of the code.

Of course, this is just an example reference. When facing your own application, you need to consider building your own business test model.

Summary #

Today I shared with you the topic of automated acceptance testing. Acceptance testing is a test that verifies whether an application meets the design specifications. Acceptance testing is an essential step in technical delivery, but the level of practice varies among different teams. Some rely on manual testing, while others use simple automation. Only a few teams have a well-established automation process.

Automated acceptance testing is also an evolving process, starting from individual efforts and gradually forming a complete system for automated acceptance testing.

Today, I introduced a method of automated acceptance testing based on Behavior Driven Development (BDD). This concept, proposed by Dan North in 2003, has developed into a mature system, especially with the development of BDD frameworks, allowing people to practice BDD in their own projects.

Using Cucumber as an example, I introduced the way to write BDD acceptance test cases. You now know the basic format of “Given… When… Then” and how to write step definitions to connect test cases with implementations.

I also provided the best practices for writing BDD test cases: describe test cases from a business perspective. When writing step definitions, you should also consider designing your own business test model.

In fact, BDD is not the only method for acceptance testing. Specification by Example (SbE) is another common method, and there are also different acceptance testing frameworks besides BDD frameworks. Tools like Concordion can even turn an acceptance test case into a complete reference document.

If you are interested, you can delve deeper into these methods. Regardless of the approach, the goal is to shorten the distance between business stakeholders and the development team, making development more efficient.

If there is only one thing you can remember from today’s content, please remember: automate your acceptance testing.

Finally, I would like to ask you to share how your team conducts acceptance testing. Feel free to share your practices in the comments.

Thank you for reading. If you found this article helpful, please feel free to share it with your friends.