38 How to Improve Delivery Quality Through Automated Testing

38 How to Improve Delivery Quality Through Automated Testing #

Hello, I’m Chen Hang.

In the previous article, I shared with you how to analyze and optimize performance issues in Flutter applications. By running the application in profiling mode on a real device, we can use performance layers to identify two types of performance bottlenecks: GPU rendering issues and CPU execution time issues. Then, we can use Flutter’s rendering switches and CPU frame profile (flame chart) to check for excessive rendering or long code execution in the application, and then locate and address the performance issues.

After completing the development work of the application, and resolving the logic and performance issues in the code, the next step is to test and validate the performance of various features of the application. Mobile application testing is usually a labor-intensive task because in order to validate the user experience, testing often requires manual execution across multiple platforms (Android/iOS) and different physical devices.

As the product’s features continue to accumulate and iterate, the workload and complexity of testing also increase significantly, making manual testing increasingly difficult. So, how can we ensure that the application continues to function properly when adding new features or modifying existing ones?

The answer is through writing automated test cases.

Automated testing is the process of replacing human-driven testing with machine execution. Specifically, through carefully designed test cases, the application is automatically tested by the machine according to the execution steps, and the execution results are output. Finally, the results are determined whether they meet the expected criteria defined in the test cases.

In other words, automated testing transforms repetitive and mechanical manual operations into automated verification steps, greatly saving manpower, time, and hardware resources, thereby improving testing efficiency.

In terms of writing automated test cases, Flutter provides capabilities for both unit testing and UI testing. Unit testing allows for easy verification of the behavior of individual functions, methods, or classes, while UI testing provides the ability to interact with Widgets to confirm their functionality.

Next, let’s take a closer look at the usage of these two types of automated test cases.

Unit Testing #

Unit testing refers to the way of verifying the smallest testable units in software and determining if the behavior of the smallest unit matches the expected results. The smallest testable unit, generally speaking, is a manually defined, minimal functional module that can be tested, such as a statement, function, method, or class.

In Flutter, we can write unit test cases by using the test package in the pubspec.yaml file. The test package provides the core framework for defining, executing, and validating unit test cases. The following code shows how to use the test package:

dev_dependencies:
  test:

Note: The declaration of the test package needs to be completed under dev_dependencies. Packages defined under this tag will only take effect in development mode.

Similar to how Flutter applications define the program entry point through the main function, unit test cases in Flutter are also defined through the main function. However, the directory location of these two program entry points differs slightly: the entry point of the application is located in the lib directory of the project, while the entry point of the test cases is located in the test directory of the project.

The directory structure of a Flutter project with unit test cases is shown below:

Figure 1 Flutter project directory structure

Next, we can declare a class for testing in main.dart. In the example below, we declare a counter class called Counter, which supports modifying the count value in an increment or decrement manner:

class Counter {
  int count = 0;
  void increase() => count++;
  void decrease() => count--;
}

After implementing the class to be tested, we can write test cases for it. In Flutter, a test case declaration usually consists of three parts: defining, executing, and validating. The definition and execution determine the smallest testable unit provided by the object to be tested and the verification is done using the expect function, which compares the execution result of the smallest testable unit with the expected result.

Therefore, to write a test case in Flutter, the following two steps are typically involved:

  1. Implement a test case that includes the definition, execution, and verification steps.
  2. Wrap it inside the test function, which is a test case encapsulation class provided by Flutter.

In the example below, we define two test cases. The first test case verifies whether the counter value is 1 after calling the increase function, while the second test case checks if 1+1 is equal to 2:

import 'package:test/test.dart';
import 'package:flutter_app/main.dart';

void main() {
  // The first test case to verify if the Counter object
  // is equal to 1 after calling the increase method
  test('Increase a counter value should be 1', () {
    final counter = Counter();
    counter.increase();
    expect(counter.value, 1);
  });
  // The second test case to check if 1+1 equals 2
  test('1+1 should be 2', () {
    expect(1 + 1, 2);
  });
}

Select the widget_test.dart file and choose “Run ’tests in widget_test’” from the right-click menu to start running the test cases.

Figure 2 Entry point for running test cases

After a short wait, the console will output the results of the test cases. Both of these test cases pass:

22:05	Tests passed: 2

If a test case fails, how does Flutter notify us? Let’s try modifying the expected result of the first counter increment test case to 2:

test('Increase a counter value should be 1', () {
  final counter = Counter();
  counter.increase();
  expect(counter.value, 2); // Checking if the Counter object equals 2 after calling increase
});

After running the test case, we can see that after executing the counter’s increment method, Flutter found that the result 1 does not match the expected result 2, so it reported an error:

Figure 3 Example of failed unit test

The above example demonstrates the method for writing individual test cases. If there are multiple test cases, they are related to each other, and we can group them together using the group function at the highest level.

In the example below, we define two test cases for counter increment and decrement respectively. We verify that the result of the increment is 1 and at the same time, we check if the result of the decrement is equal to -1. After that, we group these test cases together:

import 'package:test/test.dart';
import 'package:counter_app/counter.dart';

void main() {
  // Grouping test cases
  // The first test case to verify if the Counter object
  // is equal to 1 after calling the increase method
  // In addition, it checks if the Counter object is equal to -1 after calling the decrease method
  group('Counter', () {
    test('Increase a counter value should be 1', () {
      final counter = Counter();
      counter.increase();
      expect(counter.value, 1);
    });
    test('Decrease a counter value should be -1', () {
      final counter = Counter();
      counter.decrease();
      expect(counter.value, -1);
    });
  });
}

group('Counter', () {
  test('Increasing the counter value should be 1', () {
    final counter = Counter();
    counter.increase();
    expect(counter.value, 1);
  });

  test('Decreasing the counter value should be -1', () {
    final counter = Counter();
    counter.decrease();
    expect(counter.value, -1);
  });
});

The results of these two test cases are also passed.

**When performing unit testing of the internal functionality of a program, we may also need to obtain the data to be tested from external dependencies (such as web services).** For example, in the following example, the initialization of the Todo object is implemented through the JSON returned by the web service. Considering that there may be errors in the process of calling the web service, we also handle other exceptions where the request code is not equal to 200:

```dart
import 'package:http/http.dart' as http;

class Todo {
  final String title;
  Todo({this.title});
  // Factory constructor that converts JSON into an object
  factory Todo.fromJson(Map<String, dynamic> json) {
    return Todo(     
      title: json['title'],
    );
  }
}

Future<Todo> fetchTodo(http.Client client) async {
  final response =
  await client.get('https://xxx.com/todos/1');

  if (response.statusCode == 200) {
    // Request successful, parse JSON
    return Todo.fromJson(json.decode(response.body));
  } else {
    // Request failed, throw exception
    throw Exception('Failed to load post');
  }
}
```

Considering that these external dependencies are not controlled by our program, it is difficult to cover all possible success or failure scenarios. For example, for a normally running web service, we can hardly test how the `fetchTodo` interface responds to 403 or 502 status codes. Therefore, a better approach is to "simulate" these external dependencies in the test case, and let these external dependencies return specific results.

To **simulate the external dependency `http.client` used in the `fetchTodo` using the `mockito` package**, we first need to define a mock class that extends from `Mock` (this class can simulate any external dependency) and implements the simulated class of `http.client` in the form of an interface. Then, in the declaration of the test case, we specify any interface return for it.

In the following example, we defined a mock class `MockClient`, which gets the external interface of `http.Client` in the form of an interface declaration. Afterwards, we can use the `when` statement to inject the corresponding data for it to return when calling the web service. In the first test case, we inject the JSON result; while in the second test case, we inject a 403 exception.

```dart
import 'package:mockito/mockito.dart';
import 'package:http/http.dart' as http;

class MockClient extends Mock implements http.Client {}

void main() {
  group('fetchTodo', () {
  test('returns a Todo if successful', () async {
    final client = MockClient();

    // Use Mockito to inject the successful JSON field
    when(client.get('https://xxx.com/todos/1'))
        .thenAnswer((_) async => http.Response('{"title": "Test"}', 200));
    // Verify if the request result is an instance of Todo
    expect(await fetchTodo(client), isInstanceOf<Todo>());
  });

  test('throws an exception if error', () {
    final client = MockClient();

    // Use Mockito to inject the failed request error
    when(client.get('https://xxx.com/todos/1'))
        .thenAnswer((_) async => http.Response('Forbidden', 403));
    // Verify if the request result throws an exception
    expect(fetchTodo(client), throwsException);
  });
});
}
```

Running this test case, we can see that we successfully simulated both the normal and exception results without calling the real web service, and also passed the verification smoothly.

Next, let's take a look at UI testing.

UI Testing #

The purpose of UI testing is to simulate the behavior of real users, that is, to perform UI interactions on the application as a real user and cover various user flows. Compared to unit testing, UI testing has a wider coverage and focuses more on processes and interactions, making it possible to identify errors that cannot be found during unit testing.

To write UI test cases in Flutter, we need to use the flutter_test package in pubspec.yaml to provide the core framework for writing UI tests, which includes defining, executing, and verifying:

  • Defining: By specifying rules, we can find the specific sub-widget objects that UI test cases need to verify.
  • Executing: This means that we need to apply user interaction events to the found sub-widget objects.
  • Verifying: After applying the interaction events, we determine whether the overall behavior of the widget objects to be verified meets our expectations.

The following code demonstrates the usage of the flutter_test package:

dev_dependencies:
  flutter_test:
    sdk: flutter

Next, I will use the default Flutter template of a timer application as an example to explain the method of writing UI test cases.

In the counter application, there are two places that respond to external interaction events, including the button icon that responds to user click actions, and the text that responds to rendering refresh events. After the button is clicked, the counter will accumulate and the text will be refreshed accordingly.

Figure 4: Counter Example

To ensure the proper functionality of the program, we want to write a UI test case to verify if the button click behavior matches the text refresh behavior.

Similar to wrapping test cases using test for unit testing, UI testing uses testWidgets to wrap test cases. testWidgets provides a tester parameter, which we can use to manipulate the widget objects to be tested.

In the code below, we first declare the MyApp object that needs to be verified. After triggering the completion of its rendering with pumpWidget, we use the find.text method to find the Text widgets with string values of 0 and 1 respectively. The purpose is to verify if the initial state of the Text widget, which responds to refresh events, is 0.

Next, we use the find.byIcon method to find the button widget and use the tester.tap method to apply a click event to it. After the click event is completed, we use the tester.pump method to force the widget to complete rendering and refresh. Finally, we use the same statements as verifying the initial state of the Text widget to determine if the state of the Text widget after responding to the refresh event is 1:

import 'package:flutter_test/flutter_test.dart';
import 'package:flutter_app_demox/main.dart';

void main() {
  testWidgets('Counter increments UI test', (WidgetTester tester) async {
    // Declare the Widget object to be verified (i.e. MyApp) and trigger its rendering
    await tester.pumpWidget(MyApp());

    // Find the Widget with string value '0' and verify that it is found
    expect(find.text('0'), findsOneWidget);
    // Find the Widget with string value '1' and verify that it is not found
    expect(find.text('1'), findsNothing);

    // Find the '+' button and apply a click event
    await tester.tap(find.byIcon(Icons.add));
    // Trigger rendering
    await tester.pump();

    // Find the Widget with string value '0' and verify that it is not found
    expect(find.text('0'), findsNothing);
    // Find the Widget with string value '1' and verify that it is found
    expect(find.text('1'), findsOneWidget);
  });
}

Running this UI test case code also passed the verification successfully.

In addition to click events, the tester object also supports other interaction behaviors, such as entering text enterText, dragging drag, long pressing longPress, and so on. I won’t go into detail on each of them here. If you want to have a deeper understanding of these contents, you can refer to the official documentation of WidgetTester for learning: https://api.flutter.dev/flutter/flutter_test/WidgetTester-class.html.

Summary #

Alright, that’s all for today’s sharing. Let’s summarize the main content of today.

In Flutter, automated testing can be divided into unit testing and UI testing.

The steps for unit testing include defining, executing, and validating. Through unit test cases, we can verify whether the behavior of individual functions, methods, or classes matches the expected behavior. The steps for UI testing are also defining, executing, and validating. We can interact with the application by mimicking the behavior of real users, covering a wider range of processes.

If the test object has external dependencies such as web services, in order to make the unit testing process more controllable, we can use mockito to customize the data returned, implementing normal and abnormal test cases.

It is worth noting that although UI testing expands the scope of application testing and can find errors that cannot be found during unit testing, the development and maintenance cost of UI test cases is very high compared to unit test cases. This is because the main functionality of a mobile application is actually the UI, and the UI changes very frequently. UI tests need to be constantly maintained to ensure stability and usability.

“The cost and benefit” should always be the primary consideration when deciding whether to adopt UI testing and what level of UI testing to adopt. The principle I recommend is to consider the necessity of UI testing only after the project reaches a certain scale and the business features have a certain degree of continuity.

I have packaged the knowledge points covered in today’s sharing on GitHub. You can download it, run it multiple times to deepen your understanding and memory.

Discussion Question #

Finally, I have a discussion question for you.

In the code below, we have defined the update and increment methods for SharedPreferences. Please use the Mockito framework to mock SharedPreferences and implement corresponding unit test cases for these two methods.

Future<bool> updateSP(SharedPreferences prefs, int counter) async {
  bool result = await prefs.setInt('counter', counter);
  return result;
}

Future<int> increaseSPCounter(SharedPreferences prefs) async {
  int counter = (prefs.getInt('counter') ?? 0) + 1;
  await updateSP(prefs, counter);
  return counter;
}

Feel free to leave a comment in the comment section and share your thoughts. I will be waiting for you in the next article! Thank you for reading, and feel free to share this article with more friends to read together.