40 Balancing Online Quality of Flutter Apps We Need to Focus on These Three Metrics #

Hello, I’m Chen Hang.

In my previous article, I shared with you how to capture unhandled exceptions in Flutter applications. Exceptions refer to unexpected error events that occur during the runtime of Dart code. For a single exception, we can use try-catch or catchError to handle it. However, if we want to centrally intercept and manage exceptions, we need to use Zones and combine them with FlutterError for unified management. Once an exception is caught, we can use third-party data reporting services (such as Bugly) to report its contextual information.

Monitoring data of these online exceptions is crucial for developers to detect hidden risks as early as possible and determine the root causes of problems. If we want to further assess the overall stability of an application, we need to associate the exception information with the rendering of pages. For example, did the page rendering process encounter an exception that rendered the functionality unusable?

For Flutter applications, known for their “silky smooth” performance, we also need to pay special attention to the performance of page rendering. For example, whether the interface rendering causes frame drops or lags, or whether the page loading has performance issues that result in excessive latency. Although these problems may not render the application completely unusable, they can easily raise questions or even aversion from users about the quality of the application.

Based on the above analysis, the metrics for measuring the overall quality of a Flutter application can be divided into the following three categories:

• Page exception rate;
• Page frame rate;
• Page loading time.

Among them, the page exception rate reflects the health of the page, the page frame rate reflects the smoothness of the visual effects, and the page loading time reflects the end-to-end latency during the entire rendering process.

These three data metrics are important quality indicators for assessing the excellence of a Flutter application. By defining the statistical approach for these metrics and establishing the quality monitoring capability of a Flutter application, we can not only detect hidden risks early on, but also establish quality baselines to continuously improve the user experience.

Therefore, in today’s sharing session, I will explain in detail how to collect these three metrics.

Page Exception Rate #

The page exception rate refers to the probability of exceptions occurring during the page rendering process. It measures the unavailability of functionality at the page level, and its statistical formula is: Page Exception Rate = Number of Exception Occurrences / Overall Page PV Count.

After understanding the statistical approach of the page exception rate, let’s take a look at how to calculate the numerator and denominator in this formula.

Let’s start with the method of calculating the number of exception occurrences. In the previous article, we already know that in Flutter, unhandled exceptions need to be captured through Zone and FlutterError. So, if we want to count the number of exception occurrences, we still need to use these two methods, but we need to use a counter in the exception interception method to accumulate and record them.

The following example demonstrates the specific method of counting the number of exception occurrences. We use a global variable exceptionCount to continuously accumulate the number of exceptions caught in the callback method _reportError:

int exceptionCount = 0; 
Future<Null> _reportError(dynamic error, dynamic stackTrace) async {
  exceptionCount++; //accumulate the number of exceptions
  FlutterCrashPlugin.postException(error, stackTrace);
}

Future<Null> main() async {
  FlutterError.onError = (FlutterErrorDetails details) async {
    //forward the exception to Zone
    Zone.current.handleUncaughtError(details.exception, details.stack);
  };

  runZoned<Future<Null>>(() async {
    runApp(MyApp());
  }, onError: (error, stackTrace) async {
    //intercept the exception
    await _reportError(error, stackTrace);
  });
}

Next, let’s take a look at how to calculate the overall page PV count. The overall page PV count is actually the number of times the page is opened. Through the 21st article “Routes and Navigation: How Flutter Implements Page Switching,” we already know that Flutter page navigation needs to go through Navigator to be implemented, so the status of page transitions also needs to be sensed through Navigator.

Similar to registering page routes, in MaterialApp, we can use the NavigatorObservers property to listen to the opening and closing of pages. The following example demonstrates the specific use of NavigatorObserver. In the code below, we define an observer that inherits from NavigatorObserver, and in its didPush method, we count the page opening behavior:

int totalPV = 0;
//navigation observer
class MyObserver extends NavigatorObserver{
  @override
  void didPush(Route route, Route previousRoute) {
    super.didPush(route, previousRoute);
    totalPV++; //accumulate PV
  }
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      //set up route monitoring
      navigatorObservers: [
        MyObserver(),
      ],
      home: HomePage(),
    );
  }
}

Now, we have collected the two parameters of the number of exception occurrences and the overall page PV count, and we can calculate the page exception rate:

double pageExceptionRate() {
  if(totalPV == 0) return 0;
  return exceptionCount / totalPV;
}

As you can see, calculating the page exception rate is relatively simple.

Page Frame Rate #

Page Frame Rate, also known as FPS, is a term used in the field of image processing to refer to the number of frames transmitted per second. Due to the persistence of human vision, when the number of frames transmitted in a certain period of time is higher than a certain threshold, the visual effect is considered to be smooth. Therefore, for dynamic pages, the more frames displayed per second, the smoother the image.

From this, we can deduce that the calculation of FPS is the total number of frames rendered in a unit of time. In mobile devices, the recommended value for FPS is usually 60Hz, which means refreshing the page 60 times per second.

Why is it 60Hz instead of a higher or lower value? This is because the display process is driven by the VSync signal, and the period of the VSync signal is 60 times per second, which is also the upper limit of FPS.

After receiving the VSync signal, the CPU and GPU calculate the graphical image, prepare the rendering content, submit it to the frame buffer, and wait for the next VSync signal to display it on the screen. If the CPU or GPU does not complete the content submission within one VSync period, the frame will be discarded and displayed on the next opportunity, while the page retains the previous content, resulting in screen freezing. Therefore, when the FPS is lower than 60Hz, frame dropping occurs, and if it is lower than 45Hz, serious freezing occurs.

For easy statistics of FPS, Flutter provides a frame callback mechanism on the global window object. We can register the onReportTimings method on the window object to inform us of the recent frame drawing time (i.e. FrameTiming) through a callback. With the drawing time of each frame, we can calculate the FPS.

Note that the onReportTimings method only has data callbacks when a frame is drawn. If the user does not interact with the app and the interface state does not change, no new frames will be generated. Considering the large difference in the drawing time of individual frames, frame-by-frame calculations may result in data jumps. Therefore, to make the calculation of FPS smoother, we need to keep the most recent 25 FrameTimings for a sum calculation.

On the other hand, for the calculation of FPS, we cannot consider the frame drawing time in isolation, but should consider the period of the VSync signal, that is, 1/60 second (16.67 milliseconds), to make a comprehensive evaluation.

Since the rendering of frames is driven by the VSync signal, if the frame drawing time does not exceed 16.67 milliseconds, we need to treat it as 16.67 milliseconds, because the completed frame must wait until the next VSync signal to render. If the frame drawing time exceeds 16.67 milliseconds, it will occupy the subsequent VSync signal periods, thus disrupting the subsequent drawing sequence and causing freezing. There are two cases here:

• If the frame drawing time is exactly an integer multiple of 16.67, such as 50, it means that it took 3 VSync signal periods, so it could have drawn 3 frames, but in fact it only drew 1 frame;
• If the frame drawing time is not an integer multiple of 16.67, such as 51, then the number of VSync signal periods it took should be rounded up, that is, 4, which means it could have drawn 4 frames, but in fact it only drew 1 frame.

Therefore, our final FPS calculation formula is: FPS = 60 * actual rendered frames / frames that should have been rendered in this time.

The following example demonstrates how to calculate FPS through the onReportTimings callback function. In the code below, we define a list with a capacity of 25 to store the recent frame drawing time FrameTiming. In the FPS calculation function, we compare the drawing time of each frame in the list with the VSync period frameInterval to obtain the number of frames that should have been drawn. Finally, dividing the two gives us the FPS indicator.

Note that the FPS information displayed in the Flutter plugin provided by Android Studio also comes from the onReportTimings callback. Therefore, we need to keep the original callback reference when registering the callback, otherwise the plugin will not read the FPS information.

import 'dart:ui';

var originalCallback;

void main() {
  runApp(MyApp());
  // Set the frame callback function and save the original frame callback function
  originalCallback = window.onReportTimings;
  window.onReportTimings = onReportTimings;
}

// Cache only the recent 25 frame drawing times
const maxframes = 25;
final lastFrames = List<FrameTiming>();
// Benchmark VSync signal period
const frameInterval = const Duration(microseconds: Duration.microsecondsPerSecond ~/ 60);

void onReportTimings(List<FrameTiming> timings) {
  lastFrames.addAll(timings);
  // Only keep 25 frames
  if(lastFrames.length > maxframes) {
    lastFrames.removeRange(0, lastFrames.length - maxframes);
  }
  // If there is an original frame callback function, execute it
  if (originalCallback != null) {
    originalCallback(timings);
  }
}

double get fps {
  int sum = 0;
  for (FrameTiming timing in lastFrames) {
    // Calculate the rendering time
    int duration = timing.timestampInMicroseconds(FramePhase.rasterFinish) - timing.timestampInMicroseconds(FramePhase.buildStart);
    // Check if the time is within the Vsync signal period
    if(duration < frameInterval.inMicroseconds) {
      sum += 1;
    } else {
      // Frame dropped, round up
      int count = (duration/frameInterval.inMicroseconds).ceil();
      sum += count;
    }
  }
  return lastFrames.length/sum * 60;
}

By running this code, we can see that the FPS indicator we calculated is consistent with the FPS trend displayed by the Flutter plugin.

Figure 1. FPS trend

Page Load Duration #

Page load duration refers to the time it takes for a page to become visible from its creation. It reflects whether there is excessive rendering in the code or improper rendering that leads to a prolonged creation time for the page view.

From the definition, the calculation of page load duration is the time when the page becomes visible minus the time it was created. It is relatively easy to obtain the time of page creation; we only need to record the time in the initialization function of the page. So, how should we calculate the time when the page becomes visible?

In the 11th article, “What Are We Talking About When We Mention Lifecycle?,” I introduced Flutter’s frame callback mechanism while discussing the widget lifecycle. The WidgetsBinding provides a method called addPostFrameCallback, which registers a callback to be executed after the current frame is drawn, and it is only called once. Once we detect that the frame has been drawn, we can confirm that the page has been rendered. Therefore, we can leverage this method to obtain the time when the page becomes visible.

The example below demonstrates how to use the frame callback mechanism to obtain the page load duration. In the code below, we record the creation time startTime of the page in the initialization method of the MyHomePage widget. Then, in the initialization method of the page state, we register a frame callback using addPostFrameCallback, and in the callback function, we record the completion time of the page rendering as endTime. By subtracting these two times, we obtain the page load duration of MyHomePage:

class MyHomePage extends StatefulWidget {
  int startTime;
  int endTime;
  MyHomePage({Key key}) : super(key: key) {
    // Record the start time at the page initialization
    startTime = DateTime.now().millisecondsSinceEpoch;
  }
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  @override
  void initState() {
    super.initState();
    // Obtain the completion time of rendering through the frame callback
    WidgetsBinding.instance.addPostFrameCallback((_) {
      widget.endTime = DateTime.now().millisecondsSinceEpoch;
      int timeSpend = widget.endTime - widget.startTime;
      print("Page render time: ${timeSpend} ms");
    });
  }
  ...
}

Try running the code and observe the command line output:

flutter: Page render time: 548 ms

As you can see, the page load time calculated through the frame callback is 548 milliseconds.

With this, we have obtained three metrics: page error rate, page frame rate, and page load duration.

Conclusion #

Alright, that’s all for today’s sharing. Let’s summarize the main content.

Today we learned about three metrics for measuring the online quality of Flutter applications: page exception rate, page frame rate, and page load time, as well as the corresponding data collection methods.

Page exception rate indicates the stability of page rendering process. It can be calculated by capturing unhandled exceptions centrally, observing page PV with NavigatorObservers, and calculating the probability of unavailable functionality at the page level.

Page frame rate represents the smoothness of the page. By using the frame drawing time callback onReportTimings provided by Flutter, we can calculate the number of frames that should have been rendered, and obtain a more accurate FPS.

Page load time reflects the delay in the rendering process. We can use the single-frame callback mechanism to get the completion time of page rendering and obtain the overall page load time.

With these three data metrics and statistical methods, we now have a specific digital standard for evaluating the performance of Flutter applications. With the data, we can not only identify potential problems early, accurately locate and fix issues, but also evaluate the health of the application and the rendering performance of the pages, thereby determining the direction for further optimization.

I have packaged the knowledge covered in today’s sharing on GitHub. You can download it, run it multiple times to deepen your understanding and memory.

Thought-provoking Question #

Finally, I have a thought-provoking question for you.

If the rendering of a webpage relies on data from one or more network interfaces, how should the page load time be measured?

Please feel free to leave your thoughts in the comment section. I will be waiting for you in the next article! Thank you for reading, and please share this article with more friends for them to read as well.