26 Compilation What You Need to Understand #

As an Android engineer, we go through countless compilations every day, and for large projects, each compilation means spending a cup of coffee’s worth of time. Perhaps specific numbers will give you a better understanding. When I was at WeChat, a full compilation of the debug package took 5 minutes, while compiling the release package took more than 15 minutes.

If we could reduce the compilation time by 1 minute each time, it would save the entire Android team at WeChat 1,200 minutes (40 people × 30 compilations per day × 1 minute). Therefore, optimizing compilation speed is very important for improving the overall efficiency of the team.

So, how can we optimize compilation speed? What efforts have companies like WeChat, Google, Facebook, and others made in terms of compilation? Besides compilation speed, what other knowledge do you need to know about compiling?

About Compilation #

Although we compile every day, what exactly is compilation?

You can think of compilation as the process of converting high-level language into a low-level language that can be recognized by a machine or a virtual machine. For Android, this process involves transforming Java or Kotlin into Dalvik bytecode that can be run on the Android virtual machine.

The entire compilation process involves steps such as lexical analysis, syntax analysis, semantic checking, and code optimization. If you are interested in low-level compiler principles, you can challenge the three classic works on compiler principles: “Dragon Book,” “Tiger Book,” and “Whale Book”.

But today, our focus is not on low-level compiler principles. Instead, we hope to discuss the problems that need to be solved in Android compilation, the challenges currently faced, and the solutions provided by domestic and foreign companies.

1. Basics of Android Compilation

Whether it’s WeChat’s compilation optimization or the Tinker project, they involve a lot of compilation-related knowledge. Therefore, I have done a lot of research in Android compilation and have gained considerable experience. Android’s build process mainly includes code, resources, and Native Library. The entire process can be referred to the build process diagram in the official documentation.

Gradle is the official compilation tool for Android, and it is also an open-source project on GitHub. From Gradle’s release notes, it can be seen that this project is still being updated frequently, with new versions released every one or two months. Speaking of Gradle, I feel that the most painful part is writing Gradle Plugins because Gradle does not have comprehensive documentation in this area. Therefore, we can only rely on reading the source code or debugging with breakpoints to develop plugins.

But compilation is really important, and the situation of each company is different, so companies have to create their own “wheels”. The already open-source projects include Facebook’s Buck and Google’s Bazel.

Why do they create their own “wheels”? There are several reasons:

• Unified compilation tools. Facebook and Google have dedicated teams responsible for compilation work, and they hope that all internal projects will use the same set of build tools, including Android, Java, iOS, Go, C++, etc. The unified optimization of build tools benefits all projects.

• Code organization management architecture. Facebook and Google have a special feature in their code management, which is that all projects in the entire company are stored in the same repository. Therefore, the repository is very large, so they do not use Git. Currently, Google uses Piper, and Facebook is based on modified HG, which is also a distributed file system.

• Ultimate performance pursuit. Buck and Bazel do have better performance than Gradle, including various compilation optimizations. However, they have some customizations to varying degrees, and their support for external dependencies such as Maven and JCenter is not ideal.

“Programmers hate writing documentation, as well as when others don’t write documentation,” so their documentation is relatively scarce, and it can be painful to do secondary customization development. If you want to switch the build tool to Buck or Bazel, you need to make a big decision and consider cooperation with other upstream and downstream projects. Even if we don’t directly use them, the optimization ideas inside them are worth learning and referring to.

Gradle, Buck, and Bazel all aim for faster build speed and more powerful code optimization. Let’s take a look at the efforts they have made below.

2. Build Speed

Think back to our Android development journey. How much time and life have we wasted on compilation? As I mentioned earlier, build speed is very important for team efficiency.

When it comes to build speed, what we are most concerned about may be the speed of building debug packages, especially the speed of incremental builds, hoping for faster debugging. As shown in the figure below, every time we validate the code, we have to go through the compilation and installation steps.

• Compilation Time. Compile Java or Kotlin code into .class files, and then compile them into Dex files using dx. For incremental build, we hope to compile as few changed code and resources as possible, ideally only compiling the changed parts. However, due to dependencies between code, this is usually not possible. In most cases, we can only hope to compile fewer modules. Android Plugin 3.0 uses Implementation instead of Compile to optimize dependency relationships.

• Installation Time. We first need to go through signature verification, and after successful verification, there will be a lot of file copying work, such as APK files, library files, Dex files, etc. After that, we need to compile Odex files, which can be very time-consuming, especially on Android 5.0 and 6.0. For incremental builds, the best optimization is to directly apply the new code without reinstalling the new APK. For incremental compilation, let’s first talk about Gradle’s official solution Instant Run. Prior to Android Plugin 2.3, it used the Multidex implementation. After Android Plugin 2.3, it uses the Split APK mechanism introduced in Android 5.0.

As shown in the diagram below, resources and the Manifest are all placed in the Base APK. Only the Instant Run framework code is present in the Base APK, while the application’s own code is in the Split APK.

Instant Run has three modes. For hot and warm swapping, we do not need to reinstall a new Split APK. The difference between the two modes lies in whether the Activity is restarted. For cold swapping, we need to reinstall the modified Split APK using adb install-multiple -r -t, and the application also needs to be restarted.

Although we don’t need to reinstall the Base APK regardless of the mode, Instant Run still has poor performance in large projects due to the following reasons:

• Multi-process issue. “The app was restarted since it uses multiple processes.” If the application has multiple processes, neither hot nor warm swapping will take effect. Since most applications have multiple processes, the speed of Instant Run is greatly reduced.

• Split APK installation issue. Although Split APK installation does not generate Odex files, there is still signature verification and file copying involved (APK installation ping-pong mechanism). This step can take several seconds to minutes, which is not acceptable.

• Javac issue. Prior to Gradle 4.6, if the project used an Annotation Processor, sorry to say, both the current modification and the modules it depends on need a full Javac compilation. This process can be very slow and take several seconds. This issue was only resolved in Gradle 4.7. You can refer to this issue for discussions about the cause of this problem.

You can also take a look at this issue: “full rebuild if a class contains a constant”. If a modified class contains a “public static final” variable, then, sorry again, both the current modification and the modules it depends on need a full Javac compilation. Why is that? Because the constant pool directly compiles the value into other classes, Gradle does not know which classes might use this constant.

When asking the Gradle team, they provided the following solution:

// Original constant definition:
public static final int MAGIC = 23

// Replace the constant definition with a method:
public static int magic() {
  return 23;
}

For large projects, this is obviously not workable. As I wrote in the issue, regardless of whether we really modify this constant or not, Gradle will blindly perform a full Javac compilation, which is definitely not right. In fact, we can compare the current code change to see if any constants have truly been modified.

However, if you have used Alibaba’s Freeline or Meituan’s Quick Compile, you may wonder why their solutions don’t encounter the Annotation and constant issues.

In fact, their solutions are faster than Instant Run in most cases because they sacrifice correctness. That is, in order to pursue faster speed, they directly ignore the potential wrong compilation artifacts caused by changes in Annotations and constants. As the official solution, Instant Run prioritizes ensuring 100% correctness.

Of course, Google has also noticed the various issues with Instant Run. After Android Studio 3.5, a new solution called “Apply Changes” will be used for devices running Android 8.0 and later, replacing Instant Run. Currently, I have not found much information about this new solution, but I believe it should abandon the Split APK mechanism.

I have always had an ideal compilation solution in mind, where the installed Base APK is still just a shell APK, and the actual business code is placed in the ClassesN.dex file in the Assets folder.

• No need to reinstall. Similar to the method used in Tinker hotfix, each time only the modified and dependent classes need to be inserted at the beginning of the path class loader. If you are not familiar, you can refer to the Qzone solution in the article “WeChat Android Hotfix Practice Evolution Road”.

• Oatmeal. In order to solve the problem of the time-consuming Odex of the first run ClassesN.dex in Assets, we can use the black technology in ReDex mentioned in the “Package Optimization”: Oatmeal. It can generate a fully interpreted Odex file within 100 milliseconds.

• Disable JIT. We can disable the JIT optimization of the virtual machine by specifying android:vmSafeMode=“true” in the AndroidManifest, which will also avoid the problems encountered by Tinker in Android N hybrid compilation.

This set of solutions should completely solve the various problems of Instant Run. I also hope that students who are interested in compilation optimization can implement this set of solutions on their own and open source them.

I have a few suggestions for optimizing compilation speed:

• Replace the build machine. For companies with strong capabilities, the simplest way is to directly replace the Mac or other powerful devices as the build machine.

• Build Cache. Most of the projects that don’t change frequently can be separated and use remote Cache mode to retain the compiled cache.

• Upgrade Gradle and SDK Build Tools. We should upgrade to the latest compilation toolchain in a timely manner to enjoy Google’s latest optimization achievements.

• Use Buck. Whether it’s Buck’s exopackage or incremental compilation of code, Buck is more efficient. But as I mentioned earlier, if a large project wants to switch to Buck, there are still many considerations. WeChat integrated Buck in early 2014, but due to collaboration issues with other projects, it switched back to the Gradle solution in 2015. In comparison, the currently hottest feature in Flutter, Hot Reload, which provides instant compilation, may be more attractive.

Of course, in recent versions of Android Studio, Google has also made a lot of other optimizations, such as using AAPT2 to replace AAPT for compiling Android resources. AAPT2 implements incremental compilation of resources, splitting the compilation of resources into two steps: compile and link. The former compiles the resource files into binary Flat format, and the latter merges all the files and then packages them.

In addition to AAPT2, Google has also introduced d8 and R8. Below are some test data provided by Google.

-

So what are d8 and R8? In addition to optimizing compilation speed, what other functions do they have?

3. Code Optimization

For Debug package compilation, speed is more important. But for Release packages, code optimization is more important because we care more about the performance of the application.

Below, I will talk about the following four code optimization tools that we may use: ProGuard, d8, R8, and ReDex.

ProGuard

In the 12-minute compilation process of the WeChat Release package, ProGuard alone takes 8 minutes. Although ProGuard is really slow, it is used in almost every project. After adding ProGuard, the application build process is as follows:

ProGuard mainly has three major functions: obfuscation, shrinking, and optimization. Its entire process includes:

Among them, optimization includes more than 30 types such as inlining, modifiers, merging classes and methods, etc. For specific introductions and usage methods, you can refer to the official documentation. d8

Android Studio 3.0 introduced d8 and it officially became the default tool in 3.1. Its purpose is to compile “.class” files into Dex files, replacing the previous dx tool.

In addition to faster compilation speed, d8 also has an optimization that reduces the size of the generated Dex. According to Google’s test results, there is approximately a 3% to 5% optimization.

R8

R8 was introduced in Android Studio 3.1, and its ambition is even higher. Its goal is to replace ProGuard and d8. We can directly use R8 to convert “.class” files into Dex.

At the same time, R8 also supports the three major functions of obfuscation, shrinking, and optimization in ProGuard. Since R8 is still in the experimental stage, there is not much information available online. You can refer to the following resources:

• A comparison of ProGuard and R8: ProGuard and R8: a comparison of optimizers.
• Jake Wharton’s blog has many recent articles about R8: https://jakewharton.com/blog/.

The ultimate goal of R8 is the same as d8: faster compilation speed and more powerful code optimization.

ReDex

If R8 is the tool that aims to replace ProGuard in the future, then Facebook’s internal tool, ReDex, has already achieved that.

Many of Facebook’s internal projects have switched to using ReDex instead of ProGuard. Unlike ProGuard, ReDex directly operates on Dex objects, rather than “.class” files. In other words, it optimizes the final output directly, what you see is what you get.

In previous articles, I have mentioned the ReDex project several times because it has incredibly powerful features. For more details, you can refer to the articles in my column 《Package Volume Optimization (Part 1): How to Reduce Installation Package Size?》.

• Interdex: class and file reordering, Dex partitioning optimization.
• Oatmeal: directly generates Odex files.
• StripDebugInfo: removes debug information from Dex.

In addition, ReDex offers other excellent features such as Type Erasure and removing Access Methods from code. Both of these features are helpful for reducing package size and improving application runtime speed. Therefore, I encourage you to study and practice using them. However, the documentation for ReDex is rarely updated and it includes some Facebook-specific custom logic, making it difficult to use. Currently, I mainly study its source code to understand its principles and then separately implement them.

In fact, there are also many useful things in Buck, but the documentation does not mention anything, so “reading the source code” is still needed.

• Library Merge and Relinker
• Multi-language splitting
• Dex partitioning support
• ReDex support

Continuous Delivery #

Gradle, Buck, and Bazel represent a narrow definition of compilation. I believe that the broader definition of compilation should include processes such as packaging, code review, code engineering management, and code scanning, which are commonly referred to as continuous integration in the industry.

The most commonly used continuous integration tools currently include Jenkins, GitLab CI, Travis CI, etc. GitHub also provides its own continuous integration service. Each major company has its own continuous integration solution, such as Tencent’s RDM, Alibaba’s Ferris Wheel, Meituan-Dianping’s MCI, etc.

Now let me briefly share some of my experiences and opinions on continuous integration:

• Custom code checks. Each company will have its own coding standards, and code checks aim to prevent code that does not comply with the standards from being committed to remote repositories. For example, WeChat has defined a set of code standards and has written dedicated plugins for detection. Examples include log standards, restrictions on direct use of new Thread, new Handler, etc., and violators will be penalized. Custom code checks can be implemented entirely on your own or by extending the Findbugs plugin. Meituan, for example, uses Findbugs to implement the Android vulnerability scanning tool Code Arbiter.

• Third-party code checks. The commonly used code scanning tools in the industry include the commercial tool Coverity and the open-source tool Infer developed by Facebook. These tools can scan for various issues such as null pointers, multi-threading problems, resource leaks, etc. In addition to adding the detection process, my biggest insight is that personnel training needs to be increased at the same time. I have encountered many developers who, in order to resolve the issues uncovered by the scans, add null checks directly for null pointers, or add locks directly for multi-threading, which may ultimately cause more serious problems.

• Code Review. When it comes to code review, integrating with GitLab, Phabricator, or Gerrit are all good choices. We must value code review, as it is an opportunity to showcase our “amazing” code to others. And we should be the first code reviewer ourselves. Before reviewing someone else’s code, we should examine our own code from a third-party perspective. By passing our own test first, we not only respect others’ time but also establish a good technical reputation for ourselves.

There are many processes involved in continuous integration, and you need to consider your team’s current situation. Sometimes, blindly adding processes may have the opposite effect.

Summary #

In Android 8.0, Google introduced the Dexlayout library to rearrange classes and methods. Facebook’s Buck also promptly introduced AAPT2. ReDex, d8, and R8 complement each other. We can see that Google is also absorbing knowledge from the community, and at the same time, we can also seek inspiration from Google’s new technological developments.

When I was writing today’s content, I also had another realization. Google spent a lot of effort trying to solve the problem of Android compilation speed, but the results were not entirely satisfying. I want to say that if we dare to break free from the constraints of the system, we may be able to completely solve this problem. Just like in Flutter, we can achieve instant compilation. This is also true for our personal lives and work. We often get trapped in the dilemma of local optima or fall into “thinking loops”. In such situations, if we can break free from path dependence, rethink from a higher perspective, and examine the whole situation, our insights may be completely different.

Homework #

In your work, what compilation problems have you encountered? Have you done any specific optimization work to improve compilation speed? Do you have any questions about compilation? Feel free to leave a comment and discuss with me and other classmates.

As for the Android Build System, it can be said that there are many changes every year, and many new things come out. So we should maintain sensitivity, and you will find that many tools are very useful, such as Desugar, Dexlayout, JVM TI, App Bundle, etc.

Today’s homework is to watch the videos related to the compilation tools in the 2018 Google I/O conference, and write down your thoughts and experiences in the comments.

• What’s new with the Android build system (Google I/O ‘18)

• What’s new in Android development tools

Feel free to click “Invite Friends to Read” to share today’s content with your friends and invite them to learn together. Don’t forget to submit today’s homework in the comment section. I have also prepared a generous “Study Boost Giveaway” for students who complete the homework seriously. Look forward to discussing and progressing together with you!