22 Package Size Optimization Part1 How to Reduce the Size of Installation Packages

22 Package Size Optimization Part1 How to Reduce the Size of Installation Packages #

Back in 2015, I wrote an article on the WeMobileDev official account titled “Summary of Android Installation Package Related Knowledge” 《Android安装包相关知识汇总》, and I also released an obfuscation tool called AndResGuard AndResGuard, which many students have used.

Looking back at this article from 4 years ago, it’s like seeing my past self and it brings back a lot of emotions. Over the years, there have been numerous articles on optimizing installation packages available online. So, what are some “profound” secret techniques worth sharing?

Today, the WeChat installation package size has increased from 30MB in the past to 100MB. We often wonder if there is still any value in optimizing package volume since WiFi is so widely available now and 5G is on the horizon. What is the value of package volume optimization for users and applications?

Background knowledge about package installation #

Do you remember in the 2G era, when we only had 30MB of monthly data? At that time, the size of installation packages was indeed crucial. When I was working on the “Sogou Input Method”, we strictly required the package size to be within 5MB.

Several years have passed, have our perspectives on package size changed?

1. Why optimize package size

At the 2018 Google I/O, Google revealed the relationship between installation package size and download conversion rate on Google Play.

From this graph, it can be generally concluded that the smaller the installation package, the higher the conversion rate. The impact of package size on applications mainly includes the following points:

  • Download Conversion Rate. Even if a 100MB app is clicked to download, it may fail due to slow internet speed or sudden change of mind. For a 10MB app, after a user clicks to download, it may have already finished downloading by the time they hesitate to decide whether to proceed. However, as shown in the graph, the relationship between installation package size and conversion rate is very subtle. The difference between 10MB and 15MB may not be significant, but the difference between 10MB and 40MB is quite apparent.

  • Promotion Cost. Generally, package size has a significant impact on the unit price for channel promotion and pre-installation by manufacturers. Especially for pre-installation by manufacturers, this is mainly because the total space left for pre-installed apps is limited. If your package size is very large, it will affect the pre-installation of other apps by manufacturers.

  • App Stores. Apple’s App Store requires apps exceeding 150MB to be downloaded using WiFi network only, while Google Play requires apps exceeding 100MB to be uploaded using APK Expansion Files. This indicates that the package size does impose some pressure on the server bandwidth cost of app stores.

Currently, there are more and more mature super apps, and many products also hope to become the next super app, hoping to encompass all the features and fulfill all the needs of users. However, this also leads to the continuous increase in package size, while in fact, many users only use a small portion of the features.

Now let’s take a look at the growth of installation packages for several super apps: WeChat, QQ, Alipay, and Taobao, in recent years.

I still remember back in 2015, in order to make the package size of WeChat version 6.2 smaller than 30MB, I used various means to reduce its size from 34MB to 29.85MB. The resource obfuscation tool AndResGuard was also developed during that optimization project. Several years have passed, and now the package size of WeChat has increased to 100MB, and the situation for Taobao does not seem optimistic either. In comparison, QQ and Alipay are relatively more conservative.

2. Package size and application performance

React Native 5MB, Flutter 4MB, browser kernel 20MB, Chromium networking library 2MB… Now there are more and more third-party development frameworks and extension libraries, and the package size of many applications starts from several tens of megabytes.

Apart from the impact on conversion rate, what other effects does package size have on application performance?

  • Installation Time. File copying, library decompression, ODEX compilation, signature verification, especially for Android 5.0 and 6.0 systems (after Android 7.0, there is mixed compilation), the time taken just for compiling the ODEX of WeChat’s 13 DEX files may be about 5 minutes.

  • Runtime Memory. We have mentioned in memory optimization that resource assets, libraries, and Dex class loading all occupy a significant amount of memory.

  • ROM Space. After startup and decompression, a 100MB installation package may easily exceed 200MB. This can also put great pressure on low-end device users. As we discussed in “I/O Optimization”, if the flash memory space is insufficient, write amplification can easily occur.

For most “affordable smartphones” released one or two years ago, both Taobao and WeChat can no longer run smoothly. “Technology is overestimated in the short term, and underestimated in the long run.” Especially during rapid business development, performance is often pushed to the back.

Package size should be a very important technical indicator for technical personnel. We cannot allow it to keep growing, as it still carries significant implications for us.

  • Business Review. Deleting unused or low-value businesses is always the most effective performance optimization method. We need to regularly review past businesses and not just focus on moving forward. We should also address some “technical debts” in a timely manner.

  • Upgrade Development Patterns. If it is not possible to remove all the functionalities, it may be necessary to change the development pattern, and make more use of small programs and H5 development approaches.

Package Volume Optimization #

Chinese developers envy overseas applications because they have the unified Google Play Store. It can publish based on users’ ABI, density, and language, as well as the App Bundle introduced in 2018.

In fact, an installation package mainly consists of Dex, Resource, Assets, Library, and signature information. Next, let’s take a look at some advanced tips for domestic applications.

1. Code

For most applications, Dex is the largest part of the package volume. Looking at the data from WeChat, QQ, Alipay, and Taobao in the table above, the number of their Dex files increases from 1 to over 10. Did our code really increase so much?

Moreover, the number of Dex files also poses a significant challenge to the installation time for users. Without cutting off functionalities, let’s see how we can reduce this part of the space.

ProGuard - “Nine pitfalls in ten ProGuard configurations”, especially for various third-party SDKs. We need to carefully check the final merged ProGuard configuration file to see if there is excessive “keep”.

You can output the final ProGuard configuration using the following command, and pay attention to various “keep *” statements. In many cases, we only need to keep certain packages, certain methods, or class names.

-printconfiguration  configuration.txt

Are there any other methods to further increase the obfuscation power? In this case, we may need to target the four components and views. Generally, applications keep certain methods of the four components and views so that they can be referenced in code and XML layouts.

-keep public class * extends android.app.Activity
-keep public class * extends android.app.Application
-keep public class * extends android.app.Service
-keep public class * extends android.content.BroadcastReceiver
-keep public class * extends android.content.ContentProvider
-keep public class * extends android.view.View

In fact, we can completely obfuscate the non-exported four components and views. But we need to complete the following tasks:

  • XML Replacement. After code obfuscation, it is necessary to modify the names referenced in AndroidManifest and resource XML files.

  • Code Replacement. It is necessary to traverse other obfuscated code and modify the strings defined in variables or method bodies. It should be noted that computed class names cannot appear in the code because this will cause the replacement to fail.

    // Case 1: variables public String activityName = “com.sample.TestActivity”; // Case 2: method body startActivity(new Intent(this, “com.sample.TestActivity”)); // Case 3: Computed, not supported startActivity(new Intent(this, “com.sample” + “.TestActivity”));

For code replacement, I recommend using ASM. If you are not familiar with ASM, don’t worry, I will explain its principles and usage later. Eleme once open sourced a component, Mess, which can obfuscate the four components and views. However, it seems that it is no longer maintained, but you can refer to it.

Android Studio 3.0 introduced a new Dex compiler, D8, and a new obfuscation tool, R8. Currently, D8 has been officially released and can reduce the Dex volume by about 3%. However, R8, intended to replace ProGuard, is still in the experimental stage. We look forward to its better performance in the future.

Remove Debug Information or Line Numbers - For a certain application, a Debug package and a Release package are generated using the same ProGuard rules, where the Debug package is 4MB and the Release package is only 3.5MB.

Since their ProGuard obfuscation and optimization rules are the same, what is the difference between them? That is the DebugItem.

The DebugItem mainly consists of two types of information:

  • Debugging Information - Function parameter variables and all local variables.

  • Issue Tracking Information - Correspondence between instruction set line numbers and source file line numbers.

In fact, in the ProGuard configuration, we generally preserve line number information using the following method.

-keepattributes SourceFile, LineNumberTable

For a more detailed analysis of the removal of debug info and line numbers, I recommend carefully reading an article by Alipay, “Extreme Compression of Android Package Size”. Using this method, we can retain line numbers and reduce the Dex volume by approximately 5%.

In fact, Alipay referred to an open-source compilation tool developed by Facebook, ReDex. ReDex is an extremely hardcore open-source library in the client domain, which is worth studying carefully, although it lacks documentation.

ReDex contains a lot of great stuff, which we will discuss repeatedly later. Removing Debug Information is achieved by the StripDebugInfoPass.

{
  "redex" : {
    "passes" : [
      "StripDebugInfoPass"
    ]
  },
  "StripDebugInfoPass" : {
    "drop_all_dbg_info" : "0",     // Remove all debug information, 0 means not removing
    "drop_local_variables" : "1",  // Remove all local variables, 1 means removing
    "drop_line_numbers" : "0",     // Remove line numbers, 0 means not removing
    "drop_src_files" : "0",        
    "use_whitelist" : "0",
    "drop_prologue_end" : "1",
    "drop_epilogue_begin" : "1",
    "drop_all_dbg_info_if_empty" : "1"
  }
}

Dex Splitting - When we view an APK in Android Studio, do you know the difference between “defines 19,272 methods” and “references 40,229 methods” in the following figure?

To understand the format and definitions of each field in Dex, you can refer to the article 《Dex文件格式详解》. To deepen your understanding of Dex format, I recommend using 010 Editor.

“Define classes and methods” refers to the classes and their methods that are defined in the Dex. “Reference methods” refers to the methods that are defined as well as the methods referred to by the defined methods.

In simple terms, if Class A and Class B are compiled into different Dex files as shown in the following diagram, since method a calls method b, method b’s ID also needs to be added in classes2.dex.

What impact does this cross-Dex calling redundancy have on the size of our Dex?

  • Method ID overflow. As we know, each Dex’s method ID needs to be smaller than 65536. The substantial redundancy in method IDs reduces the number of classes that each Dex can actually accommodate, leading to an increase in the number of final compiled Dex files.

  • Redundant information. Because we need to record detailed information of methods that are called across Dex files, classes2.dex also needs to record the definitions of Class B and method b, resulting in redundancy in the string_ids, type_ids, and proto_ids sections.

In fact, I have defined a measure of Dex information efficiency, and I hope to ensure that Dex efficiency is above 80%. At the same time, in order to further reduce the number of Dex files, we hope that each Dex file has a full complement of methods, i.e. all 65536 methods are allocated.

Dex information efficiency = number of define methods / number of reference methods

So how do we improve Dex information efficiency? The key is that we need to allocate classes and methods with calling relationships to the same Dex file, in other words, reduce the occurrence of cross-Dex calls. However, as the calling relationships between classes are very complex, it is unlikely that we can calculate the optimal solution and can only obtain a local optimum.

To improve Dex information efficiency, I was once involved in writing a dependency analysis tool called Builder at WeChat. However, in the latest version 7.0 of WeChat, you can see that the number and size of Dex files in the table above have increased significantly, because they accidentally rendered this tool ineffective. The increase in the number of Dex files has a significant impact on Tinker’s hotfix time and user installation time. If this issue is fixed, the number of Dex files in WeChat version 7.0 should be reduced from 13 to about 6, and the package size can be reduced by around 10MB.

But when I was studying ReDex, I found that it also provides this optimization and implements it better than WeChat. After analyzing the class call relationships, ReDex uses a greedy algorithm to calculate the local optimum. The specific algorithm can be found in CrossDexDefMinimizer.

Why can’t we calculate the optimal solution? Because we need to find a balance between compilation speed and effectiveness. In ReDex, the configuration for this optimization is as follows:

{
  "redex" : {
    "passes" : [
      "InterDexPass"
    ]
  },
  "InterDexPass" : {
    "minimize_cross_dex_refs": true,
    "minimize_cross_dex_refs_method_ref_weight": 100,
    "minimize_cross_dex_refs_field_ref_weight": 90,
    "minimize_cross_dex_refs_type_ref_weight": 100,
    "minimize_cross_dex_refs_string_ref_weight": 90
  }
}

So, how much can you optimize the package size through Dex sub-packaging? Because the default Android sub-packaging method is not very good. If your application has more than 4 Dex files, I believe this optimization can achieve at least a 10% effect.

Dex Compression - I was amazed when reverse engineering the Facebook app and found that it only had a Dex file of around 700 KB. Since Google Play does not allow dynamic code delivery, where is the code stored?

In fact, the classes.dex file in the Facebook app is just a shell. The actual code is stored in the assets directory. They merge all the Dex files into a single secondary.dex.jar.xzs file and compress it using XZ compression.

The XZ compression algorithm, like 7-Zip, uses the LZMA algorithm internally. For Dex files, the compression ratio of XZ can be around 30% higher than that of Zip. However, you may have noticed that this solution seems to have some issues:

  • Decompression during initial startup. When the application is first started, the secondary.dex.jar.xzs file needs to be decompressed. According to the configuration information in the above image, there should be a total of 11 Dex files. Facebook uses a multi-threaded decompression method, which takes several hundred milliseconds on high-end devices and may require 3 to 5 seconds on low-end devices. Why not use Zstandard or Brotli? The main reason is the trade-off between compression ratio and decompression speed.

  • ODEX file generation. As mentioned earlier, when there are many Dex files, it increases the installation time of the application. With Facebook’s approach, the initial generation of ODEX may take minutes. To solve this problem, Facebook uses another hardcore method called oatmeal.

The principle of oatmeal is simple: it generates an ODEX file based on the format of an existing ODEX file. The result is the same as an interpreted ODEX file, without any machine code inside.

As shown in the above image, for the normal process, we need to fork a process to generate dex2oat, which usually takes a long time. With oatmeal, we can directly generate the ODEX file in the current process. For a 10 MB Dex file, the time it takes to generate an ODEX file is approximately 10 seconds or more on Android 5.0, about 1 second on Android 8.0 using the speed mode, and about 100 milliseconds using oatmeal.

I have always wanted to introduce oatmeal into Tinker, but I am concerned about compatibility issues because each version has some differences in the ODEX format, and oatmeal needs to be adapted accordingly.

2. Native Library

Nowadays, functions such as audio and video, beauty filters, AI, and VR are becoming more common in applications. However, these libraries are usually written in C or C++, which means that the size of Native Libraries in our APK is increasing.

For Native Libraries, traditional optimization methods may include removing debug information and using c++_shared. But do we have any better optimization methods?

Library Compression - Just like Dex compression, the most effective method for optimizing libraries is to use XZ or 7-Zip compression.

In the default lib directory, we only need to load a few libraries related to the startup process, and unpack the other libraries during the initial startup. For library files, the compression ratio can also be around 30% higher than that of Zip, which is very impressive.

Facebook has an open-source library for loading shared objects (So) called SoLoader, which can be used in conjunction with this solution. Just like Dex compression, the main drawback of the compression solution is the initial startup time. After all, for low-end devices, multi-threading is not very meaningful, so we need to strike a balance between package size and user experience.

Library merging and pruning - For Native Libraries, Facebook’s compilation and build tool Buck also has two hardcore techniques. Of course, you won’t find them in the official documentation; they are hidden in the source code.

  • Library merging. Before Android 4.3, there was a limitation on the number of libraries that could be loaded by a process. During the compilation process, we can automatically merge some libraries into one. You can learn more about the specific approach in the article Android native library merging and the demo.

  • Library pruning. Buck has a feature called relinker, which analyzes the JNI methods and method calls of different libraries in the code. It identifies unused export symbols and removes them. This also allows the linker to remove the corresponding unused code during compilation. This method is like implementing ProGuard Shrinking for libraries.

Package Size Monitoring #

When it comes to package size, if left unchecked, it can bring you a big “surprise” after a few versions. I have learned that some applications have very strict controls on package size, and any feature exceeding 100KB requires approval.

There are usually several types of monitoring for package size:

  • Size monitoring: This is very straightforward, comparing the package size of each version with the previous one. If the size increases significantly in a certain version, it is necessary to analyze the specific reasons and whether there is room for optimization.

  • Dependency monitoring: With every version, we need to monitor dependencies, including newly added JAR and AAR dependencies. This is because many developers are not careful enough and often accidentally introduce large open-source libraries.

  • Rule-based monitoring: If we find that the package size has increased significantly in a certain version, we need to analyze the reasons. Rule-based monitoring abstracts package size monitoring into rules, such as unused resources, large files, duplicate files, and R files. For example, when I was working at WeChat, I used ApkChecker to implement rule-based monitoring for package size.

It is best to automate and platformize the monitoring of package size as part of the release process. Otherwise, it is difficult to sustain it manually.

Summary #

Today, we analyzed methods for optimizing package size, which can be quite difficult to implement. Some may wonder if these methods are truly valuable considering the level of difficulty. From my understanding, we have now entered the realm of deep optimization for mobile applications. The generic articles found online are no longer sufficient to meet our needs. In other words, we have already mastered the simple methods and are already implementing them. We need to consider how to further optimize our applications.

At this point, we need to calm down, learn to think and explore, and delve deeper into the underlying layers. We should study the file format of APKs and further study the file formats of internal Dex, Library, and Resource files. Additionally, we should contemplate the entire compilation process in order to find areas where breakthroughs can be made.

When implementing AndResGuard, I conducted very thorough research on the resources.arsc format and the process of loading resources in Android. Over the past few years, are there any new tips for optimizing resources? In our next issue, we will discuss the topic of “resource optimization”.

From Buck and ReDex, it is evident that Facebook’s research is much more advanced than that in China. I hope they can provide some additional documentation to make our learning process easier.

Homework #

Does your application pay attention to package size? What package size optimization work have you done, and what good methods can you share with your classmates? Feel free to leave a comment to discuss with me and other classmates.

For today’s practice Sample, try using the ReDex project to optimize the package size of our application. There are several small tasks:

  • strip debuginfo

  • package optimization

You are welcome to click “Please Read with Friends” to share today’s content with your friends and invite them to learn together. Lastly, don’t forget to submit today’s homework in the comment section. I have prepared a generous “Study Booster Pack” for students who complete the homework diligently. Looking forward to improving together with you.