Appendix - Discussing the RPCs I have experienced #

Hello, I am He Xiaofeng. In the previous lesson, we learned how to support multiple RPC protocols in the production environment in order to smoothly upgrade the existing RPC framework and also to support different use cases using various protocols.

This marks the end of the technical content of our entire column, and I feel fortunate to have walked through these days together with you. During this time, we have crossed the New Year and also experienced the unexpected COVID-19 pandemic. Now, I realize that being able to breathe fresh air freely is also a kind of happiness. I wish you a safe passage through these difficult times and hope that we can take off our masks soon. To thank you for your company during these days, today let’s switch gears and talk about some lighthearted topics. I will share with you my experience with RPC frameworks.

Introduction to RPC #

When I graduated from university in 1998, the Internet was not as booming as it is today. At that time, most IT companies were focusing on the development of digital office-related software. Therefore, I followed the trend and joined a company that developed office software, which is what we often refer to as the “traditional software industry.”

Later on, with the rapid popularity of the Internet, every industry wanted to establish a connection with it, as everyone knew that the Internet represented the future. In China, the industries that developed most rapidly with the help of Internet technology were e-commerce and gaming.

Thanks to the popularity of the Internet, articles related to technology from various Internet companies were abundant. For example, Company A experienced prolonged website paralysis due to a surge in traffic, and Company B faced overwhelming database pressure due to continuous growth in orders. Whenever I read such technical articles, I couldn’t help but wonder when I would encounter problems that could be solved with technological optimization. Later on, I found it difficult to experience these “ideas” in the industry I was in at that time, so in 2011, I made a bold decision to join JD.com (also known as Jingdong).

At that time, our company happened to be in a period of rapid business development, but in terms of system stability, it was still relatively immature compared to today. In our daily work, our developers not only faced the pressure of meeting the demands of business requirements but also the pressure from the system due to the growth of online business. This dual pressure was a new experience for me, one that I had never encountered before, and it excited me.

I had previously worked on business system development for a long time. Although I had accumulated a lot of practical programming skills and had carefully studied many excellent open-source codes both domestically and internationally, including frameworks like Spring and Netty, I had never experienced complex interactions between so many systems. Therefore, in the early days of joining the company, I was always in a state of unease.

Fortunately, with the enthusiastic help of my colleagues, I quickly assimilated into the team atmosphere. Since our large team was responsible for the entire company’s infrastructure, you can simply understand that we were responsible for building the company’s PaaS (Platform as a Service) platform. Since it was a PaaS platform, it definitely involved middleware, and the most essential part of middleware should be RPC (Remote Procedure Call). Compared to other middleware, RPC had the highest usage frequency and was the foundation for building distributed systems.

From then on, I embarked on this “irreversible” road of RPC. Of course, it is not an endless road, but a road full of challenges that has never made me want to stop.

RPCs Used #

Since I have been working for a long time, my programming experience is relatively rich. Before switching to Java, I worked with .NET for a period of time. In the early days, because I mainly developed digital office software, I had more opportunities to use .NET programming. This was mainly because many applications used .NET, which had a much higher development efficiency compared to Java. However, as the applications became more complex, .NET was not as advantageous as Java in terms of performance and maintainability. In addition, for a long time, .NET did not support deployment in a Linux environment. Therefore, we gradually changed our applications to Java.

ICE #

However, there was a problem: How do we smoothly transition our .NET applications to Java? Originally, the communication between .NET applications relied on each other, but it needed to be changed to communicate between .NET and Java. To solve this problem, at that time we chose a relatively old RPC framework called ICE (https://zeroc.com). It’s possible that many people haven’t heard of it nowadays.

Hessian #

Later, there were more opportunities to develop applications using Java, and with the popularity of Spring development, using ICE for RPC communication between Java applications became less effective. So, we used a new RPC framework called Hessian (http://hessian.caucho.com/), which changed the way RPC calls were made between Java applications.

We chose Hessian because it integrated well with Spring. For developers working with Spring projects, it became easy to develop an RPC interface. We could directly expose a Java class as an RPC interface definition, without the need to define a stub like with ICE.

From the perspective of an RPC framework, Hessian is an excellent product, and even today, it has strong reference value in terms of performance and robustness. However, as the business grew to a certain extent, the communication between applications needed to consider more than just the RPC framework. It was more important to consider how to achieve service governance.

Another reason is that Hessian does not have service discovery functionality. We could only make calls through VIP exposure, and we needed to assign a VIP to each application and mount all instances of the same interface provider on the same VIP. This centralized traffic forwarding architecture put a lot of pressure on the VIP service provided by LVS. Furthermore, centralized traffic forwarding caused longer response times for the calling party.

Dubbo #

To solve the centralized problems similar to Hessian and implement large-scale application serviceification, the domestic RPC framework Dubbo played a more advanced role. As the business became more complex, the relationship between applications became more intricate. Therefore, we later chose to extend Dubbo to complete RPC communication and use a ZooKeeper cluster for service discovery. By combining these existing frameworks, we achieved rapid application serviceification and the goal of using a unified RPC framework for all applications within the company.

However, as the microservices concept became more popular, many application interfaces became finer and more granular. This led to an increasing number of interfaces that needed to be integrated into the ZooKeeper cluster. Also, because our business volume doubled each year, we needed to frequently scale up to handle sufficient call volume. As a result, the number of IP instances connected to the ZooKeeper cluster grew significantly, placing a heavy load on our ZooKeeper cluster.

Furthermore, Dubbo is relatively complex and has room for improvement in terms of performance. This forced us to consider new solutions.

In-House RPC #

Under these circumstances, we decided to develop a microservices solution tailored to our business scenario, including an RPC framework, service governance, and a multi-language solution.

Thus, our in-house RPC has continuously supported various business operations within the company.

In recent years, during the evolution of infrastructure represented by Kubernetes, an important keyword has been the sinking of application infrastructure capabilities. In the past, when we provided RPC capabilities to applications, it required the applications to import Jar packages. In RPC, we needed to integrate service discovery, routing, and the entire RPC solution into this Jar package.

Future #

Currently, Kubernetes has become the de facto standard for infrastructure. All the various infrastructure capabilities that were previously encapsulated in Jar packages have been moved from the application layer to the infrastructure layer by the Kubernetes project. The same applies to our RPC. We need to separate non-business functionalities from traditional RPC frameworks, sink them into the infrastructure, and integrate them into the infrastructure, and then connect applications and infrastructure through Mesh.

This is also the next stage of RPC development, to mesh all applications. Therefore, the road of RPC has no end, only continuous challenges and enjoyment. I hope you can also fall in love with it!

Lastly, I would like to share my thoughts on RPC.

When people talk about RPC, they may think that it is just a tool for inter-application communication. In essence, there is nothing wrong with this understanding. However, in reality, we need RPC to solve more practical problems, such as service governance. These issues need to be considered when using RPC. Therefore, I personally believe that RPC should be considered as a broader concept.

Of course, most of us may not have the opportunity to fully implement a new RPC solution right now. This is not only due to the issue of resources but also due to practical needs. So why do we still need to learn RPC well? My thoughts are simple and practical: RPC is the cornerstone of building distributed systems, just as we start learning a new programming language from “Hello World” every time. I hope you can establish a solid foundation in RPC, and one day you will experience its power!

That concludes today’s special broadcast. I look forward to hearing your story with RPC.