Appendix Detailed Explanation of Rpc Framework Code Instances

Appendix - Detailed explanation of RPC framework code instances #

Hello, I’m He Xiaofeng. It’s been a while since we last met! It has been some time since the end of our column, during which I and the editor, Dongqing, reviewed the entire course. We carefully read through the feedback from the end-of-course questionnaire, and the highest request we received was “want to see RPC code examples.” Today, I’m here to fulfill your expectations.

Do you remember that in the [Conclusion], I mentioned that before writing this column, I re-implemented the RPC framework that I was responsible for in my company. Actions speak louder than words, and now this RPC framework has been open-sourced, ready for your inspection.

In the following, I will provide a detailed analysis of this set of code. I hope it can help you connect the knowledge you have learned, practice in a real-world scenario, and gain something valuable.

Overall Structure of RPC Framework #

First, let’s talk about the overall architecture of our RPC framework. Please recall [Lesson 07]. In this lesson, I explained how to design a flexible RPC framework, with the key point being plug-inability. We can use a plug-in system to enhance the extensibility of RPC and make it a microkernel architecture, as shown in the following diagram:

Here, we can see that the RPC framework is divided into four layers: the entry layer, the cluster layer, the protocol layer, and the transport layer. Each of these layers contains a series of plug-ins, and in a practical RPC framework, there will be even more plug-ins. In the RPC framework I have open sourced, there are over 50 plug-ins, involving a considerable amount of code. Now, I will connect the core modules and classes of the RPC framework through the three main processes: Server Startup Process, Client Startup Process, and RPC Invocation Process. Understanding these three processes will greatly help you when reading the code.

Server Startup Process #

Before explaining the server startup process, let’s take a look at the example code for starting the server, as shown below:

public static void main(String[] args) throws Exception {

    DemoService demoService = new DemoServiceImpl(); // Service provider settings
    ProviderConfig<DemoService> providerConfig = new ProviderConfig<>();
    providerConfig.setServerConfig(new ServerConfig());
    providerConfig.setInterfaceClazz(DemoService.class.getName());
    providerConfig.setRef(demoService);
    providerConfig.setAlias("joyrpc-demo");
    providerConfig.setRegistry(new RegistryConfig("broadcast"));

    providerConfig.exportAndOpen().whenComplete((v, t) -> {
        if (t != null) {
            logger.error(t.getMessage(), t);
            System.exit(1);
        }
    });

    System.in.read();

}

As we can see, the providerConfig starts the server by calling the exportAndOpen() method. Why is it named like this?

Let’s take a look at the implementation of the exportAndOpen() method:

public CompletableFuture<Void> exportAndOpen() {

    CompletableFuture<Void> future = new CompletableFuture<>();
    export().whenComplete((v, t) -> {
        if (t != null) {
            future.completeExceptionally(t);
        } else {
            Futures.chain(open(), future);
        }
    });

    return future;

}

Here, the server startup process is divided into two parts: export (creating the Export object) and open (starting the server). The server startup process can also be divided into two parts: the server creation process and the server opening process.

Server Creation Process #

The ProviderConfig is the server’s configuration object, where interface, group, and registry configurations, etc., are all configured in this class. The entry point of the process is calling the export method of ProviderConfig. The entire process is as follows:

  1. Generate the registryUrl and serviceUrl based on the configuration information of ProviderConfig.
  2. Create the Registry object by calling the Registry plugin with registryUrl. The Registry object represents the registry and interacts with it.
  3. Call the open method of the Registry object to establish a connection with the registry.
  4. Call the subscribe method of the Registry object to subscribe to the configuration information and global configuration of the interface.
  5. Call the InvokerManager to create the Exporter object.
  6. InvokerManager returns the Exporter object.

The server creation process is actually the creation of the Exporter object. The Exporter object is a subclass of the Invoker interface, and the Invoker interface has two subclasses, namely Exporter and Refer. Exporter is used to handle requests received by the server, while Refer is used to send requests to the server. These two classes can be considered as the two most core classes in the entry layer.

When creating the Exporter object in the InvokerManager, there will be a series of operations, and initializing the Exporter will also involve a series of operations, such as creating a filter chain and authentication information, etc. The detailed explanation will not be provided here, you can read the source code for more information.

Server Opening Process #

After creating the Exporter object for the server, we need to open the Exporter object. The two most important operations for opening the Exporter object are to open the port of the Server in the transport layer to receive requests from the caller, and to register the server node to the registry so that the caller can discover this server node. The entire process is as follows:

  1. Call the open method of the Exporter object to start the server.
  2. The Exporter object calls the interface warmup plugin to perform interface warmup.
  3. The Exporter object calls the EndpointFactroy plugin in the transport layer to create a Server object. One Server object represents one port.
  4. Call the open method of the Server object to open the port. After the port is opened, the server can provide remote services.
  5. The Exporter object calls the register method of the Registry object to register this server node to the registry.

Whether it is the open method of Exporter, the open method of Server, or the register method of Registry, they are all asynchronous methods with a return value of CompletableFuture object. Each step of the process is also asynchronous.

The open operation of the Server is actually a complex operation, involving binding protocol adapters, initializing session managers, adding event bus event listeners, etc. Moreover, the entire process is completely asynchronous and plug-in based.

Invocation Process of the Invoker #

Before explaining the invoker’s invocation process, let’s take a look at an example of the invoker’s startup code:

public static void main(String[] args) {

    ConsumerConfig<DemoService> consumerConfig = new ConsumerConfig<>(); // Consumer configuration
    consumerConfig.setInterfaceClazz(DemoService.class.getName());
    consumerConfig.setAlias("joyrpc-demo");
    consumerConfig.setRegistry(new RegistryConfig("broadcast"));

    try {
        CompletableFuture<DemoService> future = consumerConfig.refer();
        DemoService service = future.get();

        String echo = service.sayHello("hello"); // Initiate service invocation
        logger.info("Get msg: {} ", echo);
    } catch (Throwable e) {
        logger.error(e.getMessage(), e);
    }

    System.in.read();

}

The entry point of the invoker’s invocation process is the refer method of the ConsumerConfig object. The ConsumerConfig object is the configuration object of the invoker. As you can see, the return value of the refer method is a CompletableFuture, just like the server side. The invoker’s startup process is also completely asynchronous. Now let’s take a look at the invoker’s invocation process.

The specific process of the invoker is as follows:

  1. Generate the registryUrl (URL object for the registry) and serviceUrl (URL object for the service) based on the configuration information of the ConsumerConfig.
  2. Call the Registry plugin with the registryUrl to create a Registry object, which represents the registry and interacts with it.
  3. Create a dynamic proxy object.
  4. Call the Open method of the Registry object to start the registry.
  5. Call the subscribe method of the Registry object to subscribe to the configuration information of the interface and the global configuration.
  6. Call the refer method of the InvokeManager to create a Refer object.
  7. Before creating the Refer object, the InvokeManager will first create a Cluster object. The Cluster object is the core object of the cluster layer and is responsible for maintaining the connection status between the invoker and the server nodes.
  8. Create a Refer object in the InvokeManager.
  9. Initialize the Refer object, including creating routing strategies, message distribution strategies, load balancing, call chains, adding event bus event listeners, and so on.
  10. The ConsumerConfig calls the open method of the Refer object to start the invoker.
  11. The Refer object calls the open method of the Cluster object to start the cluster.
  12. The Cluster object calls the subscribe method of the Registry object to subscribe to the changes of the server nodes. When receiving the changes, the Cluster will call the EndpointFactory plugin in the transport layer to create a Client object and establish a connection with these service nodes. The Cluster will maintain these connections.
  13. The ConsumerConfig encapsulates the Refer object into the ConsumerInvokerHandler and injects the ConsumerInvokerHandler object into the dynamic proxy object.

The most complex operations in the invoker’s startup process are the open operation of the Cluster object and the open operation of the Client object.

The Cluster object is the core object of the cluster layer and the most complex object in this RPC framework. It is responsible for maintaining the cluster information of the invoker, listening to the updates of the service nodes pushed by the registry, calling the EndpointFactory plugin in the transport layer to create Client objects, and establishing connections with the service nodes. The Cluster also sends negotiation information, security verification information, and heartbeat information to maintain the connection status with the service nodes through heartbeat mechanism.

The open operation of the Client object also involves a series of operations, such as creating a Transport object, creating a Channel object, generating and recording session information, and so on.

When constructing the call chain in the dynamic proxy object, the core logic internally is to invoke the Invoke method of the ConsumerInvokerHandler object, which ultimately invokes the Refer object. I will explain the RPC invocation process in detail below.

RPC Invocation Process #

After explaining the start-up process of the server and the start-up process of the client, now I will explain the RPC invocation process. The entire RPC invocation process consists of the client sending a request message and the server receiving and processing the request, and then responding back to the client.

Now I will explain the sending process on the client side and the receiving process on the server side.

Client Sending Process #

The client sending process is as follows:

  1. The dynamic proxy object calls the Invoke method of the ConsumerInvokerHandler object.
  2. The ConsumerInvokerHandler object generates a request message object.
  3. The ConsumerInvokerHandler object calls the Invoke method of the Refer object.
  4. The Refer object processes the request message object, such as setting interface information, group information, etc.
  5. The Refer object calls the message transmission plugin to handle the transmission information, which includes implicit parameter information.
  6. The Refer object calls the Invoker method of the FilterChain object to execute the invocation chain.
  7. The FilterChain object calls each Filter.
  8. The distribute method of the Refer object is the last Filter to be called in the invocation chain.
  9. The select method of the NodeSelecter object is called. The NodeSelecter is the router rule node selector of the cluster layer, and its select method is used to select service nodes that meet the routing rules.
  10. The route method of the Route object is called. The Route object is a routing dispatcher and also an object in the cluster layer. The default routing distribution strategy is Failover, which means that the request can be retried after a failure. You can review the question in the 12th lesson, where I asked about the phase in which exception retry is performed in the RPC invocation.
  11. The select method of the LoadBalance object is called by the Route object to select a node through load balancing.
  12. The invokeRemote method of the Refer object is called back by the Route object.
  13. The invokeRemote method of the Refer object calls the Client object in the transport layer to send the message to the server node.

In the client sending process, the message is finally sent to the server through the transport layer. The detailed operations in the transport layer are not explained here, but the internal process in the transport layer is complex, and there will be a series of operations, such as creating Future objects, managing Future objects using the FutureManager, request message protocol conversion processing, encoding and decoding, and timeout processing, etc.

Once the client sends the request message, the server will receive and process the request message. Now let’s look at the receiving process on the server side.

Server Receiving Process #

The transport layer of the server receives the request message, performs encoding and decoding, and deserialization on the request message, and then calls the invoke method of the Exporter object. The specific process is as follows:

  1. The transport layer receives the request, triggering the protocol adapter ProtocolAdapter.
  2. The ProtocolAdapter object traverses the implementation classes of the Protocol plugin to match the protocol.
  3. After matching the protocol, the Server object in the transport layer binds the codec (Codec object) and chain channel handler (ChainChannelHandler object) of the protocol.
  4. Decode and deserialize the received message.
  5. Execute the chain channel handler.
  6. Call the message handling chain (MessageHandle plugin) in the business thread pool.
  7. Call the handle method of the BizReqHandle object to process the request message.
  8. The BizReqHandle object calls the restore method to process the request message data based on the connection session information, and retrieves the Exporter object based on the interface name, group name, and method name of the request.
  9. The invoke method of the Exporter object is called, and the Exporter object returns a CompletableFuture object.
  10. The Exporter object calls the invoke method of the FilterChain.
  11. The FilterChain executes all Filter objects.
  12. The invokeMethod method of the Exporter object is the last Filter and is called last.
  13. The invokeMethod method of the Exporter object processes the request context and executes reflection.
  14. The Exporter object asynchronously notifies the BizReqHandle object of the request result obtained after reflection.
  15. The BizReqHandle object calls the Channel object in the transport layer to send the response result.
  16. The transport layer converts, serializes, encodes the response message, and finally transmits it back to the client through the network.

Summary #

Today, we analyzed the code of an open-source RPC framework, and connected the core modules and classes of the RPC framework through the three main processes: server startup process, client startup process, and RPC invocation process.

In the server startup process, the core task is to create and start the Exporter object. Before creating the Exporter object, the ProviderConfig first creates a Registry object, subscribes to interface configuration and global configuration from the registry center, and then creates the Exporter object. When the Exporter is started, a Server object is started to open a port. After the Exporter is successfully started, it will register with the registry center through the Registry object.

In the client startup process, the core task is to create and start the Refer object. The most complex part of opening the Refer object is the open operation of the Cluster, which is responsible for cluster management operations on the client side. This includes listening to registry center service node change events, establishing connections with server nodes, and managing the connection status of server nodes.

When the client initiates a call to the server, it goes through dynamic proxy first, and then calls the invoke method of the Refer object. The Refer object first processes the messages to be transmitted, and then executes the Filter chain. The last Filter on the client side selects a group of server nodes that meet the conditions based on the configured routing rules. After that, the route method of the Route object is called internally, and the route method selects a server node based on the configured load balancing strategy, and finally sends the request message to the selected server node.

The server-side transport layer receives the request message sent by the client, and after a series of processing (such as decoding, deserialization, protocol conversion, etc.), it handles the message in the business thread pool. The key logic is to call the invoke method of the Exporter object, which will execute the Filter chain configured on the server side. Finally, it executes the business logic through reflection or pre-compiled object, and encapsulates the final result into a response message and sends it back to the client through the transport layer.

In this lecture, we did not cover asynchronous calls when the client initiates a call to the server. In fact, the implementation logic of the invoke method in Refer object is completely asynchronous, and the invoke method in Exporter object is also asynchronous. The Refer and Exporter classes are both implementations of the Invoker interface on the client side. You can take a look at the definition of the invoke method in the Invoker interface:

/** * Invoke * * @param request Request * @return */ CompletableFuture invoke(RequestMessage request);

The JoyRPC framework is a purely asynchronous RPC framework, and the so-called synchronous behavior is just waiting for the asynchronous process to complete.

The core objects of the entry layer are the Exporter object and the Refer object, which handle most of the core logic of the entry layer.

The core objects of the cluster layer are the Cluster object and the Registry object. The internal logic of the Cluster object is also very complex, and the core logic is to interact with the Registry, subscribe to service node change events, and manage the connections established with server nodes. We did not go into much detail about the Cluster object here, you can check the code.

The core objects of the protocol layer are the various subclasses of the Protocol interface.

Next is the transport layer. We did not cover the specific implementation of the transport layer in this lecture, because it is difficult to explain it completely with limited content. It is recommended that you check the source code for a clear understanding. The transport layer is completely asynchronous and completely plugin-based. Its entry point is the EndpointFactory plugin. By obtaining an EndpointFactory object through the EndpointFactory plugin, which is a factory class that is used to create Client objects and Server objects.

For a complete RPC framework, today we only provided a rough explanation of the server startup process, the client startup process, and the RPC invocation process. The actual implementation is much more complex because it involves many detailed issues. However, once the main framework is understood, it should be very helpful. For more details, it is recommended that you read the source code yourself!

That’s all for today’s extra sharing. If you have any questions, feel free to leave a message and let’s communicate!