15 ZooKeeper Registration Center Implementation Official Recommendation #

Dubbo supports ZooKeeper as a registry service, which is also the recommended registry for Dubbo. In order to better understand the application of ZooKeeper in Dubbo, let’s briefly review ZooKeeper first.

Dubbo itself is a distributed RPC open-source framework, and each service node that depends on Dubbo is deployed separately. In order for the provider and consumer to be able to obtain each other’s information in real time, they rely on a consistent service discovery component for registration and subscription. Dubbo can integrate with various service discovery components, such as ZooKeeper, etcd, Consul, Eureka, etc. Among them, Dubbo particularly recommends using ZooKeeper.

ZooKeeper is a high-availability and consistent open-source coordination service designed for distributed applications. It is a tree-structured directory service that supports change push, making it very suitable for applications in production environments.

The following is an image from the official Dubbo documentation, showing the node hierarchy of Dubbo in ZooKeeper:

Dubbo data stored in ZooKeeper

The “dubbo” node in the picture is the root node of Dubbo in ZooKeeper. “dubbo” is the default name of this root node, but it can also be modified through configuration.

The nodes at the “Service” layer in the picture are named after the full name of the service interface. For example, in the demo example, the name of this node is “org.apache.dubbo.demo.DemoService”.

The nodes at the “Type” layer in the picture are the categories of URLs, including providers (service provider list), consumers (service consumer list), routes (routing rule list), and configurations (configuration rule list).

According to different Type nodes, the nodes in the URL layer in the picture include Provider URL, Consumer URL, Routes URL, and Configurations URL.

ZookeeperRegistryFactory #

In the previous Lesson 13, when introducing the core concepts of Dubbo registry center, we explained the RegistryFactory interface and its subclass AbstractRegistryFactory. AbstractRegistryFactory only provides the functionality of caching Registry objects, and does not actually implement the creation of Registry. The specific creation logic is completed by subclasses. In the SPI configuration file of the dubbo-registry-zookeeper module (located in the directory shown in the figure below), the implementation class of RegistryFactory - ZookeeperRegistryFactory is specified.

Location of SPI configuration file for RegistryFactory

ZookeeperRegistryFactory implements AbstractRegistryFactory, and its createRegistry() method creates a ZookeeperRegistry instance, which will complete the interaction with ZooKeeper.

In addition, ZookeeperRegistryFactory also provides a setZookeeperTransporter() method. You can review the Dubbo SPI mechanism we introduced earlier, which will be automatically loaded through SPI or Spring IoC.

ZookeeperTransporter #

The dubbo-remoting-zookeeper module is a sub-module of the dubbo-remoting module, but it does not depend on other modules in dubbo-remoting, so we can directly introduce this module here.

Simply put, the dubbo-remoting-zookeeper module encapsulates a set of ZooKeeper clients based on Apache Curator, integrating the interaction with ZooKeeper into the Dubbo system.

The dubbo-remoting-zookeeper module has two core interfaces: ZookeeperTransporter and ZookeeperClient.

ZookeeperTransporter is only responsible for one thing, which is to create ZookeeperClient objects.

@SPI("curator")
public interface ZookeeperTransporter {
    @Adaptive({Constants.CLIENT_KEY, Constants.TRANSPORTER_KEY})
    ZookeeperClient connect(URL url);
}

We can see from the code that the ZookeeperTransporter interface is annotated with @SPI and becomes an extension point. The default implementation selected by @SPI annotation is the one with the extension name "curator". The connect() method is used to create an instance of ZookeeperClient (this method is annotated with @Adaptive, so we can override the default extension name specified by @SPI annotation through the client or transporter parameter in the URL).

![Drawing 2.png](../images/CgqCHl9ga2CAVhNZAACNo2yx1q4384.png)

Following the analysis of the Registry component, as an abstract implementation, AbstractZookeeperTransporter definitely has some additional enhancement features other than creating ZookeeperClient, which are then inherited by its subclasses. Otherwise, it would be sufficient for CuratorZookeeperTransporter to directly implement the ZookeeperTransporter interface to create and return an instance of ZookeeperClient, without adding another layer of abstract class in the inheritance relationship.

    public class CuratorZookeeperTransporter extends 
            AbstractZookeeperTransporter {

        // create ZookeeperClient instance
        public ZookeeperClient createZookeeperClient(URL url) {
            return new CuratorZookeeperClient(url); 
        }
    }

The core functions of AbstractZookeeperTransporter are as follows:

- Caching ZookeeperClient instances
- Switching to backup Zookeeper address when a Zookeeper node cannot be connected

When configuring the Zookeeper address, we can configure multiple addresses for Zookeeper nodes. In this way, when a Zookeeper node goes down, Dubbo can actively switch to other Zookeeper nodes. For example, we provide the following URL configuration:

    zookeeper://127.0.0.1:2181/org.apache.dubbo.registry.RegistryService?backup=127.0.0.1:8989,127.0.0.1:9999

The connect() method of AbstractZookeeperTransporter will first get the addresses of three Zookeeper nodes, 127.0.0.1:2181, 127.0.0.1:8989, and 127.0.0.1:9999, as configured in the above URL, and then look for an available ZookeeperClient instance from the ZookeeperClientMap cache (which is a Map, with the Zookeeper node address as the key and the corresponding ZookeeperClient instance as the value). If the search is successful, the ZookeeperClient instance is reused; if the search fails, a new ZookeeperClient instance is created, returned, and the ZookeeperClientMap cache is updated.

Once the ZookeeperClient instance is connected to the Zookeeper cluster, it can understand the topology of the entire Zookeeper cluster. If a Zookeeper node goes down later, the fault tolerance is completed by the Zookeeper cluster itself and Apache Curator.

### ZookeeperClient

As the name suggests, the ZookeeperClient interface is a Zookeeper client encapsulated by Dubbo. This interface defines a large number of methods for interacting with Zookeeper.

- create() method: creates a ZNode node, and also provides overloaded methods for creating temporary ZNode nodes.
- getChildren() method: gets the collection of child nodes for a specified node.
- getContent() method: gets the content stored in a node.
- delete() method: deletes a node.
- add*Listener() / remove*Listener() methods: adds/removes listeners.
- close() method: closes the current ZookeeperClient instance.

**AbstractZookeeperClient, as the abstract implementation of the ZookeeperClient interface**, mainly provides the following capabilities:

• Caches persistent ZNode nodes created by the current ZookeeperClient instance.
• Manages various listeners added by the current ZookeeperClient instance.
• Manages the running status of the current ZookeeperClient.

Let’s take a look at the core fields of AbstractZookeeperClient. First, the persistentExistNodePath field of type ConcurrentHashSet caches the paths of persistent ZNode nodes created by the current ZookeeperClient. Before creating a ZNode node, this cache is checked instead of interacting with Zookeeper to determine if the persistent ZNode node exists, reducing one interaction with Zookeeper.

dubbo-remoting-zookeeper provides three types of listeners to the outside world: StateListener, DataListener, and ChildListener.

• StateListener: Mainly responsible for listening to the connection status between Dubbo and the Zookeeper cluster, including SESSION_LOST, CONNECTED, RECONNECTED, SUSPENDED, and NEW_SESSION_CREATED.

• DataListener: Mainly listens to data changes stored in a certain node.

• ChildListener: Mainly listens to changes in the children nodes under a certain ZNode.

In AbstractZookeeperClient, the stateListeners, listeners, and childListeners sets are maintained, managing the three types of listeners mentioned above, respectively. Although the contents of the listeners are different, their management methods are similar. Therefore, we will only analyze the operations on the listeners set:

public void addDataListener(String path, 

      DataListener listener, Executor executor) {

    // Get the DataListener set on the specified path

    ConcurrentMap<DataListener, TargetDataListener> dataListenerMap = 

      listeners.computeIfAbsent(path, k -> new ConcurrentHashMap<>());

    // Query the TargetDataListener associated with the DataListener

    TargetDataListener targetListener = 

        dataListenerMap.computeIfAbsent(listener, 

            k -> createTargetDataListener(path, k));

    // Add a listener on the specified path through TargetDataListener

    addTargetDataListener(path, targetListener, executor);

}

The createTargetDataListener() method and addTargetDataListener() method here are abstract methods implemented by the AbstractZookeeperClient subclass. TargetDataListener is a generic type marked in AbstractZookeeperClient.

Why does AbstractZookeeperClient use generics? This is because different implementations of ZookeeperClient may depend on different Zookeeper client components, and the implementations of listener in different Zookeeper client components may also be different. However, the listening interfaces exposed by the entire dubbo-remoting-zookeeper module are unified, which are the three types mentioned above. Therefore, a layer of conversion is needed to decouple them, and this decoupling is achieved through TargetDataListener.

Although Dubbo 2.7.7 only supports Curator, in the source code of Dubbo 2.6.5, we can see that there is also an implementation using ZkClient for ZookeeperClient.

In the latest version of Dubbo, CuratorZookeeperClient is the only implementation class of AbstractZookeeperClient. In its constructor, the Curator client is initialized and blocked to wait for a successful connection:

public CuratorZookeeperClient(URL url) {

    super(url);

    int timeout = url.getParameter("timeout", 5000);

    int sessionExpireMs = url.getParameter("zk.session.expire", 

        60000);

    CuratorFrameworkFactory.Builder builder = 

        CuratorFrameworkFactory.builder()

            .connectString(url.getBackupAddress())//zk address (including backup address)

            .retryPolicy(new RetryNTimes(1, 1000)) // retry configuration

            .connectionTimeoutMs(timeout) // connection timeout duration

            .sessionTimeoutMs(sessionExpireMs); // session expiration time

    ... // omit the logic of handling authentication

    client = builder.build();

    // Add connection status listener

    client.getConnectionStateListenable().addListener(

          new CuratorConnectionStateListener(url));

    client.start();

    boolean connected = client.blockUntilConnected(timeout, 

        TimeUnit.MILLISECONDS);

    ... // Check the returned value connected; if the connection fails, throw an exception

}

All operations of CuratorZookeeperClient interacting with Zookeeper revolve around this Apache Curator client. The specific usage of Apache Curator has been introduced in lessons 6 and 7, so it will not be repeated here.

The inner class CuratorWatcherImpl is the generic class specified when implementing CuratorZookeeperClient as AbstractZookeeperClient. It implements the TreeCacheListener interface and can be added to the TreeCache to listen for changes in its own nodes and child nodes. In the implementation of the childEvent() method, when the tree structure monitored by TreeCache changes, the triggering path, node content, and event type are passed to the associated DataListener instance for callback.

        ChildListener zkListener = listeners.computeIfAbsent(listener, k -> (parentPath, currentChilds) -> {
    
            zkClients.forEach(zkClient -> {
    
                // 当节点下的子节点变更，该方法会被触发
    
                ZookeeperRegistry.this.notify(url, listener, toUrlsWithEmpty(url, parentPath, currentChilds));
    
            });
    
        });
    
        // 添加 ChildListener 监听器
    
        zkClient.addChildListener(path, zkListener);
    
        // 这里需要注意的是，如果节点已存在，但是节点下没有子节点
    
        // 例如：dubbo://192.168.0.101:20880/com.alibaba.dubbo.demo.DemoService
    
        // 那么 ZookeeperRegistry 将创建一个持久 ZNode 节点，节点数据为空
    
        List<String> children = zkClient.addChildListener(path, zkListener);
    
        // 将订阅到的 URL 添加到集合中
    
        urls.addAll(toUrlsWithEmpty(url, path, children));
    
    }

另一个分支是处理：订阅的 URL 中没有明确指定 Service 层接口的订阅请求。该分支会监听所有与 Consumer 相关的 Category 路径，并在每个 Category 路径上添加 ChildListener 监听器。下面是这个分支的核心源码分析：

// 构建通配符path：/dubbo/com.alibaba.dubbo.demo.DemoService

String wildcardPath = toRootPath() + Constants.PATH_SEPARATOR + ANY_VALUE;

// 获取所有通配符path下的节点数据

List<String> wildcardChildren = zkClient.addChildListener(wildcardPath, (parentPath, currentChilds) -> {

    // 注意：这里的逻辑只有在节点发生变化的时候才会触发

    // 这里需要根据 child 和 parent 两个路径来确定需要订阅哪个 URL

    for (String child : currentChilds) {

        // 如果是 URL 类型路径，则直接解析

        if (child.contains(Constants.PROVIDERS_CATEGORY) 
            || child.contains(Constants.CONSUMERS_CATEGORY) 
            || child.contains(Constants.ROUTERS_CATEGORY) 
            || child.contains(Constants.CONFIGURATORS_CATEGORY)) {

            List<String> children = zkClient.addChildListener(path, zkListener);

            // 将订阅到的 URL 添加到集合中

            urls.addAll(toUrlsWithEmpty(url, path, children));

        }

    }

});

// 将订阅到的 URL 添加到集合中

urls.addAll(toUrlsWithEmpty(url, wildcardPath, wildcardChildren));

上述源码翻译如下：

// 构建通配符path：/dubbo/com.alibaba.dubbo.demo.DemoService

String wildcardPath = toRootPath() + Constants.PATH_SEPARATOR + ANY_VALUE;

// 获取所有通配符path下的节点数据

List<String> wildcardChildren = zkClient.addChildListener(wildcardPath, (parentPath, currentChilds) -> {

    // 注意：这里的逻辑只有在节点发生变化的时候才会触发

    // 这里需要根据 child 和 parent 两个路径来确定需要订阅哪个 URL

    for (String child : currentChilds) {

        // 如果是 URL 类型路径，则直接解析

        if (child.contains(Constants.PROVIDERS_CATEGORY) 
            || child.contains(Constants.CONSUMERS_CATEGORY) 
            || child.contains(Constants.ROUTERS_CATEGORY) 
            || child.contains(Constants.CONFIGURATORS_CATEGORY)) {

            List<String> children = zkClient.addChildListener(path, zkListener);

            // 将订阅到的 URL 添加到集合中

            urls.addAll(toUrlsWithEmpty(url, path, children));

        }

    }

});

// 将订阅到的 URL 添加到集合中

urls.addAll(toUrlsWithEmpty(url, wildcardPath, wildcardChildren));

经过点到点注册中心以及 Zookeeper 注册中心的源码分析，相信大家对 Dubbo 使用 Zookeeper 注册中心的实现细节有了一个更深入的了解。接下来我们将有一个课后作业和大家一起来分析下 Dubbo 的注册策略实现。敬请期待！

// ZookeeperRegistry.notify() method, which will callback the current NotifyListener

ChildListener zkListener = listeners.computeIfAbsent(listener, 

  k -> (parentPath, currentChilds) -> 

      ZookeeperRegistry.this.notify(url, k, 

          toUrlsWithEmpty(url, parentPath, currentChilds)));

// Try to create a persistent node, mainly to ensure that the current path exists on Zookeeper

zkClient.create(path, false);

// This ChildListener will be added to multiple paths

List<String> children = zkClient.addChildListener(path, 

    zkListener);

if (children != null) {

    // If there is no registered Provider, the toUrlsWithEmpty() method will return a URL with the empty protocol

    urls.addAll(toUrlsWithEmpty(url, path, children));

}

}

// When subscribing for the first time, notify() method is called proactively to notify NotifyListener to process the current registered data such as URLs

notify(url, listener, urls);



The other branch of the doSubscribe() method handles subscription requests to listen to all Service layer nodes. For example, the Monitor will send out this kind of subscription request because it needs to monitor changes in all Service nodes. The processing logic of this branch is to add a ChildListener listener to the root node. When a Service layer node appears, this ChildListener will be triggered, and it will retrigger the doSubscribe() method to execute the logic of the previous branch (the subscription branch for the specific Service layer interface analyzed earlier).

Here is an analysis of the core code for this branch:



String root = toRootPath(); // Get the root node

// Get the ChildListener corresponding to NotifyListener

ConcurrentMap<NotifyListener, ChildListener> listeners = 

    zkListeners.computeIfAbsent(url, k -> new ConcurrentHashMap<>());

ChildListener zkListener = listeners.computeIfAbsent(listener, k -> 

  (parentPath, currentChilds) -> {

    for (String child : currentChilds) {

        child = URL.decode(child);

        if (!anyServices.contains(child)) {

            anyServices.add(child); // Record that the node has been subscribed

            // This ChildListener triggers the subscription for specific Service nodes

            subscribe(url.setPath(child).addParameters("interface", 

                child, "check", String.valueOf(false)), k);

        }

    }

});

zkClient.create(root, false); // Ensure that the root node exists

// When subscribing for the first time, process the current existing Service layer nodes

List<String> services = zkClient.addChildListener(root, zkListener);

if (CollectionUtils.isNotEmpty(services)) {

    for (String service : services) {

        service = URL.decode(service);

        anyServices.add(service);

        subscribe(url.setPath(service).addParameters(INTERFACE_KEY,

           service, "check", String.valueOf(false)), listener);

    }

}



The doUnsubscribe() method provided by ZookeeperRegistry will remove the ChildListener corresponding to the URL and NotifyListener from the relevant path, thus no longer listening to that path.

### Summary

In this lesson, we focused on the core implementation of Dubbo connecting to Zookeeper as the registry center.

First, we quickly reviewed the basics of Zookeeper, as well as the specific content stored in Zookeeper when it is used as the Dubbo registry center. Then we introduced the implementation of ZookeeperRegistryFactory, the RegistryFactory implementation for Zookeeper.

Next, we explained the component implementations used when Dubbo connects to Zookeeper, with a focus on the ZookeeperTransporter and ZookeeperClient implementations. They rely on Apache Curator and Zookeeper to complete the interaction at the underlying level.

Finally, we also explained how ZookeeperRegistry accesses Zookeeper through the ZookeeperClient to implement the functionality of the Registry.

If you have any questions or ideas about this lesson, please feel free to leave a message and share with me.