47 Configuration Center Design and Implementation Centralized Configuration and I Want Localization Too Above

47 Configuration Center Design and Implementation Centralized Configuration and I Want Localization Too Above #

Starting from version 2.7.0, Dubbo officially supports configuration center. The configuration center is also relied upon in the service introspection architecture to map Service ID to Service Name. The configuration center in Dubbo primarily serves two purposes:

  • Externalized configuration
  • Service governance, responsible for storing and notifying service governance rules.

One of the objectives of externalized configuration is to achieve centralized configuration management. There are already many mature professional configuration management systems available (such as Apollo developed by Ctrip and Nacos developed by Alibaba). The purpose of the Dubbo configuration center is not to reinvent the wheel, but to ensure that Dubbo can work properly with these mature configuration management systems.

Dubbo can support multiple configuration sources simultaneously. During the initialization process of Dubbo, configurations are obtained from multiple sources and integrated based on a fixed priority, achieving the effect of high priority configurations overriding low priority configurations. The aggregated result of these configurations will participate in forming the URL as well as the subsequent service publishing and service reference.

Dubbo currently supports the following four configuration sources, with priorities decreasing from 1 to 4:

  1. System Properties, i.e., -D parameters;
  2. Externalized configuration, which is the configuration center we will introduce in this lesson;
  3. Configurations received through programming methods such as API interface, annotations, XML configuration, which ultimately lead to objects like ServiceConfig and ReferenceConfig;
  4. Local dubbo.properties configuration file.

Configuration #

The Configuration interface is the basic interface for all configurations in Dubbo. It defines methods to obtain corresponding configuration values based on specified keys, as shown in the following diagram:

Drawing 0.png

Core methods of the Configuration interface

From the diagram above, we can see that the Configuration interface has corresponding get_() methods for different boolean, int, and String return types, as well as get_() methods with default values. These get_() methods first call the getInternalProperty() method to obtain the configuration value, and then call the convert() method to convert the obtained value into the return type before returning it. getInternalProperty() is an abstract method and is specifically implemented by subclasses of the Configuration interface.

The following diagram shows the implementation of the Configuration interface in Dubbo, including: SystemConfiguration, EnvironmentConfiguration, InmemoryConfiguration, PropertiesConfiguration, CompositeConfiguration, ConfigConfigurationAdapter, and DynamicConfiguration. We will now introduce their implementations one by one along with specific code examples.

Drawing 1.png

Inheritance diagram of Configuration

SystemConfiguration & EnvironmentConfiguration #

SystemConfiguration obtains the corresponding configuration item from Java Properties configuration (i.e., -D configuration parameters), while EnvironmentConfiguration obtains the corresponding configuration from environmental variables. The implementations of the getInternalProperty() methods are as follows:

public class SystemConfiguration implements Configuration {
    public Object getInternalProperty(String key) {
        return System.getProperty(key); // Read -D configuration parameters
    }
}
public class EnvironmentConfiguration implements Configuration {
    public Object getInternalProperty(String key) {
        String value = System.getenv(key);
        if (StringUtils.isEmpty(value)) {
            // Obtain corresponding configuration from environmental variables
            value = System.getenv(StringUtils.toOSStyleKey(key));
        }
        return value;
    }
}

InmemoryConfiguration #

InmemoryConfiguration maintains a Map collection (store field) in memory, and its getInternalProperty() method obtains the corresponding configuration value from the store collection:

public class InmemoryConfiguration implements Configuration {
    private Map<String, String> store = new LinkedHashMap<>();
    @Override
    public Object getInternalProperty(String key) {
        return store.get(key);
    }
    // Omitted methods to write to the store collection, like addProperty()
}

PropertiesConfiguration #

PropertiesConfiguration involves OrderedPropertiesProvider, which is defined as follows:

@SPI
public interface OrderedPropertiesProvider {
    // Used for sorting
    int priority();
    // Obtain Properties configuration
    Properties initProperties();
}

In the constructor of PropertiesConfiguration, all extension implementations of the OrderedPropertiesProvider interface are loaded and sorted according to the priority() method. Then, the default dubbo.properties.file configuration file is loaded. Finally, configurations provided by OrderedPropertiesProvider are used to override configurations from the dubbo.properties.file. The implementation of the constructor of PropertiesConfiguration is as follows:

public PropertiesConfiguration() {
    // Get all extension names of the OrderedPropertiesProvider interface
    ExtensionLoader<OrderedPropertiesProvider> propertiesProviderExtensionLoader = ExtensionLoader.getExtensionLoader(OrderedPropertiesProvider.class);
    Set<String> propertiesProviderNames = propertiesProviderExtensionLoader.getSupportedExtensions();
    if (propertiesProviderNames == null || propertiesProviderNames.isEmpty()) {

return;
}
// Load all extensions of OrderedPropertiesProvider interface
List<OrderedPropertiesProvider> orderedPropertiesProviders = new ArrayList<>();
for (String propertiesProviderName : propertiesProviderNames) {
    orderedPropertiesProviders.add(propertiesProviderExtensionLoader.getExtension(propertiesProviderName));
}
// Sort the extensions of OrderedPropertiesProvider interface
orderedPropertiesProviders.sort((OrderedPropertiesProvider a, OrderedPropertiesProvider b) -> {
    return b.priority() - a.priority();
});
// Load the default dubbo.properties.file configuration file and store the result in ConfigUtils.PROPERTIES
Properties properties = ConfigUtils.getProperties();
// Use the extensions of OrderedPropertiesProvider to override the default configuration in the dubbo.properties.file
for (OrderedPropertiesProvider orderedPropertiesProvider :
        orderedPropertiesProviders) {
    properties.putAll(orderedPropertiesProvider.initProperties());
}
// Update ConfigUtils.PROPERTIES field
ConfigUtils.setProperties(properties);

In the PropertiesConfiguration.getInternalProperty() method, the overridden configuration information is directly obtained from the ConfigUtils.PROPERTIES Properties object.

public Object getInternalProperty(String key) {
    return ConfigUtils.getProperty(key);
}

CompositeConfiguration #

CompositeConfiguration is a composite Configuration object, which combines multiple Configuration objects together and appears as a single Configuration object externally.

The CompositeConfiguration saves the combined Configuration objects in the configList field (a LinkedList<Configuration> collection). The CompositeConfiguration provides the addConfiguration() method to add Configuration objects to the configList collection. Below is the relevant code in the addConfiguration() method:

public void addConfiguration(Configuration configuration) {
    if (configList.contains(configuration)) {
        return; // Do not add the same Configuration object repeatedly
    }
    this.configList.add(configuration);
}

In the CompositeConfiguration, a prefix field and an id field are maintained. Both can be used as a prefix to query for a Key. The relevant code in the getProperty() method is as follows:

public Object getProperty(String key, Object defaultValue) {
    Object value = null;
    if (StringUtils.isNotEmpty(prefix)) { // Check the prefix
        if (StringUtils.isNotEmpty(id)) { // Check the id
            // Both prefix and id are used as prefixes, then concatenate the key for querying
            value = getInternalProperty(prefix + id + "." + key);
        }
        if (value == null) {
            // Only use prefix as a prefix, then concatenate the key for querying
            value = getInternalProperty(prefix + key);
        }
    } else {
        // If prefix is empty, query directly using the key
        value = getInternalProperty(key);
    }
    return value != null ? value : defaultValue;
}

In the getInternalProperty() method, all Configuration objects in the configList collection will be iterated in order, and the corresponding Value for the queried Key will be returned as the first successful match. Here is an example of the code:

public Object getInternalProperty(String key) {
    Configuration firstMatchingConfiguration = null;
    for (Configuration config : configList) { // Iterate all Configuration objects
        try {
            if (config.containsKey(key)) { // Get the first Configuration object that contains the specified Key
                firstMatchingConfiguration = config; 
                break;
            }
        } catch (Exception e) {
            logger.error("...");
        }
    }
    if (firstMatchingConfiguration != null) { // Query the Key and return the configuration value through this Configuration
        return firstMatchingConfiguration.getProperty(key);
    } else {
        return null;
    }
}

ConfigConfigurationAdapter #

Dubbo abstracts the configuration corresponding to each instance through the AbstractConfig class, as shown in the following diagram:

Drawing 2.png

These AbstractConfig implementations correspond to specific configurations and define the associated fields and getter/setter() methods. For example, the RegistryConfig implementation corresponds to the configuration of the registry center, which includes a series of fields related to the registry center, such as address, protocol, port, timeout, and corresponding getter/setter() methods to receive registry center configurations provided by users through XML, Annotation, or API. ConfigConfigurationAdapter is an adapter between AbstractConfig and Configuration, which converts an AbstractConfig object into a Configuration object. In the constructor of ConfigConfigurationAdapter, all fields of the AbstractConfig object are obtained and converted into a Map collection, which will be referenced by the metaData field of ConfigConfigurationAdapter. The relevant example code is as follows:

public ConfigConfigurationAdapter(AbstractConfig config) {
    // Get all fields and their values from the AbstractConfig object
    Map<String, String> configMetadata = config.getMetaData();
    metaData = new HashMap<>(configMetadata.size());
    // Modify the names of keys in the metaData collection based on the prefix and id configured in AbstractConfig
    for (Map.Entry<String, String> entry : configMetadata.entrySet()) {
        String prefix = config.getPrefix().endsWith(".") ? config.getPrefix() : config.getPrefix() + ".";
        String id = StringUtils.isEmpty(config.getId()) ? "" : config.getId() + ".";
        metaData.put(prefix + id + entry.getKey(), entry.getValue());
    }
}

In the implementation of the getInternalProperty() method in ConfigConfigurationAdapter, the configuration value is obtained directly from the metaData collection, as shown below:

public Object getInternalProperty(String key) {
    return metaData.get(key);
}

DynamicConfiguration #

DynamicConfiguration is an abstraction of dynamic configuration in Dubbo, and it has the following core methods:

  • getProperties()/ getConfig() / getProperty() methods: Get the specified configuration from the configuration center. A timeout can be specified when using these methods.
  • addListener()/ removeListener() methods: Add or remove a listener for the specified configuration.
  • publishConfig() method: Publish a configuration.

Each of the three types of methods mentioned above has multiple overloads, and the overloads include a group parameter, which means the configurations in the configuration center can be grouped by group.

Similar to many other interfaces in Dubbo, the DynamicConfiguration interface itself is not annotated with @SPI (i.e., it is not an extension interface), but it is annotated with @SPI on DynamicConfigurationFactory to make it an extension interface.

The getDynamicConfiguration() static method in DynamicConfiguration provides a way to parse the protocol type from the connectionURL parameter and get the corresponding DynamicConfigurationFactory implementation, as shown below:

static DynamicConfiguration getDynamicConfiguration(URL connectionURL) {
    String protocol = connectionURL.getProtocol();
    DynamicConfigurationFactory factory = getDynamicConfigurationFactory(protocol);
    return factory.getDynamicConfiguration(connectionURL);
}

The definition of the DynamicConfigurationFactory interface is as follows:

@SPI("nop") 
public interface DynamicConfigurationFactory {
    DynamicConfiguration getDynamicConfiguration(URL url);
    static DynamicConfigurationFactory getDynamicConfigurationFactory(String name) {
        // Get the DynamicConfigurationFactory implementation based on the extension name
        Class<DynamicConfigurationFactory> factoryClass = DynamicConfigurationFactory.class;
        ExtensionLoader<DynamicConfigurationFactory> loader = getExtensionLoader(factoryClass);
        return loader.getOrDefaultExtension(name);
    }
}

The inheritance relationships of the DynamicConfigurationFactory interface and the DynamicConfiguration interface are as follows:

11.png

Inheritance relationships of DynamicConfigurationFactory

Drawing 4.png

Inheritance relationships of DynamicConfiguration

Let’s first look at the implementation of AbstractDynamicConfigurationFactory. It maintains a dynamicConfigurations collection (Map type), and in the getDynamicConfiguration() method, this collection is filled to achieve the caching effect of DynamicConfiguration objects. Additionally, AbstractDynamicConfigurationFactory provides a createDynamicConfiguration() method for subclasses to implement and create DynamicConfiguration objects.

Taking the ZookeeperDynamicConfigurationFactory implementation as an example, createDynamicConfiguration() method creates a ZookeeperDynamicConfiguration object:

protected DynamicConfiguration createDynamicConfiguration(URL url) {
    // The ZookeeperTransporter used to create ZookeeperDynamicConfiguration is the implementation for Zookeeper mentioned in the Transport layer in the previous article
    return new ZookeeperDynamicConfiguration(url, zookeeperTransporter);
}

Next, let’s analyze the specific implementation of the DynamicConfiguration interface with ZookeeperDynamicConfiguration as an example.

First, let’s look at the core fields of ZookeeperDynamicConfiguration.

  • executor (Executor type): Used to execute the listeners in a thread pool.
  • rootPath (String type): When Zookeeper is used as the configuration center, configurations are also stored as ZNodes. The rootPath records the root path of all configuration nodes.
  • zkClient (ZookeeperClient type): The client used to interact with the Zookeeper cluster.
  • initializedLatch (CountDownLatch type): Blocks and waits for the registration of listeners related to ZookeeperDynamicConfiguration to be completed.
  • cacheListener (CacheListener type): A listener used to listen for configuration changes.
  • url (URL type): The URL object corresponding to the configuration center.

In the constructor of ZookeeperDynamicConfiguration, these core fields are initialized. The specific implementation is as follows:

ZookeeperDynamicConfiguration(URL url, ZookeeperTransporter zookeeperTransporter) {

this.url = url;
// Determine the root path of the configuration center ZNode based on the config.namespace parameter in the URL (default value is dubbo)
rootPath = PATH_SEPARATOR + url.getParameter(CONFIG_NAMESPACE_KEY, DEFAULT_GROUP) + "/config";
// Initialize initializedLatch and cacheListener
// After the cacheListener is successfully registered, the cacheListener.countDown() method will be called
initializedLatch = new CountDownLatch(1);
this.cacheListener = new CacheListener(rootPath, initializedLatch);
// Initialize the executor field for executing listener logic
this.executor = Executors.newFixedThreadPool(1, new NamedThreadFactory(this.getClass().getSimpleName(), true));
// Initialize the Zookeeper client
zkClient = zookeeperTransporter.connect(url);
// Register cacheListener as a listener on the rootPath
zkClient.addDataListener(rootPath, cacheListener, executor);
try {
    // Get the maximum duration for the current thread to block and wait for the Zookeeper listener to be registered successfully from the URL
    long timeout = url.getParameter("init.timeout", 5000);
    // Block the current thread and wait for the listener to be registered
    boolean isCountDown = this.initializedLatch.await(timeout, TimeUnit.MILLISECONDS);
    if (!isCountDown) {
        throw new IllegalStateException("...");
    }
} catch (InterruptedException e) {
    logger.warn("...");
}