11 Code Governance How to Implement Dynamic Configuration Management in a Distributed Environment

11 Code Governance How to Implement Dynamic Configuration Management in a Distributed Environment #

With the continuous development of distributed systems and microservice architecture, unified governance of various services and resources in the system has become a fundamental point in system architecture design. ShardingSphere, as a distributed database middleware, also integrates configuration governance features.

Today’s content revolves around how to use the configuration governance feature provided by ShardingSphere. The course ideas are consistent with the previous lesson, that is, first discuss the abstraction process of configuration governance in ShardingSphere, and then provide a system transformation plan based on the configuration center to introduce the integrated configuration governance feature in the development process.

How ShardingSphere Abstracts Configuration Governance? #

ShardingSphere’s configuration governance features are very rich, and closely related to daily development are its configuration center and registration center features. ShardingSphere provides its own abstraction and implementation solutions for these two features.

Configuration Center in ShardingSphere #

For the management of configuration information, the common practice is to store them in configuration files. We can maintain the configuration information based on YAML or XML format configuration files, which are also supported in ShardingSphere. In a monolithic system, configuration files can meet the requirements, and the configuration management work centered around configuration files usually does not pose too much challenge. However, in a distributed system, the increasing number of runtime instances makes it difficult to manage scattered configurations, and the problem of configuration inconsistency is very serious. Concentrating the configuration in the configuration center can facilitate more effective management.

The adoption of a configuration center also means adopting the design idea of centralized configuration management. In a centralized configuration center, configuration information of different environments such as development, testing, and production is uniformly stored in the configuration center, which is one dimension. Another dimension is to ensure that all service instances of the same type of service in a distributed cluster save the same configuration file and can be synchronously updated. The diagram below illustrates the configuration center:

1.png Design idea of centralized configuration management

In ShardingSphere, multiple implementation solutions for configuration centers are provided, including mainstream tools such as ZooKeeper, Etcd, Apollo, and Nacos. Developers can also implement their own configuration center as needed and load it into the ShardingSphere runtime environment through the SPI mechanism.

On the other hand, configuration information is not immutable. Uniform distribution of modified configuration information is another important capability that the configuration center can provide. Any changes to the configuration information in the configuration center can be synchronized to each service instance in real-time. In ShardingSphere, through the configuration center, dynamic switching of data sources, data tables, sharding, and read-write separation strategies can be supported.

At the same time, based on the centralized configuration management solution, ShardingSphere also supports the implementation solution of loading configuration information from the local. If we want to take the local configuration information as the main reference and override the configuration in the configuration center, we can easily achieve it through a switch.

Registration Center in ShardingSphere #

In terms of implementation, the registration center is very similar to the configuration center. ShardingSphere also provides implementation solutions for registration centers based on third-party tools such as ZooKeeper and Etcd, which can also be used as configuration centers.

The difference between the registration center and the configuration center lies in the type of data they store. The configuration center obviously manages configuration data, while the registration center stores various dynamic/temporary state data of the ShardingSphere runtime, and the most typical runtime state data is the current DataSource instance. So, what is the purpose of storing these dynamic and temporary state data? Let’s take a look at this diagram:

2.png Diagram of registration center data storage and listening mechanism

The registration center generally provides a distributed coordination mechanism. In the registration center, all DataSources create temporary nodes under the specified root directory path, and all business services accessing these DataSources will listen to this directory. When a new DataSource joins, the registration center will instantly notify all business services, and the business services will maintain the corresponding routing information. When a DataSource fails, the business services will also receive notifications through the listening mechanism.

Based on this mechanism, we can provide governance capabilities for DataSources, including circuit breakers for accessing a specific DataSource, or disabling access to slave DataSources.

In ShardingSphere, the registration center is more oriented towards internal framework usage, and there is currently limited functionality for developers specifically built for the registration center. Therefore, today we will focus on the usage of the configuration center, and we will discuss the registration center in the source code analysis section. Next, I will guide you to complete the system transformation work of integrating the configuration center.

System Transformation: How to Integrate the Configuration Center? #

Since the creation of the configuration center depends on third-party tools, we need to first prepare the development environment.

Prepare the Development Environment #

To integrate the configuration center, the first step is to introduce the dependency package in ShardingSphere related to configuration governance. In the Spring Boot environment, this dependency package is sharding-jdbc-orchestration-spring-boot-starter:

<dependency>
    <groupId>org.apache.shardingsphere</groupId>         
    <artifactId>sharding-jdbc-orchestration-spring-boot-starter</artifactId>
</dependency>

In the following content, we will demonstrate how to implement the configuration center based on ZooKeeper, a distributed coordination tool. In ShardingSphere, the integrated ZooKeeper client component is Curator, so we also need to introduce the sharding-orchestration-reg-zookeeper-curator component:

<dependency>
    <groupId>org.apache.shardingsphere</groupId>         
    <artifactId>sharding-orchestration-reg-zookeeper-curator</artifactId>
</dependency>

Of course, if we are using Nacos, we also need to add the relevant dependency packages:

<dependency>
    <groupId>org.apache.shardingsphere</groupId>
    <artifactId>sharding-orchestration-reg-nacos</artifactId>
</dependency> 
<dependency>
    <groupId>com.alibaba.nacos</groupId>
    <artifactId>nacos-client</artifactId>
</dependency>

After configuring these, the development environment is ready. For the configuration center, the main job of developers is to configure it. Let’s take a look at the configuration items for the configuration center together.

Mastering Configuration Items #

For the configuration center, ShardingSphere provides a series of DataSources, including OrchestrationShardingDataSource for data sharding, OrchestrationMasterSlaveDataSource for read-write separation, and OrchestrationEncryptDataSource for data encryption. Around these DataSources, there are also corresponding DataSourceFactory classes. Let’s take OrchestrationMasterSlaveDataSourceFactory as an example to see the configuration class required to create a DataSource:

public final class OrchestrationMasterSlaveDataSourceFactory {
    
    public static DataSource createDataSource(final Map<String, DataSource> dataSourceMap, final MasterSlaveRuleConfiguration masterSlaveRuleConfig, 
                                              final Properties props, final OrchestrationConfiguration orchestrationConfig) throws SQLException {
        if (null == masterSlaveRuleConfig || null == masterSlaveRuleConfig.getMasterDataSourceName()) {
            return createDataSource(orchestrationConfig);
        }
        MasterSlaveDataSource masterSlaveDataSource = new MasterSlaveDataSource(dataSourceMap, new MasterSlaveRule(masterSlaveRuleConfig), props);
        return new OrchestrationMasterSlaveDataSource(masterSlaveDataSource, orchestrationConfig);
    }
     …
}

As you can see, there is a governance rule configuration class called OrchestrationConfiguration, which is also used by other DataSourceFactory classes:

public final class OrchestrationConfiguration {
    // Governance rule name
    private final String name;
    // Registry (configuration) center configuration class
    private final RegistryCenterConfiguration regCenterConfig;
    // Flag to indicate if local configuration should override server configuration
    private final boolean overwrite;
}

In the OrchestrationConfiguration, we can see the overwrite flag used to specify whether the local configuration overrides the server configuration. We can also see a subclass of configuration called RegistryCenterConfiguration, which has several commonly used configuration options:

public final class RegistryCenterConfiguration extends TypeBasedSPIConfiguration {

    // Configuration center server list
    private String serverLists;
    
    // Namespace
    private String namespace;
    
    ...
}

This includes the serverLists for the configuration center server list and the namespace for identifying uniqueness. Since RegistryCenterConfiguration extends TypeBasedSPIConfiguration, it automatically inherits the type and properties configuration options as well.

Implementing the Configuration Center #

Now, let’s implement a configuration center based on ZooKeeper. First, you need to download the ZooKeeper server component and make sure it starts successfully. If using the default configuration, ZooKeeper will listen for requests on port 2181.

Next, create a configuration file and enter the configuration options. Since we are still using read-write separation as an example, we set up one master and two slaves, a total of three data sources in the configuration file. This part of the configuration options has already been explained when introducing the read-write separation mechanism, so we won’t go into detail here:

spring.shardingsphere.datasource.names=dsmaster,dsslave0,dsslave1 
spring.shardingsphere.datasource.dsmaster.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dsmaster.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.dsmaster.jdbc-url=jdbc:mysql://localhost:3306/dsmaster
spring.shardingsphere.datasource.dsmaster.username=root
spring.shardingsphere.datasource.dsmaster.password=root 
spring.shardingsphere.datasource.dsslave0.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dsslave0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.dsslave0.jdbc-url=jdbc:mysql://localhost:3306/dsslave0
spring.shardingsphere.datasource.dsslave0.username=root
spring.shardingsphere.datasource.dsslave0.password=root 
spring.shardingsphere.datasource.dsslave1.type=com.zaxxer.hikari.HikariDataSource
spring.shardingsphere.datasource.dsslave1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.dsslave1.jdbc-url=jdbc:mysql://localhost:3306/dsslave1
spring.shardingsphere.datasource.dsslave1.username=root
spring.shardingsphere.datasource.dsslave1.password=root 
spring.shardingsphere.masterslave.load-balance-algorithm-type=random
spring.shardingsphere.masterslave.name=health_ms
spring.shardingsphere.masterslave.master-data-source-name=dsmaster
spring.shardingsphere.masterslave.slave-data-source-names=dsslave0,dsslave1 
spring.shardingsphere.props.sql.show=true

Next, specify the configuration center. We set overwrite to true, which means that these local configuration options will overwrite the configuration options saved on the ZooKeeper server. In other words, we are using a local configuration mode. Then, we set the configuration center type to ZooKeeper, the server list to localhost:2181, and the namespace to orchestration-health_ms:

spring.shardingsphere.orchestration.name=health_ms
spring.shardingsphere.orchestration.overwrite=true
spring.shardingsphere.orchestration.registry.type=zookeeper
spring.shardingsphere.orchestration.registry.server-lists=localhost:2181
spring.shardingsphere.orchestration.registry.namespace=orchestration-health_ms

Now, let’s start the service, and the console will display the relevant log information about communicating with ZooKeeper:

2020-05-30 18:13:45.954  INFO 20272 --- [           main] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@585ac855
2020-05-30 18:13:46.011  INFO 20272 --- [0:0:0:0:1:2181)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-05-30 18:13:46.012  INFO 20272 --- [0:0:0:0:1:2181)] org.apache.zookeeper.ClientCnxn          : Socket connection established to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181, initiating session
2020-05-30 18:13:46.029  INFO 20272 --- [0:0:0:0:1:2181)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181, sessionid = 0x10022dd7e680001, negotiated timeout = 40000
2020-05-30 18:13:46.034  INFO 20272 --- [ain-EventThread] o.a.c.f.state.ConnectionStateManager     : State change: CONNECTED

Meanwhile, the ZooKeeper server also responds to requests from the application. We can use some ZooKeeper visualization client tools to view the data on the server. Here, I use the tool called ZooInspector. Since ZooKeeper is essentially a tree structure, the configuration information is added to the root node:

3.png Configuration Node Diagram in ZooKeeper

We focus on the “config” section, where the “rule” node contains the read-write separation rule settings:

4.png “Rule” Configuration Item in ZooKeeper

The “datasource” node clearly contains the various data source information specified earlier.

Since we set the spring.shardingsphere.orchestration.overwrite configuration item to true in the local configuration file, the changes in the local configuration will affect the server-side configuration, and thus affect all applications that use these configurations. If you do not want this impact to occur and instead want to use the configuration stored in the configuration center, what should you do?

It’s simple, we just need to set spring.shardingsphere.orchestration.overwrite to false. Closing this configuration switch means that we will only read the configuration from the configuration center, which means that the local configuration no longer needs to store any information, only including the relevant content specified in the configuration center:

spring.shardingsphere.orchestration.name=health_ms
spring.shardingsphere.orchestration.overwrite=false
spring.shardingsphere.orchestration.registry.type=zookeeper
spring.shardingsphere.orchestration.registry.server-lists=localhost:2181
spring.shardingsphere.orchestration.registry.namespace=orchestration-health_ms

After executing the test cases, you will find that the read-write separation rule is still effective.

If you choose to use another framework to build the configuration center server, such as Alibaba’s Nacos, it’s also simple. Just set spring.shardingsphere.orchestration.registry.type to nacos and provide the corresponding server-lists:

spring.shardingsphere.orchestration.name=health_ms
spring.shardingsphere.orchestration.overwrite=true
spring.shardingsphere.orchestration.registry.type=nacos
spring.shardingsphere.orchestration.registry.server-lists=localhost:8848
spring.shardingsphere.orchestration.registry.namespace=

Summary #

In this lesson, we discussed the support for orchestration and governance-related functions in ShardingSphere. ShardingSphere provides two governance mechanisms: the configuration center and the registry center. These two mechanisms have similar underlying designs but target different application scenarios. Based on the configuration center, we provided a specific development process. For the configuration center, the key is to understand how to use ZooKeeper as a distributed coordination tool to achieve dynamic updating and synchronization between local and remote configuration information.

Here’s a question for you to think about: What are the similarities and differences in the design of the configuration center and the registry center in ShardingSphere?

This lesson is the last topic in the core functionality of ShardingSphere in this column. Starting from the next lesson, we will delve into the analysis of ShardingSphere’s source code. I will discuss how to efficiently read the ShardingSphere source code. Remember to come to the class on time.