29 Kafka Dynamic Configuration Unveiled

29 Kafka Dynamic Configuration Unveiled #

Hello, I’m Hu Xi. Today, the topic I would like to discuss with you is the dynamic broker parameter configuration in Kafka.

What is dynamic broker parameter configuration? #

Before we begin today’s sharing, let’s review how to set Kafka parameters, especially the parameters on the broker side.

In the config directory of the Kafka installation directory, there is a server.properties file. Usually, we specify the path to this file to start the broker. If we want to set any parameters on the broker side, we must explicitly add a corresponding configuration line in this file and then start the broker process to make the parameters take effect. Our common practice is to set all the parameters at once before starting the broker. When we need to change any parameter later, we must restart the broker. However, how can we restart the server in a production environment? Therefore, modifying broker parameters is currently a very painful process.

Based on this pain point, the community officially introduced dynamic broker parameters (Dynamic Broker Configs) in version 1.1.0. The so-called dynamic means that after modifying the parameter value, it can take effect immediately without restarting the broker, while the parameters configured in server.properties before are called static parameters (Static Configs). Obviously, it is very practical to adjust parameter values dynamically without restarting the service. If you want to experience dynamic broker parameters, please upgrade to version 1.1 as soon as possible.

Of course, in the latest version 2.3, there are more than 200 broker parameters, but the community did not upgrade each parameter to dynamic parameters. It only made some parameters adjustable dynamically. So how can we distinguish which parameters are dynamic?

If you open the Kafka official website after version 1.1 (including 1.1), you will find that the Broker Configs table has added a Dynamic Update Mode column. This column has three types of values, namely read-only, per-broker, and cluster-wide. Let me explain what they mean.

  • read-only: Parameters marked as read-only behave the same as the original parameters, and modifications will only take effect after restarting the broker.
  • per-broker: Parameters marked as per-broker are dynamic parameters. After modifying them, they will only take effect on the corresponding broker.
  • cluster-wide: Parameters marked as cluster-wide are also dynamic parameters. After modifying them, they will take effect across the entire cluster, which means they will affect all brokers. You can also modify cluster-wide parameters for specific brokers.

Let me give an example to illustrate the difference between per-broker and cluster-wide. The broker-side parameter listeners should be familiar to you. It is a per-broker parameter, which means you can only dynamically adjust the listeners for a single broker, but not directly for a batch of brokers. The parameter log.retention.ms is a cluster-wide parameter. Kafka allows you to set a unified log retention time value for all brokers in the cluster. Of course, you can also modify this value for a single broker.

Usage Scenarios #

You may wonder what are the usage scenarios for dynamic Broker parameters. In fact, because there is no need to restart the Broker, the usage scenarios for dynamic Broker parameters are very wide-ranging, usually including but not limited to the following:

  • Dynamically adjust various thread pool sizes on the Broker side to respond to sudden traffic.
  • Dynamically adjust connection information or security configuration on the Broker side.
  • Dynamically update the SSL Keystore’s validity period.
  • Dynamically adjust the performance of the Compact operation on the Broker side.
  • Real-time changes to JMX metric collectors (JMX Metrics Reporter).

Among these usage scenarios, dynamically adjusting thread pool sizes should be the most practical feature. Many times, when the inbound data on Kafka Broker increases sharply, it will cause a backlog of requests on the Broker side. With dynamic parameters, we can dynamically increase the number of network threads and I/O threads to quickly consume some of the backlog. When the sudden traffic is over, we can also adjust the number of threads back to reduce resource waste. The entire process does not require restarting the Broker. You can even encapsulate this set of actions for adjusting thread counts into a scheduled task to achieve automatic scaling.

How to Save? #

Due to the special nature of dynamic configuration, it must have a different mechanism for saving compared to regular read-only parameters. Let me explain how Kafka saves dynamic configurations.

Firstly, Kafka saves dynamic broker parameters in ZooKeeper, with the specific znode path shown in the following diagram.

Now, let me explain the contents of the diagram. “changes” is used to monitor real-time changes to dynamic parameters, but it does not store the parameter values. “topics” is used to store Kafka topic-level parameters. Although they are not dynamic broker-side parameters, they can also be dynamically changed.

“users” and “clients” are znode nodes used to dynamically adjust client quotas (Quota). Quota refers to the limitation set by Kafka administrators on the throughput or CPU resources used by clients connecting to the cluster.

Analyzing this, we can see that the /config/brokers znode is where the dynamic broker parameters are actually stored. This znode has two main types of child nodes. The first type of child node is only one, with a fixed name called “< default >”, and it stores the cluster-wide dynamic parameters mentioned earlier. The other type is named after the broker.id and stores per-broker parameters for specific brokers. As per-broker parameters are specific to each broker, there may be multiple child nodes of this type.

Let’s take a look at another image that shows the dynamic broker parameters in my Kafka cluster environment.

In this image, I first checked the child nodes under /config/brokers. We can see that there is a < default > node and child nodes named 0 and 1. The < default > node contains the cluster-wide parameters I have set, while the 0 and 1 nodes store the per-broker parameters I have set for Broker 0 and Broker 1, respectively.

Next, I show the settings for cluster-wide and per-broker parameters separately. Taking the num.io.threads parameter as an example, its cluster-wide value has been dynamically adjusted to 12, but it has been set to 16 for Broker 0 and 8 for Broker 1. The values I set specifically for Broker 0 and Broker 1 will override the cluster-wide value, but on other brokers, the parameter will still be calculated based on the default value of 12.

Adding static parameters to the discussion, the priority order of cluster-wide, per-broker, and static parameters is as follows: per-broker parameters > cluster-wide parameters > static parameters > Kafka default values.

Additionally, if you carefully examine the ephemeralOwner field in the above diagram, you will see that its value is 0x0. This indicates that these znodes are persistent nodes and will always exist. Even if the ZooKeeper cluster restarts, this data will not be lost, ensuring that these dynamic parameter values will remain effective.

How to configure? #

After discussing the saving principle, let’s talk about how to configure dynamic Broker parameters. Currently, the only tool for setting dynamic parameters is the kafka-configs script that comes with Kafka. Next, I will demonstrate how to dynamically adjust the unclean.leader.election.enable parameter.

The following command shows how to set a global value at the cluster level, which means setting a cluster-wide value.

$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-default --alter --add-config unclean.leader.election.enable=true
Completed updating default config for brokers in the cluster,

In general, the command is very simple, but one thing to note is that if you want to set a cluster-wide dynamic parameter, you need to explicitly specify entity-default. Now, let’s use the following command to check if the configuration we just set is successful.

$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-default --describe
Default config for brokers in the cluster are:
  unclean.leader.election.enable=true sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:unclean.leader.election.enable=true}

From the output, we can see that we have successfully set the parameter value to true at the global level. Note the word “sensitive=false”, which indicates that the parameter we are adjusting is not sensitive data. If we are adjusting a parameter like a password, this field will be true, indicating that it is sensitive data.

Now, after adjusting the cluster-wide parameter, let me demonstrate how to set per-broker parameters. Let’s continue with the unclean.leader.election.enable parameter as an example, and I will now set a different value for Broker ID 1. The command is as follows:

$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-name 1 --alter --add-config unclean.leader.election.enable=false
Completed updating config for broker: 1.

Similarly, let’s use the following command to check if the setting we just made takes effect.

$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-name 1 --describe
Configs for broker 1 are:
  unclean.leader.election.enable=false sensitive=false synonyms={DYNAMIC_BROKER_CONFIG:unclean.leader.election.enable=false, DYNAMIC_DEFAULT_BROKER_CONFIG:unclean.leader.election.enable=true, DEFAULT_CONFIG:unclean.leader.election.enable=false}

The output of this command contains a lot of information, but there are two points of interest.

  1. At the per-broker level, the parameter has been set to false, indicating that the command ran successfully.
  2. From the second last line, we can see that at the global level, the parameter value is still true. This indicates that the cluster-wide parameter value we set earlier is still effective.

If we want to delete a cluster-wide parameter or a per-broker parameter, it is also very simple, just execute the following commands respectively.

# Delete cluster-wide parameter
$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-default --alter --delete-config unclean.leader.election.enable
Completed updating default config for brokers in the cluster,


# Delete per-broker parameter
$ bin/kafka-configs.sh --bootstrap-server kafka-host:port --entity-type brokers --entity-name 1 --alter --delete-config unclean.leader.election.enable
Completed updating config for broker: 1.

To delete a dynamic parameter, specify delete-config. After removing the dynamic parameter configuration, if we run the command to view again, the result will be as follows:

# View cluster-wide parameters
$ bin/kafka-configs.sh --bootstrap-server kafka-host:port  --entity-type brokers --entity-default --describe
Default config for brokers in the cluster are:


# View dynamic parameter configuration on Broker 1
$ bin/kafka-configs.sh --bootstrap-server kafka-host:port  --entity-type brokers --entity-name 1 --describe
Configs for broker 1 are:

At this point, all the dynamic parameters configured have been successfully removed.

Just now, I only gave an example of one parameter. If you want to know what dynamic Broker parameters are available, one way is to check the Broker-side parameter list on the Kafka official website. Another way is to directly run the kafka-configs script without any parameters, and the documentation of the script will tell you what dynamic Broker parameters are currently available. Let’s take a look at these two figures first.

Seeing so many dynamic Broker parameters, you may ask: Do I need to adjust all of them? Can you tell me the most commonly used ones? Based on my actual experience, I will share with you some parameters that are likely to be dynamically adjusted.

1. log.retention.ms.

Changing the log retention time is considered a relatively frequent operation, after all, we cannot perfectly estimate the message retention time for all businesses. Although this parameter has corresponding topic-level parameters that can be set, having the ability to dynamically change it at the global level is still a great feature highlight.

2. num.io.threads and num.network.threads.

These are the two sets of thread pools we mentioned earlier. Personally, I think this is the most practical scenario for dynamic Broker parameters. After all, in actual production environments, the processing capacity of Broker-side requests often needs to be scaled out on demand. If there were no dynamic Broker parameters, we would not be able to achieve this.

3. Parameters related to SSL.

This mainly includes 4 parameters (ssl.keystore.type, ssl.keystore.location, ssl.keystore.password, and ssl.key.password). Allowing them to be dynamically adjusted in real-time enables us to create SSL certificates with short expiration times. Whenever we make an adjustment, Kafka will reconfigure the Socket connection channel and update the Keystore. The new connections will use the new Keystore, and adjusting this group of parameters periodically is beneficial for increasing security.

4. num.replica.fetchers.

This is also one of the most practical dynamic Broker parameters in my opinion. Slow Follower replica fetch has always been a difficult problem in online Kafka environments. To address this issue, a common approach is to increase the value of this parameter to ensure that there are sufficient threads to perform Follower replica fetch from Leader replica. With dynamic parameters, you don’t need to restart the Broker to take effect on the Follower side, so I consider this to be a very practical use case.

Summary #

Alright, let’s summarize. Today, we focused on the dynamic broker parameters introduced in Kafka 1.1.0. The biggest advantage of these parameters is that they can take effect without restarting the broker, which greatly reduces operational costs. In addition, I also provided the mechanism for saving dynamic parameters and their setting methods. In the later part of this column, I will also present another method for setting dynamic parameters. Stay tuned.

Open Discussion #

Currently, only some of the Broker parameters have been upgraded to dynamic parameters in the community. In actual usage, what other parameters do you think should also be dynamically modifiable?

Feel free to write down your thoughts and answers, and let’s discuss together. If you find it helpful, please consider sharing this article with your friends.