32 What Does a Typical Distributed Caching System Look Like

32 What Does a Typical Distributed Caching System Look Like #

In this lesson, we will specifically look at what a typical distributed cache system looks like.

Distributed Redis Service #

Since this course focuses on caching, next, I will introduce a typical composition of a distributed cache system using the example of the distributed Redis service system within Weibo.img

The Redis service inside Weibo is also known as RedisService. The overall architecture of RedisService is shown in the figure. It consists of five main components: Proxy, Storage, Cluster Manager, Configuration Center, and Graphite.

  • The Proxy in RedisService follows a stateless multi-tenant model, where different business storages can be mounted under each Proxy, and business is differentiated by ports.
  • The storage is based on Redis and only retains basic storage functionality when it comes to cluster data storage. It supports customizable migration features, but the storage itself is stateless and does not store key-slot mapping relationships.
  • The Configuration Center is used to record and distribute various metadata, such as the IP, port, and configuration of the Proxy. Subscribers can be promptly aware of any changes.
  • The Graphite system is used to record and display status data of the system, business, components, and instances.
  • The Cluster Manager is responsible for daily operations and maintenance management, business SLA monitoring, and alarms. At the same time, the Cluster Manager integrates Proxy, Redis backend storage, and Configuration Center to manage business data in a cluster.

Multi-tenant Proxyimg #

The Proxy in the RedisService has no state, and all instances of the Proxy have the same startup parameters. However, before the Proxy starts, the clusterManager sets the business and storage configuration information for the instance in the configuration center. After the Proxy starts, it retrieves and subscribes to the configuration from the configuration center using its own IP, and then initializes. The Proxy uses a long connection to communicate with the backend Redis storage. When the Client sends concurrent requests to the Proxy, the Proxy packages the requests and sends them to the backend in batches using a pipeline to improve request efficiency. For the multi-tenant Proxy, since the storage location for different businesses may be different, each request needs to be differentiated by business. There are generally two ways to achieve this.

Solution 1: Differentiate the business based on the namespace prefix of the key. For example, if the Client requests key k1 under the user, graph, and feed businesses respectively, the business Client constructs {user}k1, {graph}k1, and {feed}k1, and then sends them to the Proxy. The Proxy parses the key prefix to determine the corresponding business for the key.

Solution 2: Assign a dedicated port for each business, and different businesses access their own ports. The Proxy determines the business type based on the port. This method does not require parsing the key prefix and does not need to reconstruct requests, making it more efficient. However, it requires configuring ports for businesses, which increases management costs. In practice, since business Redis resources generally use different ports, the business Proxy can use the minimum port of the business resource shard as the port identifier for the business.

Redis Data Storage img #

Redis storage in RedisService is based on the Redis 5.0 extension, internally referred to as wredis. Wredis does not store key-slot mappings, only records the count of keys stored in each instance. Wredis handles any received operation commands, and the correctness of data sharding access is ensured by the requesting end. During the daily off-peak period, clusterManager scans the Redis storage to check for any abnormal slot storage. Due to the large number of small value keys in Weibo, if key-slot mappings are added to the cluster, the storage cost will greatly increase. By eliminating key-slot mappings and implementing other related optimizations, some businesses can reduce storage capacity by more than 20%.

Wredis supports synchronous and asynchronous migration of slots. It also supports hot upgrades, which can be completed within milliseconds. Wredis also supports full incremental replication and multiple data structures supported by Weibo’s internal extensions. Hot upgrades, full incremental replication, and data structure extensions were covered in previous lessons, which can be referred to for more details in the “Redis Function Extension” lesson.

Configuration Center configService img #

Weibo’s configuration center, internally referred to as configService, is a fundamental component for managing configuration metadata within Weibo. ConfigService itself is deployed across multiple IDCs. Configuration information is stored using a multi-version data structure and supports version tracking. At the same time, configuration data can be quickly verified for consistency using a Merkle hash tree. All business, resource, and proxy configurations in RedisService are stored in configService. These configurations are written and modified by the cluster. Proxies and business clients obtain and subscribe to the required configuration data. When configuration changes occur in configService, only the affected nodes are notified, reducing the overhead of obtaining the entire data for subscribers after configuration changes.

ClusterManager is an operations background tool primarily used for operations work such as backend resource deployment, proxy instance deployment, configuration changes, and version upgrades. It is also used for cluster management of data. ClusterManager internally stores the mapping of business data clusters and performs data migration and failover when necessary. Migration is performed using slots, allowing for traffic control based on load. ClusterManager also supports Proxy domain management for business access, monitors the status of cluster nodes, monitors business SLA metrics, and alerts on exceptions for timely operations handling.

Cluster Data Synchronization img #

The data in RedisService is stored in multiple regions, with multiple IDCs in each region. The deployment method is a combination of core intranet and public cloud. The use of public cloud is mainly determined by the characteristics of Weibo’s business. During sudden or hotspot events, it is easy to create a traffic peak, resulting in a significant increase in read and write TPS. By using public cloud, the system can be quickly and cost-effectively scaled to significantly increase its processing capacity. According to the characteristics of the business, wredis is divided into cache and storage types. For Redis cache, it is mainly driven by message bus for updates, while for Redis storage, it adopts master-slave replication for updates. The difference in update methods is mainly due to the fact that Redis, as a cache type for business data, has different hotspot data in different regions or IDCs. If master-slave replication is used, the IDCs where the slave is deployed will have the problem of hot data not being able to enter the cache, while cold data cannot be eliminated because the elimination of the slave also depends on the master. For business scenarios where Redis is used for storage, since the cache stores the complete data, using master-slave replication directly ensures data consistency, which is the most convenient approach.