04 Intra-City Dual-Active How to Realize Data Synchronization Between Computer Rooms #
Hello, I’m Xu Changlong. Today, let’s take a look at another phase of the user center transformation: building multiple data centers.
In the early stages of business, considering the investment cost, many companies usually provide services using only one data center. However, as the business grows and the traffic increases, we have higher requirements for the response speed and availability of our services. At this point, we need to consider distributing services to different regions to provide better service. This is the path that internet companies must go through during the phase of traffic growth.
In my previous company, the traffic continuously increased for three years. Once, the external network of the data center was suddenly disconnected, and all online services went offline. The network provider was unreachable. Due to the lack of a backup data center, we had to urgently coordinate for three days to establish a new connection and restore the service. This incident had a significant impact, resulting in losses of tens of millions of yuan for the company.
After this painful lesson, we migrated our services to a large-scale data center and decided to build dual data centers in the same city to improve availability. In this way, when one data center encounters a problem and cannot be accessed, users can quickly switch to a fault-free data center through the HttpDNS interface.
In order to ensure that the other data center can directly handle the traffic when one data center is damaged, the equipment in these two data centers must be purchased in a 1:1 ratio. However, it is too wasteful to leave one data center as a cold standby for a long time. Therefore, we hope that both data centers can simultaneously provide services externally, which means achieving dual active status in the same city.
Regarding this, we encountered a key issue: how to achieve database synchronization between data centers in the same city?
Core Data Center Design #
Due to the master-slave architecture of the database, there can only be one master database in the entire network. Therefore, we can only have one data center to store the master database for data updates, and then this data center will synchronize the data to other backup data centers. Although there are dedicated lines connecting the data centers, it cannot guarantee a completely stable network. If a network failure occurs, we need to find a way to ensure that the data centers can quickly resume data synchronization after the network is repaired.
Some may suggest using a distributed database directly. However, it takes a considerable amount of time and cost to change the existing service system and invest in the trend of distributed databases, which is impractical for most companies. Therefore, we need to explore how to transform the existing system to achieve dual-active data center database synchronization within the same city, which is the goal of this lesson.
The core database center scheme is a common implementation method, suitable for data centers that are no more than 50 kilometers apart.
In this scheme, the master database is centralized in one data center, and the databases in other data centers are all slave databases. When there is a data modification request, the master database in the core data center completes the modification first, and then transmits the modified data to the slave databases in the backup data centers through database master-slave synchronization. Since the information accessed by users is usually obtained from the cache, backup data centers will update the modified data to the local cache first in order to reduce the master-slave delay.
At the same time, the client will record the last timestamp of data modification locally (if not available, the current time will be used). When the client requests the server, the server will automatically compare the update time of the corresponding data in the cache with the modification time recorded locally by the client.
If the update time in the cache is earlier than the modification time recorded in the client, the server will trigger a synchronization command to attempt to find the latest data in the slave databases. If not found, the latest data obtained from the master database will be put into the cache of the visited data center. This way, the problem of delayed user data updates between data centers can be avoided.
In addition, the client will use a request scheduling interface to ensure that a user only accesses one data center within a short period of time, preventing data update conflicts caused by switching between data centers.
Overall, this is a relatively simple design, but it has many disadvantages. For example, if the core data center goes offline, the other data centers will not be able to update. During a failure, manual switching of the master-slave configuration in each proxy is required to restore the service. Even after the failure is resolved, manual intervention is still required to restore the master-slave synchronization.
In addition, due to the large master-slave synchronization delay, the recently updated data in the business needs to be delayed for some time before it can be found in the backup data center. This requires manual consideration in our business and overall implementation is inconvenient.
Here is a common reference for network latency:
- Within the same data center: 0.1 ms
- Within the same city (within 100 kilometers): 1 ms (10 times of within the same data center)
- Beijing to Shanghai: 38 ms (380 times of within the same data center)
- Beijing to Guangzhou: 53 ms (530 times of within the same data center)
Note that the above are only for a single round-trip time (RTT) request, while data synchronization between data centers involves multiple sequentially layered requests. If a large amount of data needs to be updated, the synchronization delay between master and slave databases will increase. Therefore, this dual-active data center scheme can only handle small data volume and the business should not frequently update data.
In addition, if there is a strong consistency requirement for the service, and all operations must be “remotely executed” on the master database, then these operations will also increase the master-slave synchronization delay.
In addition to the above issues, there may also be occasional failures in the dedicated lines between data centers. I have encountered a situation where the dedicated lines between data centers were disconnected for two hours, and during that period, temporary synchronization had to be maintained through the public network. However, public network synchronization is very unstable, with network latency fluctuating between 10ms and 500ms, and the master-slave delay reaching more than one minute. Fortunately, the user center service primarily stores data in long-term cache, and there were no major issues with the main business processes, except that user information modification became slow.
Sometimes, the master-slave synchronization between the data centers may be temporarily disconnected, so it is recommended to handle this with alerts. Once this happens, a notification should be sent to the fault alarm group for manual repair by the DBA.
Furthermore, during the period of master-slave data inconsistency, I encountered cases where duplicate registrations with auto-increment IDs caused primary key conflicts. Here, I recommend replacing the auto-increment IDs with “IDs calculated by the SnowFlake algorithm” to reduce primary key conflict issues caused by data inconsistency between data centers.
As can be seen, although the core database central scheme achieves dual-active data center synchronization within the same city, it requires a significant amount of manpower. The DBA needs to manually maintain the synchronization, and it is also very troublesome to restore the synchronization after master-slave synchronization is disconnected, which is time-consuming and labor-intensive. In addition, developers need to constantly monitor the situation of master-slave data inconsistency, making overall maintenance inconvenient. Therefore, I recommend another solution here: the database synchronization tool Otter.
Cross-Datacenter Synchronization Tool: Otter #
Otter is a database synchronization tool developed by Alibaba. It enables fast data synchronization across different datacenters, cities, and countries. As shown in the diagram below, Otter monitors the Row binlog of the primary MySQL database using Canal and synchronizes the data updates in parallel to other MySQL databases in different datacenters.
To achieve dual-active datacenters in the same city, we can use Otter to implement a dual-master configuration (note: dual-master is not applicable for high consistency requirements and is not recommended for such business use cases). This allows bidirectional synchronization between the active datacenters:
As shown in the diagram, each datacenter has its own master and slave databases. The cache can be either cross-datacenter replication or local replication, depending on the business requirements. Otter synchronizes the data changes from the master database in each datacenter to the Otter Node, and then synchronizes them to the Node in the opposite datacenter after processing through Otter’s SETL, achieving data synchronization between the dual datacenters.
Now, let’s talk about how Otter resolves conflicts that arise when two datacenters simultaneously modify the same data.
There are two types of conflicts in Otter: row conflicts and column conflicts. Row conflicts can be resolved by comparing the modification timestamps of the data, or by querying the source database to overwrite the target database when conflicts occur. For column conflicts, we can overwrite or merge multiple modification actions based on the modification timestamps. For example, if data is modified in datacenter A and datacenter B, the merged result would be -2. This ensures eventual consistency of the data.
However, please note that this merging approach is not suitable for data management scenarios such as inventory management, as it can lead to overselling. If there is a similar requirement, it is recommended to use long-term caching to resolve it.
Otter not only supports dual-active datacenters, but also supports multi-datacenter synchronization, such as star-shaped bidirectional synchronization and cascading synchronization (as shown in the diagram below). However, these complex structures are not practical as they are difficult to troubleshoot and recovery becomes complicated if the unique decision-making database encounters issues. Therefore, unless absolutely necessary, these complex structures are not recommended.
Furthermore, I want to emphasize that the dual-active bidirectional synchronization solution we talked about is only suitable for the same city. Generally, datacenter synchronization within 50 to 100 kilometers is considered within the same city.
If the distance exceeds this limit, it is recommended to only perform data synchronization backups. This is because the synchronization latency becomes too high, and the cost of focusing on latency in each step becomes significant. If our business has a high requirement for consistency, it is advisable to design the system in such a way that this consistency requirement is limited to the same datacenter, with other databases used only to store result statuses.
Why must the distance between datacenters be within 100 kilometers? You can understand it by looking at the synchronization performance and latency reference provided by Otter.
The specific table is as follows:
To improve the efficiency of cross-datacenter data synchronization, Otter consolidates the operation logs used for master-slave replication, merging multiple modifications to the same data into a single log. It also optimizes network transmission and synchronization strategies with sliding windows and parallel optimization.
Compared to MySQL replication, Otter provides a 5x performance improvement. Looking at the table above, with data synchronization implemented using Otter, the concurrency is improved and the latency is reduced. As long as we can control user requests within the same datacenter without frequent switching, there will be fewer conflicts for the same data modification.
When using Otter to implement bidirectional synchronization, our business does not need to make significant changes to adapt to dual-master dual-active datacenters. Specifically, the business only needs to operate on the local master database, replacing “auto-increment primary keys” with “primary keys generated by the Snowflake algorithm” and “exclusive unique indexes” with “distributed mutex locks” to meet most requirements.
However, please note that when adopting a bidirectional synchronization solution with dual-active datacenters in the same city, data updates should not be too frequent, as it can result in greater synchronization delay. This solution works best when the amount of data being operated on by the business is small.
Speaking of which, let’s talk about Otter’s failover mechanism. Currently, Otter provides a simple master-slave failover feature. Clicking on “Switch” in the Manager allows you to switch between Canal and database master-slave replication. If dual-active datacenters are used, we do not need to make any changes to the existing code related to database operations because this synchronization is bidirectional.
When one datacenter experiences a failure, redirect the user traffic from the faulty datacenter to the one that is functioning properly. Once the fault is resolved, resume data synchronization without the need to switch the MySQL master-slave IP addresses in the business code. Remember, if one of the dual-active datacenters experiences a failure, the datacenter in other cities should only be used for backup or temporarily running independently. Do not perform dual-active datacenter synchronization across cities, as high synchronization latency can lead to business data corruption.
Finally, let me reiterate some notes on using Otter: First, to ensure data integrity, when modifying table structures, it is generally recommended to first modify them on the slave database. Therefore, when setting up Otter synchronization, it is suggested to set the pipeline synchronization to ignore DDL synchronization errors. Second, when adding new columns to the database table, only add them at the end of the table; do not delete old columns. It is also recommended to synchronize the new columns to the target database first before syncing them to the primary database, to avoid data loss. Third, bidirectional synchronization tables should not have default values for newly added columns, and Otter does not support synchronization for tables without primary keys.
Conclusion #
Data synchronization between data centers has always been a pain in the industry due to the high implementation cost. If active-active mode cannot be achieved, there will always be one data center running idle with the same number of machines. And when a failure occurs, no one can guarantee that the cold backup data center can immediately serve external requests.
However, the maintenance cost of active-active mode is also high. Data synchronization between data centers often stops due to network latency or data conflicts, ultimately leading to inconsistent data between the two data centers. Fortunately, Otter has taken many measures to ensure data integrity in most cases and reduce the difficulty of implementing active-active mode in the same city.
Even so, during the operation of the business, we still need to manually comb through the business to avoid multiple data centers simultaneously modifying the same data. In this regard, we can use HttpDNS scheduling to make a user active in one data center for a certain period of time, which can reduce the occurrence of data conflicts.
For services with frequent modifications and high contention, general practice is to perform overall transaction execution locally in the data center to prevent synchronization errors caused by simultaneous modifications across data centers.
We believe that in the future, there will be better solutions to the synchronization of multi-active data centers with the development of the industry. That’s all for today’s content. I look forward to interacting and exchanging ideas with you in the comments section!
Thought Question #
If the synchronized link of Otter is circular, how can we ensure that data does not keep synchronizing indefinitely?