20 Summary Two - 10 Key Questions of Distributed Architectural Design #

Hello, I am Ouchuangxin.

In the previous sections, we focused on domain modeling, microservice design, and frontend design methods. When combined, they form a comprehensive solution for building a middle platform. Most middle platforms are built on distributed microservice architectures, which are worth our attention and consideration in enterprise-level digital transformations.

We should not only focus on the enterprise’s business model, business boundaries, and the integration of the frontend and middle platform, but also pay attention to data technology systems, microservice design, and other multi-domain designs and collaborations, such as multi-tenancy. Taking into account implementation experience and considerations, today we will discuss several key questions in the context of distributed architecture.

1. What kind of distributed database to choose? #

The application scenarios of data in distributed architecture are much more complex than centralized architecture, and it will generate many data-related problems. When it comes to data, the first thing is to choose the appropriate distributed database.

Distributed databases mostly adopt the approach of data multi-replica to achieve high-performance, multi-active, and disaster recovery access to data. Currently, there are mainly three different solutions for distributed databases. Their main differences lie in the way they handle multi-replica data and the database middleware used.

1. Integrated distributed database solution #

It supports multi-replica data and high availability and often employs the Paxos protocol. Writing to multiple data replicas at once, success is achieved when the majority of replicas have the write operation successful. Representative products include OceanBase and GaussDB.

2. Centralized database + database middleware solution #

It is a solution that combines a centralized database with a database middleware. Data routing and global data management are implemented through the database middleware. The database middleware and the database are deployed separately, and the consistency of the primary-replica data is achieved through the synchronization mechanism of the database itself. The main centralized databases are MySQL and PostgreSQL databases. Many solutions have been derived from these two databases, such as the open-source database middleware MyCat+MySQL solution, TBase (based on PostgreSQL but with significant encapsulation and modifications), and other solutions.

3. Centralized database + library for sharding solution #

It is a lightweight database middleware solution. The sharding library is actually a basic JAR package that is deployed with the application software to achieve data routing and data aggregation. It is suitable for relatively simple read-write transaction scenarios and relatively weak in terms of strong consistency and aggregate analysis queries. The typical underlying sharding components include ShardingSphere.

Summary: These three solutions have different implementation costs and differences in business support capabilities. The integrated distributed database is mainly developed by Internet giants and has a very strong data processing capability, but it mostly requires a cloud computing infrastructure, and the implementation cost and technical requirements are relatively high. The centralized database + database middleware solution has moderate implementation cost and technical requirements, and it can meet the business requirements of medium to large enterprises. The third sharding library solution can handle simple business scenarios, and the cost and skill requirements are relatively low. When choosing a database, we need to consider our own capabilities, costs, and business needs in order to choose the appropriate solution.

2. How to design sharding keys for database sharding? #

After choosing a distributed database, the next step is to consider database sharding, in which the design of sharding keys plays a crucial role.

For the key business operations involving customer interactions, I suggest using the customer ID as the sharding key. This ensures that the data of the same customer is distributed within the same data shard, avoiding frequent data access across shards. Frequent service calls across data centers or queries across data shards can have a fatal impact on system performance.

Keeping all the customer’s data in the same data shard also makes it easier to provide customer consistency services. For an enterprise, being “customer-centric” in terms of business capabilities starts with being “customer-centric” in terms of data.

Of course, you can also use other business attributes as sharding keys based on your specific business needs, such as organization or user.

3. Data Synchronization and Replication of Databases #

In a microservices architecture, data is further divided. In order to achieve data integration, batch data synchronization and replication between databases are essential. Data synchronization and replication are mainly used for data synchronization between databases, including business data migration, data backup, data replication from different channel core business data to data platforms or data hubs, and integration of different thematic data.

Traditional data transfer methods include ETL tools and scheduled extraction programs, but they have shortcomings in terms of timeliness. Distributed architectures generally use Change Data Capture (CDC) technology based on database logical log incremental data capture. It can achieve near-real-time data replication and transmission, decoupling data processing from application logic, and is easier to use.

There are many database log capture technology components available for mainstream PostgreSQL and MySQL databases. CDC can also be used in domain event-driven design as a technology for obtaining incremental data of domain events.

4. How to handle cross-database queries? #

Cross-database queries are a weakness of distributed databases and can impact query performance. In domain modeling, many entities are distributed across different microservices. However, there are often business requirements that necessitate cross-database queries between these entities.

There are two types of scenarios for cross-database queries:

1. The first type involves querying data based on a specific dimension or a specific domain. For example, querying data based on a customer’s comprehensive business view, which requires querying multiple microservices across different business lines.

2. The second type involves querying data across different tables, such as joining an organization table with a business table that are distributed in different microservices.

How can we solve these two types of cross-database queries?

For the first scenario, since the data is distributed across different microservices, it is not possible to query this data directly across multiple microservices. In this case, you can establish a distributed database based on a specific theme or topic, which consolidates data from various business microservices. By using database log capture technology, data can be collected and consolidated in near-real-time from different microservices into the theme database. During data consolidation, you can perform data association in advance (such as combining data from multiple tables into one wide table) or establish a data model. Construct a query microservice based on the theme database. With this setup, you can obtain comprehensive business data for a customer in a single query. You can also design appropriate sharding keys based on themes or scenarios to improve query efficiency.

For the second scenario, when dealing with cross-database queries between tables that are not in the same database, you can use small-table broadcasting. This involves adding a redundant code sub-table in the business database. When the main table data changes, you can asynchronously refresh all sub-table data through domain event-driven patterns using message publishing and subscribing. This approach can not only solve table-to-table cross-database queries but also improve query efficiency.

5. How to handle high-frequency hot data? #

For high-frequency hot data, such as commodity and institution code data, which are accessed by multiple applications simultaneously, they need to have high concurrency response capabilities. They can bring huge pressure to the database, affecting system performance.

A common approach is to load these high-frequency hot data from the database into a cache such as Redis, and provide data access services through the cache. This can not only reduce the pressure on the database, but also improve data access performance.

In addition, for high-frequency data that requires fuzzy queries, you can also choose to use search engines like ElasticSearch.

Caches are like seasonings, they require little investment and have quick effects, greatly enhancing the user experience.

6. Handling Data Dependencies between Microservices #

In microservice design, it is common to encounter data that needs to be associated with data from preceding microservices. For example, in the insurance industry, after the insurance microservice generates a policy, the policy will be associated with preceding insurance application data. In e-commerce, a shipping order will be associated with preceding order data. Since the associated data is scattered across preceding microservices, you cannot establish data associations by accessing different microservice databases.

How can we solve this issue of entity associations between preceding and succeeding microservices?

Generally speaking, preceding and succeeding data are related to domain events. You can use the domain event handling mechanism to transfer and duplicate the preceding data to the current microservice database as needed through domain event entities.

You can design preceding data as entities or value objects that are referenced by the current entity. When designing, you need to consider the following: if the preceding data can only be modified as a whole in the current microservice and does not require querying or statistical analysis, you can design it as a value object; if the preceding data consists of multiple records and requires querying or statistical analysis, you can design it as an entity.

In this way, in the shipping microservice, you can retrieve the list data of the preceding orders and shipping orders all at once and provide the entire dataset to the frontend application, reducing cross-microservice calls. If the preceding data is designed as an entity, you can also use it as a query condition to perform multidimensional comprehensive data queries within the local microservice. Only when necessary, retrieve the detailed data of the preceding entity from the preceding microservice. This approach ensures data integrity, reduces microservice dependencies, minimizes cross-microservice calls, and improves system performance.

7. Data Middle Platform and Enterprise-level Data Integration #

Although the distributed microservices architecture improves application elasticity and high availability, the original centralized data becomes fragmented as a result of microservices decomposition, leading to difficulties in data integration and enterprise-level data utilization. You can solve the problem of data application and integration in a distributed architecture by implementing a data middle platform.

You can build a data middle platform in three steps.

First, unify the data standards and complete the collection and storage of different microservices and channel business data to solve the problems of data fragmentation and primary data sharing.

Second, establish a thematic data model to process the data according to different themes and scenarios, and build data views for different themes, such as unified customer view, agent view, and channel view.

Third, establish a business-driven data system to support business and business model innovation.

The data middle platform is not limited to analytical scenarios, but also applicable to transactional scenarios. You can build it on top of the data warehouse and data platform, and provide it to the front-end business after platformization, in order to support transactional scenarios.

VIII. BFF and Enterprise Business Orchestration and Collaboration #

Enterprise business processes are often carried out by multiple microservices cooperating together, with each individual microservice serving a specific function like a building block. So, how do we organize these microservices to achieve enterprise business orchestration and collaboration?

One approach is to add a layer of BFF microservices (Backend for Frontends) between the microservices and the frontend applications. The main responsibility of BFF is to handle service composition and orchestration among microservices. On the other hand, application services within microservices handle service composition and orchestration internally. What are the differences between the two?

The BFF layer sits on top of the mid-tier microservices and mainly focuses on coordinating services between microservices. Application services, on the other hand, primarily handle service composition and orchestration within the microservice. When designing the architecture, it is important to push reusable service capabilities to lower layers. This enables capability reuse and avoids frequent cross-center service calls.

BFF acts as a gear, aligning the coordination between frontend applications and microservices. It achieves this by providing Façade services to adapt to different frontends and organizing and coordinating microservices through service composition and orchestration. BFF microservices can be synchronized and released in coordination with frontend application versions according to requirements and process changes. This prevents frequent modifications and releases of mid-tier microservices to adapt to frontend requirements, thereby ensuring the stability of the core business logic of the microservices.

If your BFF is powerful enough, it can serve as a business capability platform that integrates the capabilities of different mid-tier microservices and is oriented towards multi-channel applications.

9. Distributed Transactions or Event-Driven Mechanism? #

In a distributed architecture, internal calls that used to be within a monolithic system become distributed calls. If an operation involves modifying data in multiple microservices, it will result in data consistency issues. There are two types of data consistency - strong consistency and eventual consistency - with different implementation approaches and costs.

For business scenarios that require real-time strong consistency, you can use distributed transactions. However, distributed transactions come with a performance cost. During design, we need to balance business decomposition, data consistency, performance, and implementation complexity, and try to avoid the occurrence of distributed transactions when possible.

The asynchronous approach of domain event-driven design is a commonly used method in distributed architecture. It can solve eventual consistency issues in non-real-time scenarios. Domain events, published and subscribed based on message middleware, can decouple microservices effectively. By smoothing out traffic peaks and valleys, it can reduce the stress of real-time database access, improving business throughput and processing capability. You can also improve database access performance through event-driven design and achieve read-write separation. For scenarios that require eventual consistency, I recommend using the domain event-driven design approach.

Ten. Design for Multi-Center and Multi-Active #

High availability in distributed architecture is mainly achieved through multi-active design. Multi-center and multi-active is a very complex engineering task. Below, I will mainly list several key designs.

1. Choosing the appropriate distributed database. The database should support multi-data center deployment, meet the requirements of multiple data copies, underlying data replication and synchronization technologies, as well as the timeliness requirements for data recovery.

2. Modular architecture design. Business units consisting of several applications serve as the basic unit of deployment, enabling both local and remote multi-active deployment as well as cross-center elastic scaling. Each unit has self-contained business functions, and all business processes can be completed within the unit. Data from any unit has replicas in multiple data centers, ensuring that data is not lost due to failures. Failures in any unit do not affect the normal operation of other similar units. When designing the modular architecture, we should try to avoid cross-data center and cross-unit calls.

3. Access routing. Access routing includes routing at the access layer, application layer, and data layer, ensuring that front-end access can accurately reach the data center and business unit where the business data is located, enabling accurate writing or retrieval of business data from the corresponding database.

4. Global configuration data management. Implement unified management of global configuration data in each data center, and ensure real-time synchronization of global configuration data in each data center to maintain data consistency.

Summary #

Implementing enterprise-level distributed architecture is a very complex system engineering task, involving numerous technical frameworks and methods. Today, I have listed 10 key design areas, each of which is actually very complex and requires a lot of investment and research. When implementing, you and your company need to choose suitable technological components and implementation plans based on your own situation.