Answer Compilation Design Compilation Thoughts Answer Collection #

Today, let’s continue analyzing the discussion questions from lectures 21 to 26 of the “Design” module. These questions cover 6 major topics: code duplication, interface design, cache design, production readiness, asynchronous processing, and data storage.

Let’s analyze each question one by one.

21 | Code Duplication: Three Techniques to Eliminate Code Duplication #

Question 1: Besides the template method design pattern, the observer pattern is also commonly used to reduce code duplication (and it promotes loose coupling). Spring also provides similar tools (click here to see). Can you think of any application scenarios for the observer pattern?

Answer: Actually, just like using MQ to decouple systems and their calls, we can also use the observer pattern to decouple the calls between components within an application, especially when the application is a monolith. Apart from promoting loose coupling between components, the observer pattern is also beneficial for eliminating code duplication.

This is because a complex business logic inevitably involves a lot of calls to other components. Although we don’t duplicate the code that handles the internal logic of these components, these complex calls themselves constitute duplicated code.

We can consider abstracting the code logic and abstracting many events around which the processing takes place. In this way, the processing mode changes from “imperative” to “environment-aware.” Each component is like living in a scene, perceiving various events in the scene, and then publishing the processing results as another event.

Through this abstraction, the call logic between complex components becomes “event abstraction + event publishing + event subscription,” making the entire code more simplified.

To further clarify, besides the observer pattern, we often hear about the publish-subscribe pattern. What’s the difference between them?

In fact, the observer pattern can also be called the publish-subscribe pattern. However, strictly speaking, the former is loosely coupled, while the latter requires the involvement of an MQ broker to achieve full decoupling between publishers and subscribers.

Question 2: Regarding bean property copying tools, besides the simple usage of Spring’s BeanUtils utility class, do you know any other object mapping libraries? What are their functionalities?

Answer: Among the many object mapping tools, MapStruct stands out. It is based on JSR 269, which is a Java annotation processor (you can think of it as a compile-time code generator). It uses pure Java methods instead of reflection for property assignment and achieves compile-time type safety.

If you’re using IntelliJ IDEA, you can further install the IDEA MapStruct Support plugin to enable features like auto-completion and jumping to definition for mapping configurations. You can find more information about this plugin’s specific features here.

22 | Interface Design: The Language of System Dialogues Must be Unified #

Question 1: In the example in the section “Interface responses should clearly indicate the result of the interface’s processing”, the code field in the interface response structure represents the error code of the execution result. For interfaces with complex business scenarios, there may be many error cases, and there may be dozens or even hundreds of code values. Client developers will need to write if-else statements for each error case to handle different interactions, which can be very cumbersome. Do you have any suggestions for improvement? As a server, is it necessary to inform the client of the error code of the interface execution?

Answer: There are two purposes for the server to provide the error code to the client. First, it allows the client to display the error code for troubleshooting purposes. Second, the client can differentiate its interactions based on different error codes.

For the first purpose of facilitating client troubleshooting, the server should properly converge and organize the error codes, instead of blindly throwing all the possible error codes from various systems and layers of the service to the client as prompts for the user.

My suggestion is to develop an error code service specifically for managing error codes, implementing logic for code translation, classification, and convergence. It may even include a backend system for product teams to input the required error code prompt messages.

In addition, I also suggest that error codes be composed based on certain rules. For example, the first digit of the error code can represent the error type (e.g., A indicates errors from users; B indicates errors from the current system, often due to business logic errors or poor program robustness; C indicates errors from third-party services), the second and third digits can represent the system numbers (e.g., 01 for user service, 02 for merchant service, etc.), and the following three digits can be incremental error code IDs.

As for the second purpose of differentiating interactions based on different error codes, I think it is more practical to adopt a server-driven approach, where the server informs the client on how to handle the errors. Simply put, the client only needs to follow the instructions without necessarily understanding the meanings of the error codes (even if the client displays the error codes, it is only for troubleshooting purposes).

For example, the server’s response can include two fields: actionType and actionInfo. The former represents the interaction action that the client should perform, and the latter represents the information required by the client to complete this interaction action. Among them, actionType can be toast (a message prompt that does not require confirmation), alert (a popup prompt that requires confirmation), redirectView (redirect to another view), redirectWebView (open a web view), etc.; actionInfo can be the toast message, alert message, redirect URL, etc.

By allowing the server to explicitly specify the client’s interaction behavior after requesting the API, there are two main benefits: flexibility and uniformity.

Flexibility lies in two aspects: First, in urgent situations, emergency measures can be taken through redirection. For example, in special cases where urgent logic modifications are required, we can directly switch to an H5 implementation without having to release a new version. Second, we can provide a backend system that allows product managers or operations teams to configure the interaction methods and information (instead of changing the interaction or prompt, which would require a client-side release).

Uniformity: Sometimes we encounter situations where different clients (such as iOS, Android, and frontend) implement interactions inconsistently. If the API response can specify this part of the content, then this problem can be completely avoided.

Question 2: In the example in the section “Consider a version control strategy for interface evolution”, we used the @APIVersion custom annotation to mark it on classes or methods, implementing a unified interface version definition based on URL. Can you use a similar approach (i.e., custom RequestMappingHandlerMapping) to implement a unified version control based on request headers?

Answer: I have updated my implementation in the source code of Lecture 21 on GitHub. You can click here to view it. The main principle is to define your own RequestCondition to match the request header.

public class APIVersionCondition implements RequestCondition<APIVersionCondition> {

    @Getter
    private String apiVersion;

    @Getter
    private String headerKey;

    public APIVersionCondition(String apiVersion, String headerKey) {
        this.apiVersion = apiVersion;
        this.headerKey = headerKey;
    }

    @Override
    public APIVersionCondition combine(APIVersionCondition other) {
        return new APIVersionCondition(other.getApiVersion(), other.getHeaderKey());
    }

    @Override
    public APIVersionCondition getMatchingCondition(HttpServletRequest request) {
        String version = request.getHeader(headerKey);
        return apiVersion.equals(version) ? this : null;
    }

    @Override
    public int compareTo(APIVersionCondition other, HttpServletRequest request) {
        return 0;
    }
}

And customize the RequestMappingHandlerMapping to associate the method with the custom RequestCondition:

public class APIVersionHandlerMapping extends RequestMappingHandlerMapping {

    @Override
    // ... implementation
}

protected boolean isHandler(Class<?> beanType) {

    return AnnotatedElementUtils.hasAnnotation(beanType, Controller.class);

}

@Override

protected RequestCondition<APIVersionCondition> getCustomTypeCondition(Class<?> handlerType) {

    APIVersion apiVersion = AnnotationUtils.findAnnotation(handlerType, APIVersion.class);

    return createCondition(apiVersion);

}

@Override

protected RequestCondition<APIVersionCondition> getCustomMethodCondition(Method method) {

    APIVersion apiVersion = AnnotationUtils.findAnnotation(method, APIVersion.class);

    return createCondition(apiVersion);

}

private RequestCondition<APIVersionCondition> createCondition(APIVersion apiVersion) {

    return apiVersion == null ? null : new APIVersionCondition(apiVersion.value(), apiVersion.headerKey());

}

23 | Cache Design: Caches Can Be a Blessing or a Curse #

Question 1: When talking about cache concurrency issues, we mentioned the problem of hot-spot key back-to-source pressure on the database. If a key is particularly hot, can we distribute the cache query pressure of this hot-spot key to multiple Redis nodes if we use Redis for caching?

Answer: Starting from Redis 4.0, if LFU algorithm is enabled as the maxmemory-policy, the –hotkeys option can be used in conjunction with the redis-cli command-line tool to probe hot-spot keys. In addition, we can use the MONITOR command to collect all commands executed by Redis, and then use the redis-faina tool to analyze hot-spot keys, hot-spot prefixes, and other information.

To alleviate the pressure on a single Redis node caused by hot-spot keys, we can consider adding a random number as a suffix to each key, so that one key is split into multiple keys, effectively partitioning the hot-spot keys.

Besides distributing the pressure on Redis, we can also consider adding another layer of short-term local caching and using the Keyspace notification feature of Redis to handle data synchronization between the local cache and Redis.

Question 2: Large keys are another problem that can occur in data caching. If the value of a key is very large, it may have a huge impact on the performance of Redis. Since Redis adopts a single-threaded model, operations such as querying or deleting large keys may cause Redis to block or even lead to high-availability switching. Do you know how to query large keys in Redis and how to split large keys in the design?

Answer: Large keys in Redis may cause the uneven distribution of memory in the cluster, and operations on large keys may also cause blocking.

To query large keys in Redis, we can use the redis-cli –bigkeys command to probe large keys in real-time. In addition, we can use the redis-rdb-tools tool to analyze the RDB snapshot of Redis and get a CSV file containing information such as the number of bytes, the number of elements, and the maximum element length for each key. Then, we can import this CSV file into MySQL and write SQL to analyze the data.

Regarding large keys, we can optimize them in two ways:

First, consider whether it is necessary to store such a large amount of data in Redis. In general, we store presentation-related data in the cache system, rather than the raw data. For computing raw data, we can consider using other NoSQL databases, such as document-based or search-based databases.

Second, consider splitting keys with a hierarchical structure (such as List, Set, or Hash) into multiple smaller keys to retrieve them independently (or use MGET to retrieve them).

Furthermore, it is worth mentioning that deleting large keys may cause performance issues. Starting from Redis 4.0, we can use the UNLINK command instead of the DEL command to delete large keys in the background. And for versions before 4.0, we can consider using a cursor to delete data in large keys instead of directly using the DEL command. For example, for Hash, we can use HSCAN+HDEL combined with the pipeline feature for deletion.

24 | Is the Production Ready Once the Business Code Is Completed? #

Question 1: Spring Boot Actuator provides a large number of built-in endpoints. What is the difference between an endpoint and a custom @RestController? Can you develop a custom endpoint according to the official documentation?

Answer: Endpoint is a concept abstracted by Spring Boot Actuator for monitoring and configuration. By using the @Endpoint annotation, we can easily develop monitoring points that are automatically exposed via HTTP or JMX by using the @ReadOperation, @WriteOperation, and @DeleteOperation annotations on methods.

If we only want to expose endpoints via HTTP, we can use the @WebEndpoint annotation. If we only want to expose endpoints via JMX, we can use the @JmxEndpoint annotation.

On the other hand, the @RestController annotation is usually used to define business interfaces. If the data needs to be exposed via JMX, we need to manually develop it.

For example, the following code shows how to define an accumulator endpoint that provides both read and increment operations:

@Endpoint(id = "adder")
@Component
public class TestEndpoint {
    private static AtomicLong atomicLong = new AtomicLong();

    // Read the value
    @ReadOperation
    public String get() {
        return String.valueOf(atomicLong.get());
    }

    // Increment the value
    @WriteOperation
    public String increment() {
        return String.valueOf(atomicLong.incrementAndGet());
    }
}

Then, we can operate on this accumulator through HTTP or JMX. This way, we have implemented a custom endpoint that can be manipulated via JMX:

Question 2: When introducing the metrics in the previous question, we saw that InfluxDB stores some application metrics automatically collected by the Micrometer framework. Can you refer to the two JSON files for Grafana configuration in the source code and configure a complete application monitoring dashboard in Grafana with these metrics?

Answer: We can refer to the classes in the binder package of the Micrometer source code to understand some of the metrics that Micrometer automatically collects for us.

• JVM uptime: process.uptime
• System CPU usage: system.cpu.usage
• JVM process CPU usage: process.cpu.usage
• 1-minute system load average: system.load.average.1m
• JVM used memory: jvm.memory.used
• JVM committed memory: jvm.memory.committed
• JVM maximum memory: jvm.memory.max
• JVM thread states: jvm.threads.states
• JVM GC pauses: jvm.gc.pause, jvm.gc.concurrent.phase.time
• Free disk space: disk.free
• Logback log events: logback.events
• Tomcat thread states (maximum, busy, current): tomcat.threads.config.max, tomcat.threads.busy, tomcat.threads.current

For the specific panel configuration, refer to Lesson 24. Here, I will share two small tips that you can use during the configuration.

The first tip is to configure common tags as dropdowns fixed in the page header. Generally, we configure one panel for all applications (we save application name, IP address, and other information for each metric, and this feature can be implemented using Micrometer’s CommonTags, see section 5.2 of the documentation). We can use Grafana’s Variables feature to display application name and IP as two dropdowns and provide an ad hoc filter to freely add filter conditions.

For example:

In the Variables panel, you can see the three variables I configured:

The query statements for the Application and IP variables are:

SHOW TAG VALUES FROM jvm_memory_used WITH KEY = "application_name"

SHOW TAG VALUES FROM jvm_memory_used WITH KEY = "ip" WHERE application_name=~ /^$Application$/

The second tip is to use the GROUP BY feature to display detailed curves. For metrics such as jvm_threads_states and jvm.gc.pause, there are additional state labels that represent different thread or GC pause states. Generally, when displaying charts, we group and display curves based on these state labels.

For example:

The InfluxDB query statement for this configuration is:

SELECT max("value") FROM "jvm_threads_states" WHERE ("application_name" =~ /^$Application$/ AND "ip" =~ /^$IP$/) AND $timeFilter GROUP BY time($__interval), "state" fill(none)

In this example, the values of the application_name and ip conditions are associated with the variables we configured earlier, and the GROUP BY clause is added to group by state. The complete program log output is as follows, which can be seen to be consistent with the structure diagram and detailed explanation of the process we posted earlier:

[21:59:48.625] [http-nio-45678-exec-1] [INFO ] [o.g.t.c.a.r.DeadLetterController:24 ] - Client 发送消息 msg1

[21:59:48.640] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:27 ] - Handler 收到消息：msg1

[21:59:48.641] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:33 ] - Handler 消费消息：msg1 异常，准备重试第1次

[21:59:51.643] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:27 ] - Handler 收到消息：msg1

[21:59:51.644] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:33 ] - Handler 消费消息：msg1 异常，准备重试第2次

[21:59:54.646] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:27 ] - Handler 收到消息：msg1

[21:59:54.646] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1] [INFO ] [o.g.t.c.a.rabbitmqdlx.MQListener:40 ] - Handler 消费消息：msg1 异常，已重试 2 次，发送到死信队列处理！

[21:59:54.649] [org.springframework.amqp.rabbit.RabbitListenerEndpointContainer#1-1] [ERROR] [o.g.t.c.a.rabbitmqdlx.MQListener:62 ] - DeadHandler 收到死信消息： msg1

Next, let’s compare this implementation with the Spring retry in Lecture 25. In fact, there are significant differences between these two implementation approaches, which can be summarized in the following two points.

Firstly, the Spring retry retries within the processing logic, sleeping within the thread to delay the retry, and the message will not be resent to the message queue (MQ). In our solution, the failed messages will be sent to RabbitMQ (RMQ) for delayed processing.

Secondly, the Spring retry solution only involves two queues (or exchanges), the normal queue and the dead letter queue (DLQ). In our implementation, it involves three queues - the work queue, the buffer queue (used to store messages waiting for delayed retry), and the dead letter queue (the messages that need to be manually processed).

Of course, if you want to separate the queues storing normal messages from the queues storing messages for retry, you can further split the queues in our solution into four queues - the work queue, the retry queue, the buffer queue (associated with the retry queue as the DLX), and the dead letter queue.

Here I want to emphasize again that although we use the DLX dead letter exchange function of RMQ, we use DLX as the work queue because we utilize its ability to automatically receive expired messages from the buffer queue.

The source code for this part is quite long, so I have directly put it on GitHub. If you are interested, you can click on the link here to view it.

Lecture 26 | Data Storage: How do NoSQL and RDBMS complement each other? #

Question 1: We mentioned that InfluxDB should not contain too many tags. Can you write some test code to simulate this issue and observe InfluxDB’s memory usage?

Answer: We can write the following test code to insert a large number of metrics into InfluxDB. Each metric is associated with 10 tags, and each tag is a random number within 100,000. This approach will cause high series cardinality issues and occupy a large amount of memory in InfluxDB.

@GetMapping("influxdbwrong")
public void influxdbwrong() {
    OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient().newBuilder()
            .connectTimeout(1, TimeUnit.SECONDS)
            .readTimeout(60, TimeUnit.SECONDS)
            .writeTimeout(60, TimeUnit.SECONDS);

    try (InfluxDB influxDB = InfluxDBFactory.connect("http://127.0.0.1:8086", "root", "root", okHttpClientBuilder)) {
        influxDB.setDatabase("performance");
        
        // Insert 100000 records
        IntStream.rangeClosed(1, 100000).forEach(i -> {
            Map<String, String> tags = new HashMap<>();
            
            // Each record has 10 tags with values between 1 and 100000
            IntStream.rangeClosed(1, 10).forEach(j -> tags.put("tagkey" + i, "tagvalue" + ThreadLocalRandom.current().nextInt(100000)));
            
            Point point = Point.measurement("bad")
                    .tag(tags)
                    .addField("value", ThreadLocalRandom.current().nextInt(10000))
                    .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
                    .build();

            influxDB.write(point);
        });
    }
}

However, because InfluxDB’s default parameter configuration limits the number of tag values and the number of database series:

max-values-per-tag = 100000
max-series-per-database = 1000000

This program will quickly encounter errors and will not cause an OOM. You can change these two parameters to 0 to remove this restriction.

By running the program, we can see that InfluxDB consumes a large amount of memory and eventually encounters an OOM error.

Question 2: MongoDB is another commonly used NoSQL database. What do you think are the advantages and disadvantages of MongoDB? In what scenarios is it suitable to be used?

Answer: MongoDB is currently a popular document-oriented NoSQL database. Although MongoDB has transactional functionality after version 4.0, its overall stability is still somewhat lacking compared to MySQL. Therefore, MongoDB is not suitable as the main database for important data, but it can be used to store data that is not highly important, such as logs and crawler data, which have high write concurrency.

Although MongoDB has high write performance, its performance in complex queries is not much better than Elasticsearch. Although MongoDB has sharding functionality, it is still relatively unstable. Therefore, personally, I recommend using Elasticsearch instead of MongoDB in scenarios where the data write volume is not large, updates are infrequent, and transactions are not needed.

That’s the answer to the discussion questions from Lectures 21-26 of our course.

If you have any questions about these topics or the underlying knowledge points, please feel free to leave a comment and discuss it with me. You are also welcome to share today’s content with your friends or colleagues for further discussion.