15 Serialization Once Gone and Back Are You Still the Original You

15 Serialization Once Gone and Back Are You Still the Original You

15 Serialization Once Gone and Back Are You Still the Original You #

Today, let’s talk about the pitfalls and best practices related to serialization.

Serialization is the process of converting an object into a byte stream for easy transmission or storage. Deserialization, on the other hand, is the process of converting a byte stream back into an object. When I talked about file I/O, I mentioned that character encoding is the process of converting characters into binary, and how to convert them depends on the rules specified by the character set. Similarly, the serialization and deserialization of objects also require rules specified by the serialization algorithm.

As for the serialization algorithm, in the past few years, commonly used ones include JDK (Java) serialization and XML serialization. However, the former cannot be used across different languages, and the latter has poor performance (with large time and space overhead). Nowadays, JSON serialization is the most commonly used in RESTful applications, while performance-oriented RPC frameworks (such as gRPC) use protobuf serialization. Both of these methods are cross-language and perform well, making them widely used.

During the architectural design stage, we may focus on algorithm selection, balancing performance, ease of use, and cross-platform compatibility, but there are fewer pitfalls in this area. Typically, common pitfalls related to serialization are more often encountered in business scenarios, such as Redis, parameter, and response serialization and deserialization.

Today, let’s talk about some common pitfalls in serialization during development.

Consistency of Serialization and Deserialization is Important #

When dealing with serialization in business code, it is essential to ensure consistency between the algorithms used for serialization and deserialization. There was a time when I needed to investigate cache hit rate issues and asked for the help of an operations colleague to fetch the keys from Redis. However, they reported that the data stored in Redis appeared to be gibberish, suspecting a possible attack on Redis. In reality, this issue was caused by serialization algorithms. Let’s take a closer look.

In this scenario, the development team used RedisTemplate to operate on Redis for data caching. Compared to Jedis, using RedisTemplate provided by Spring not only eliminates the need to consider connection pooling and is more convenient but also seamlessly integrates with other components such as Spring Cache. If you are using Spring Boot, you can use it directly without any additional configuration.

In order to store data (including keys and values) in Redis, it is necessary to serialize the data into a string using a serialization algorithm. Although Redis supports various data structures, such as hashes, the value for each field is still a string. If the value itself is already a string, is there a convenient way to use RedisTemplate without the need for serialization?

The answer is yes, and it is called StringRedisTemplate.

So, what is the difference between StringRedisTemplate and RedisTemplate? And what caused the gibberish values mentioned earlier? Let’s investigate with these questions in mind.

Let’s write a test code that sets two sets of data to Redis after application initialization. The first time, we will use RedisTemplate to set the key as “redisTemplate” and the value as a User object. The second time, we will use StringRedisTemplate to set the key as “stringRedisTemplate” and the value as a JSON-serialized User object:

@Autowired
private RedisTemplate redisTemplate;

@Autowired
private StringRedisTemplate stringRedisTemplate;

@Autowired
private ObjectMapper objectMapper;

@PostConstruct
public void init() throws JsonProcessingException {
    redisTemplate.opsForValue().set("redisTemplate", new User("zhuye", 36));
    stringRedisTemplate.opsForValue().set("stringRedisTemplate", objectMapper.writeValueAsString(new User("zhuye", 36)));
}

If you think that the only difference between StringRedisTemplate and RedisTemplate is that one reads the value as a string and the other as an object, you are greatly mistaken because the data stored and retrieved with these two methods are completely incompatible.

Let’s conduct a small experiment by using RedisTemplate to read the value with the key “stringRedisTemplate” and using StringRedisTemplate to read the value with the key “redisTemplate”:

log.info("redisTemplate get {}", redisTemplate.opsForValue().get("stringRedisTemplate"));
log.info("stringRedisTemplate get {}", stringRedisTemplate.opsForValue().get("redisTemplate"));

The result is that we are unable to read the values with either method:

[11:49:38.478] [http-nio-45678-exec-1] [INFO ] [.t.c.s.demo1.RedisTemplateController:38  ] - redisTemplate get null
[11:49:38.481] [http-nio-45678-exec-1] [INFO ] [.t.c.s.demo1.RedisTemplateController:39  ] - stringRedisTemplate get null

If you connect to Redis using the redis-cli client tool, you will notice that there is no key named “redisTemplate”, which is why StringRedisTemplate is unable to find any data:

By examining the source code of RedisTemplate, we can see that, by default, RedisTemplate uses the JDK serialization for keys and values:

public void afterPropertiesSet() {
  ...
  if (defaultSerializer == null) {
    defaultSerializer = new JdkSerializationRedisSerializer(
        classLoader != null ? classLoader : this.getClass().getClassLoader());
  }

  if (enableDefaultSerializer) {
    if (keySerializer == null) {
      keySerializer = defaultSerializer;
      defaultUsed = true;
    }

    if (valueSerializer == null) {
      valueSerializer = defaultSerializer;
      defaultUsed = true;
    }

    if (hashKeySerializer == null) {
      hashKeySerializer = defaultSerializer;
      defaultUsed = true;
    }

    if (hashValueSerializer == null) {
      hashValueSerializer = defaultSerializer;
      defaultUsed = true;
    }
  }
  ...
}

The seemingly gibberish string “\xac\xed\x00\x05t\x00\rredisTemplate” that you see in redis-cli is actually the result of the string “redisTemplate” being serialized using the JDK serialization. This answers the earlier question about the gibberish values. When RedisTemplate attempts to read the data with the key “stringRedisTemplate”, it also tries to deserialize this string using the JDK serialization, resulting in the failure to retrieve the data.

On the other hand, StringRedisTemplate uses String serialization for keys and values, and both must be strings:

public class StringRedisTemplate extends RedisTemplate<String, String> {

  public StringRedisTemplate() {
    setKeySerializer(RedisSerializer.string());
    setValueSerializer(RedisSerializer.string());
    setHashKeySerializer(RedisSerializer.string());
    setHashValueSerializer(RedisSerializer.string());
  }
}

public class StringRedisSerializer implements RedisSerializer<String> {

  @Override
  public String deserialize(@Nullable byte[] bytes) {
    return (bytes == null ? null : new String(bytes, charset));
  }

  @Override
  public byte[] serialize(@Nullable String string) {
    return (string == null ? null : string.getBytes(charset));
  }
}

Now that we have reached this point, we should understand that the data stored with RedisTemplate and StringRedisTemplate is not compatible. The way to fix this issue is to read the data using the method that was used to store it:

When retrieving a value with RedisTemplate, since the value is of type Object, we can simply cast it to a User object for easier use. Although convenient, the keys and values stored in Redis using RedisTemplate are not easily readable.
When retrieving a string with StringRedisTemplate, we need to manually deserialize the JSON string back into a User object.

// Retrieving the value with RedisTemplate does not require deserialization to get the actual object. Although convenient, the keys and values stored in Redis are not easy to read.
User userFromRedisTemplate = (User) redisTemplate.opsForValue().get("redisTemplate");
log.info("redisTemplate get {}", userFromRedisTemplate);

// Using StringRedisTemplate, although the Key is normal, the Value needs to be manually serialized into a string
User userFromStringRedisTemplate = objectMapper.readValue(stringRedisTemplate.opsForValue().get("stringRedisTemplate"), User.class);
log.info("stringRedisTemplate get {}", userFromStringRedisTemplate);

This way, you can get the correct output:

[13:32:09.087] [http-nio-45678-exec-6] [INFO] [.t.c.s.demo1.RedisTemplateController:45] - redisTemplate get User(name=zhuye, age=36)
[13:32:09.092] [http-nio-45678-exec-6] [INFO] [.t.c.s.demo1.RedisTemplateController:47] - stringRedisTemplate get User(name=zhuye, age=36)

You may say that using RedisTemplate to get the Value is convenient, but the Key and Value are not easy to read; while using StringRedisTemplate, the Key is a regular string, but the Value needs to be manually serialized into a string. Is there a better way to achieve both goals?

Of course, there is. You can define your own serialization method for the Key and Value of RedisTemplate: Use RedisSerializer.string() (which is StringRedisSerializer) for Key serialization and use Jackson2JsonRedisSerializer for Value serialization.

@Bean
public <T> RedisTemplate<String, T> redisTemplate(RedisConnectionFactory redisConnectionFactory) {

    RedisTemplate<String, T> redisTemplate = new RedisTemplate<>();

    redisTemplate.setConnectionFactory(redisConnectionFactory);

    Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);

    redisTemplate.setKeySerializer(RedisSerializer.string());

    redisTemplate.setValueSerializer(jackson2JsonRedisSerializer);

    redisTemplate.setHashKeySerializer(RedisSerializer.string());

    redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);

    redisTemplate.afterPropertiesSet();

    return redisTemplate;
}

Write some code to test the storage and retrieval. Inject a field called userRedisTemplate with the type RedisTemplate. In the right2 method, use the injected userRedisTemplate to store a User object, and then retrieve this object using both userRedisTemplate and StringRedisTemplate.

@Autowired
private RedisTemplate<String, User> userRedisTemplate;

@GetMapping("right2")
public void right2() {

    User user = new User("zhuye", 36);

    userRedisTemplate.opsForValue().set(user.getName(), user);

    Object userFromRedis = userRedisTemplate.opsForValue().get(user.getName());

    log.info("userRedisTemplate get {} {}", userFromRedis, userFromRedis.getClass());
    log.info("stringRedisTemplate get {}", stringRedisTemplate.opsForValue().get(user.getName()));
}

At first glance, everything looks fine. StringRedisTemplate successfully retrieved the data we stored:

[14:07:41.315] [http-nio-45678-exec-1] [INFO] [.t.c.s.demo1.RedisTemplateController:55] - userRedisTemplate get {name=zhuye, age=36} class java.util.LinkedHashMap
[14:07:41.318] [http-nio-45678-exec-1] [INFO] [.t.c.s.demo1.RedisTemplateController:56] - stringRedisTemplate get {"name":"zhuye","age":36}

You can also find the Key as a pure string and the Value as a JSON-serialized User object in Redis:

![img](../images/ac20bd2117053fafee390bbb6ce1eccc.png)

However, there is a hitch here. As shown in the first log output, the Value obtained from userRedisTemplate is of type LinkedHashMap and does not match the User type set in the generic of RedisTemplate.

If we change the variable type of the Value obtained from Redis from Object to User in the code, there won't be any compilation errors, but a ClassCastException will occur:

java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to org.geekbang.time.commonmistakes.serialization.demo1.User

The fix is to modify the code for the custom RedisTemplate and set a custom ObjectMapper for the Jackson2JsonRedisSerializer, enabling the activateDefaultTyping method to include type information as an attribute in the serialized data (of course, you can adjust the JsonTypeInfo.As enum to save type information in other forms):

...
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper objectMapper = new ObjectMapper();

// Include type information as an attribute in the Value
objectMapper.activateDefaultTyping(objectMapper.getPolymorphicTypeValidator(), ObjectMapper.DefaultTyping.NON_FINAL, JsonTypeInfo.As.PROPERTY);
jackson2JsonRedisSerializer.setObjectMapper(objectMapper);
...

Alternatively, you can use the RedisSerializer.json() shortcut method, which internally uses the GenericJackson2JsonRedisSerializer and directly sets the type as an attribute in the Value:

redisTemplate.setKeySerializer(RedisSerializer.string());

redisTemplate.setValueSerializer(RedisSerializer.json());

redisTemplate.setHashKeySerializer(RedisSerializer.string());

redisTemplate.setHashValueSerializer(RedisSerializer.json());

Restart the program and call the right2 method to test. You will see that the Value obtained from the custom RedisTemplate is of type User (first log output), and the Value stored in Redis includes the fully qualified name of the type (second log output):

[15:10:50.396] [http-nio-45678-exec-1] [INFO] [.t.c.s.demo1.RedisTemplateController:55] - userRedisTemplate get User(name=zhuye, age=36) class org.geekbang.time.commonmistakes.serialization.demo1.User
[15:10:50.399] [http-nio-45678-exec-1] [INFO] [.t.c.s.demo1.RedisTemplateController:56] - stringRedisTemplate get ["org.geekbang.time.commonmistakes.serialization.demo1.User",{"name":"zhuye","age":36}]

Therefore, when deserializing, you can directly obtain a Value of type User.

Through the analysis of the RedisTemplate component, we can see the necessity of using consistent serialization algorithms for reading and writing data that needs to be serialized in Redis, otherwise, it would be like playing the piano to a cow.

Here, let me summarize the four RedisSerializers provided by Spring:

By default, RedisTemplate uses JdkSerializationRedisSerializer, which is JDK serialization and can easily result in garbled characters in Redis.

To improve readability, you can set the serializer for Keys to StringRedisSerializer. However, using RedisSerializer.string() is equivalent to using the UTF_8 encoded StringRedisSerializer, so you need to pay attention to the character set.

If you want the Value to be serialized in JSON format, you can set the serializer for Values to Jackson2JsonRedisSerializer. By default, the type information won't be saved in the Value, even if the generic type of RedisTemplate is set to the actual type. If you want to obtain the real data type directly, you can enable the activateDefaultTyping method of the Jackson ObjectMapper to include type information when serializing the Value.

If you want the Value to be saved in JSON format with type information included, a simpler way is to directly use the RedisSerializer.json() shortcut method to get the serializer.

Pay Attention to the Handling of Extra Fields in Jackson JSON Deserialization #

As I mentioned earlier, by using the activateDefaultTyping method of the JSON serialization tool Jackson, you can write object types during data serialization. In fact, Jackson has many parameters that can control serialization and deserialization. It is a powerful and complete serialization tool. Therefore, many frameworks use Jackson as the JDK serialization tool, such as Spring Web. But for this reason, we need to be careful about the configuration of each parameter when using it.

For example, when developing a Spring Web application, if you customize the ObjectMapper and register it as a Bean, it is very likely to cause the ObjectMapper used by Spring Web to be replaced, leading to bugs.

Let’s take a look at an example. In the beginning, the program was normal, but one day, a developer wanted to modify the behavior of the ObjectMapper to serialize an enumeration as an index value instead of a string value. For example, by default, serializing Color.BLUE in an enumeration Color would result in the string “BLUE”:

@Autowired
private ObjectMapper objectMapper;

@GetMapping("test")
public void test() throws JsonProcessingException {
  log.info("color:{}", objectMapper.writeValueAsString(Color.BLUE));
}

enum Color {
    RED, BLUE
}

So, this developer redefined an ObjectMapper bean and enabled the WRITE_ENUMS_USING_INDEX feature:

@Bean
public ObjectMapper objectMapper(){

    ObjectMapper objectMapper=new ObjectMapper();

    objectMapper.configure(SerializationFeature.WRITE_ENUMS_USING_INDEX,true);

    return objectMapper;
}

After enabling this feature, the enumeration Color.BLUE is serialized into the index value 1:

[16:11:37.382] [http-nio-45678-exec-1] [INFO ] [c.s.d.JsonIgnorePropertiesController:19  ] - color:1

Although the modified logic for serializing enumerations meets the requirements, a large number of 400 errors were thrown online, and the logs also showed many UnrecognizedPropertyExceptions:

JSON parse error: Unrecognized field "ver" (class org.geekbang.time.commonmistakes.serialization.demo4.UserWrong), not marked as ignorable; nested exception is com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "version" (class org.geekbang.time.commonmistakes.serialization.demo4.UserWrong), not marked as ignorable (one known property: "name"])\
 at [Source: (PushbackInputStream); line: 1, column: 22] (through reference chain: org.geekbang.time.commonmistakes.serialization.demo4.UserWrong["ver"])

From the exception message, we can see that this is because when deserializing, there was an additional version property in the original data. Further analysis found that we used the UserWrong class as the input parameter for the wrong method of the Web controller, which only has one name attribute:

@Data
public class UserWrong {

    private String name;

}

@PostMapping("wrong")
public UserWrong wrong(@RequestBody UserWrong user) {

    return user;

}

However, the actual data passed by the client had an additional version property. So, why didn’t we have this problem before?

The problem lies in the fact that when we enable the WRITE_ENUMS_USING_INDEX serialization feature of the custom ObjectMapper, it overwrites the ObjectMapper automatically created by Spring Boot. And this automatically created ObjectMapper sets the deserialization feature FAIL_ON_UNKNOWN_PROPERTIES to false to ensure that no exception is thrown when encountering unknown fields. The source code is as follows:

public MappingJackson2HttpMessageConverter() {
  this(Jackson2ObjectMapperBuilder.json().build());
}

public class Jackson2ObjectMapperBuilder {
...
  private void customizeDefaultFeatures(ObjectMapper objectMapper) {

    if (!this.features.containsKey(MapperFeature.DEFAULT_VIEW_INCLUSION)) {
      configureFeature(objectMapper, MapperFeature.DEFAULT_VIEW_INCLUSION, false);
    }

    if (!this.features.containsKey(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)) {
      configureFeature(objectMapper, DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    }
  }
}

To fix this problem, there are three ways:

The first way is to also disable the FAIL_ON_UNKNOWN_PROPERTIES of the custom ObjectMapper:

@Bean
public ObjectMapper objectMapper(){

    ObjectMapper objectMapper=new ObjectMapper();

    objectMapper.configure(SerializationFeature.WRITE_ENUMS_USING_INDEX,true);

    objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES,false);

    return objectMapper;

}

The second way is to add the @JsonIgnoreProperties annotation with the ignoreUnknown attribute enabled to the custom type, in order to ignore the extra data during deserialization:

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
public class UserRight {

    private String name;

}

The third way is to not customize the ObjectMapper directly, but to modify the functionality of the default ObjectMapper of Spring directly in the configuration file. For example, enable the serialization of enumerations as index numbers directly in the configuration file:

spring.jackson.serialization.write_enums_using_index=true

Or you can directly define a Jackson2ObjectMapperBuilderCustomizer bean to enable the new feature:

@Bean
public Jackson2ObjectMapperBuilderCustomizer customizer(){
    return builder -> builder.featuresToEnable(SerializationFeature.WRITE_ENUMS_USING_INDEX);
}

This case tells us two things:

Jackson has many detailed functional features for serialization and deserialization. We can refer to the official Jackson documentation to learn about these features, see SerializationFeature, DeserializationFeature, and MapperFeature for details.

Ignoring extra fields is a configuration item that we are most likely to encounter when writing business code. Spring Boot provides a convenient global setting during auto-configuration. If you need to set more features, you can directly modify the configuration file spring.jackson.** or set the Jackson2ObjectMapperBuilderCustomizer callback interface to enable more settings, without redefining the ObjectMapper bean.

Be cautious of class constructors when deserializing #

When using Jackson for deserialization, besides being mindful of ignoring additional fields, you should also be cautious of the class constructors. Let’s take a look at an actual case where this caused a problem.

There is an APIResult class that wraps the response body of a REST API (used as the output of a web controller). The success field represents whether the processing was successful, and the code field represents the processing status code, which is of type boolean and int, respectively.

Initially, every time an APIResult was returned, the success field would be set based on the code. If the code was 2000, then success would be true; otherwise, it would be false. Later, to reduce duplicate code, this logic was moved into the constructor of the APIResult class:

@Data
public class APIResultWrong {

    private boolean success;

    private int code;

    public APIResultWrong() {

    }

    public APIResultWrong(int code) {
        this.code = code;
        if (code == 2000) success = true;
        else success = false;
    }
}

After making this change, it was discovered that even with a code of 2000, the APIResult’s success field was still false. For example, if we deserialize the APIResult twice, once with code equal to 1234 and once with code equal to 2000:

@Autowired
ObjectMapper objectMapper;

@GetMapping("wrong")
public void wrong() throws JsonProcessingException {
    log.info("result :{}", objectMapper.readValue("{\"code\":1234}", APIResultWrong.class));
    log.info("result :{}", objectMapper.readValue("{\"code\":2000}", APIResultWrong.class));
}

The log output is as follows:

[17:36:14.591] [http-nio-45678-exec-1] [INFO ] [DeserializationConstructorController:20  ] - result :APIResultWrong(success=false, code=1234)
[17:36:14.591] [http-nio-45678-exec-1] [INFO ] [DeserializationConstructorController:21  ] - result :APIResultWrong(success=false, code=2000)

As we can see, the success field of both APIResult instances is false.

The reason for this problem is that by default, when deserializing, the Jackson framework only calls the no-argument constructor to create objects. If you want to use a custom constructor, you need to specify it using @JsonCreator and set the JSON property name corresponding to the constructor parameter using @JsonProperty:

@Data
public class APIResultRight {
    ...
    @JsonCreator
    public APIResultRight(@JsonProperty("code") int code) {
        this.code = code;
        if (code == 2000) success = true;
        else success = false;
    }
}

Running the program again, we obtain the correct output:

[17:41:23.188] [http-nio-45678-exec-1] [INFO ] [DeserializationConstructorController:26  ] - result :APIResultRight(success=false, code=1234)
[17:41:23.188] [http-nio-45678-exec-1] [INFO ] [DeserializationConstructorController:27  ] - result :APIResultRight(success=true, code=2000)

As we can see, when passing code as 2000, success can be correctly set to true.

Two Big Pitfalls of Using Enum as API Interface Parameters or Return Values #

In the previous example, I demonstrated how to serialize an enum into an index value. However, when it comes to enums, I recommend using them internally in your program rather than as API interface parameters or return values. The reason is that there are two big pitfalls when it comes to serialization and deserialization of enums.

The first pitfall is that an exception will occur when the enum definitions on the client and server sides are inconsistent. For example, suppose the client version of the enum defines four enum values:

@Getter
enum StatusEnumClient {

    CREATED(1, "已创建"),

    PAID(2, "已支付"),

    DELIVERED(3, "已送到"),

    FINISHED(4, "已完成");

    private final int status;

    private final String desc;

    StatusEnumClient(Integer status, String desc) {
        this.status = status;
        this.desc = desc;
    }
}

and the server defines five enum values:

@Getter
enum StatusEnumServer {

    ...

    CANCELED(5, "已取消");

    private final int status;

    private final String desc;

    StatusEnumServer(Integer status, String desc) {
        this.status = status;
        this.desc = desc;
    }
}

To test this, you can use RestTemplate to send a request to the server and have it return an enum value that is not defined on the client side:

@GetMapping("getOrderStatusClient")
public void getOrderStatusClient() {

    StatusEnumClient result = restTemplate.getForObject("http://localhost:45678/enumusedinapi/getOrderStatus", StatusEnumClient.class);
    log.info("result {}", result);

}

@GetMapping("getOrderStatus")
public StatusEnumServer getOrderStatus() {

    return StatusEnumServer.CANCELED;

}

When you access the interface, an exception will occur indicating that the enum value “CANCELED” cannot be found in the StatusEnumClient:

JSON parse error: Cannot deserialize value of type `org.geekbang.time.commonmistakes.enums.enumusedinapi.StatusEnumClient` from String "CANCELED": not one of the values accepted for Enum class: [CREATED, FINISHED, DELIVERED, PAID];

To solve this problem, you can enable the read_unknown_enum_values_using_default_value deserialization feature in Jackson, which uses a default value when the enum value is unknown:

spring.jackson.deserialization.read_unknown_enum_values_using_default_value=true

And add a default value to the enum by annotating it with @JsonEnumDefaultValue:

@JsonEnumDefaultValue
UNKNOWN(-1, "未知");

It is important to note that this enum value must be added to the StatusEnumClient on the client side, as the deserialization uses the client-side enum.

There is also a small pitfall here: simply configuring this will not make RestTemplate use this deserialization feature. You also need to configure RestTemplate to use the MappingJackson2HttpMessageConverter provided by Spring Boot:

@Bean
public RestTemplate restTemplate(MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter) {

    return new RestTemplateBuilder()
            .additionalMessageConverters(mappingJackson2HttpMessageConverter)
            .build();
}

Now, when you request the interface, it will return the default value:

[21:49:03.887] [http-nio-45678-exec-1] [INFO ] [o.g.t.c.e.e.EnumUsedInAPIController:25] - result UNKNOWN

The second pitfall, and the bigger one, is that implementing custom serialization and deserialization for enums is very tricky and may involve bugs in Jackson. For example, consider the following interface that takes a list of enums, adds a CANCELED enum value to the list, and then returns the updated list:

@PostMapping("queryOrdersByStatusList")
public List<StatusEnumServer> queryOrdersByStatus(@RequestBody List<StatusEnumServer> enumServers) {
    enumServers.add(StatusEnumServer.CANCELED);
    return enumServers;
}

If we want to serialize the enum based on its “Desc” field, and pass “已送到” (Delivered) as the input:

An exception will be thrown, indicating that “已送到” is not a valid enum value:

JSON parse error: Cannot deserialize value of type `org.geekbang.time.commonmistakes.enums.enumusedinapi.StatusEnumServer` from String "已送到": not one of the values accepted for Enum class: [CREATED, CANCELED, FINISHED, DELIVERED, PAID]

Clearly, the deserialization is using the enum’s name, and the serialization is the same:

You may know that to make the serialization and deserialization of enums use the “Desc” field, you can annotate the field with @JsonValue and modify the StatusEnumServer and StatusEnumClient as follows:

@JsonValue
private final String desc;

Now, when you try again, it will accept “已送到” as the input, and the output will use the enum’s “Desc” field:

However, if you think this solves the problem perfectly, you are completely wrong. You can try to add the @JsonValue annotation to the status field of int type, which means you want the serialization and deserialization to go through the status field:

@JsonValue
private final int status;

Write a client test, passing in the enum values CREATED and PAID:

@GetMapping("queryOrdersByStatusListClient")
public void queryOrdersByStatusListClient() {

    List<StatusEnumClient> request = Arrays.asList(StatusEnumClient.CREATED, StatusEnumClient.PAID);

    HttpEntity<List<StatusEnumClient>> entity = new HttpEntity<>(request, new HttpHeaders());

    List<StatusEnumClient> response = restTemplate.exchange("http://localhost:45678/enumusedinapi/queryOrdersByStatusList",
            HttpMethod.POST, entity, new ParameterizedTypeReference<List<StatusEnumClient>>() {}).getBody();
    log.info("result {}", response);
}

We can see from the request interface that the values passed in are CREATED and PAID, but the returned values are DELIVERED and FINISHED. Just as the title says, you are no longer the original you:

[22:03:03.579] [http-nio-45678-exec-4] [INFO ] [o.g.t.c.e.e.EnumUsedInAPIController:34  ] - result [DELIVERED, FINISHED, UNKNOWN]

The reason for this problem is that serialization uses the value of the status field, but deserialization does not rely on the status, and still uses the ordinal() index value of the enum. This is a bug that Jackson has not solved so far (until 2.10), and it should be fixed in version 2.11.

As shown in the figure below, when we call the server interface and pass in a non-existent status value 0, it can still be successfully deserialized, and the response returned by the server is 1:

One solution is to use @JsonCreator to force the use of a custom factory method during deserialization, which can use the status field of the enum to obtain the value. We add this code to the StatusEnumServer enum class:

@JsonCreator
public static StatusEnumServer parse(Object o) {
    return Arrays.stream(StatusEnumServer.values()).filter(value->o.equals(value.status)).findFirst().orElse(null);
}

It is important to note that we also need to add the corresponding method for StatusEnumClient. Because in addition to the deserialization involved in the server interface receiving the StatusEnumServer parameter, there will also be a deserialization when converting the return value from the server to a List:

@JsonCreator
public static StatusEnumClient parse(Object o) {
    return Arrays.stream(StatusEnumClient.values()).filter(value->o.equals(value.status)).findFirst().orElse(null);
}

After re-calling the interface, we find that although the result is correct, the non-existent enum value CANCELED on the server side is set to null instead of the UNKNOWN set by @JsonEnumDefaultValue.

We have already solved this problem before by setting the @JsonEnumDefaultValue annotation, but now it reappears:

[22:20:13.727] [http-nio-45678-exec-1] [INFO ] [o.g.t.c.e.e.EnumUsedInAPIController:34  ] - result [CREATED, PAID, null]

The reason is simple. The custom parse method we implemented returns null when the enum value cannot be found.

To completely solve this problem and avoid using @JsonCreator to define a complex factory method in the enum, we can implement a custom deserializer. The following code is a bit complicated, but I added detailed comments:

class EnumDeserializer extends JsonDeserializer<Enum> implements ContextualDeserializer {

    private Class<Enum> targetClass;

    public EnumDeserializer() {
    }

    public EnumDeserializer(Class<Enum> targetClass) {
        this.targetClass = targetClass;
    }

    @Override
    public Enum deserialize(JsonParser p, DeserializationContext ctxt) {

        // Find the field with the @JsonValue annotation in the enum, which is the benchmark field for deserialization
        Optional<Field> valueFieldOpt = Arrays.asList(targetClass.getDeclaredFields()).stream()
                .filter(m -> m.isAnnotationPresent(JsonValue.class))
                .findFirst();
        if (valueFieldOpt.isPresent()) {
            Field valueField = valueFieldOpt.get();
            if (!valueField.isAccessible()) {
                valueField.setAccessible(true);
            }
            // Iterate through the enum items to find the enum item whose field value equals the deserialized string
            return Arrays.stream(targetClass.getEnumConstants()).filter(e -> {
                try {
                    return valueField.get(e).toString().equals(p.getValueAsString());
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
                return false;
            }).findFirst().orElseGet(() -> Arrays.stream(targetClass.getEnumConstants()).filter(e -> {
                // If not found, we need to find the default enum value to replace it, and iterate through all enum items to find the enum item marked by @JsonEnumDefaultValue annotation
                try {
                    return targetClass.getField(e.name()).isAnnotationPresent(JsonEnumDefaultValue.class);
                } catch (Exception ex) {
                    ex.printStackTrace();
                }
                return false;
            }).findFirst().orElse(null));
        }
        return null;
    }

    @Override
    public JsonDeserializer<?> createContextual(DeserializationContext ctxt,
                                                BeanProperty property) throws JsonMappingException {

        targetClass = (Class<Enum>) ctxt.getContextualType().getRawClass();
        return new EnumDeserializer(targetClass);
    }
}

Then register this custom deserializer with Jackson:

@Bean
public Module enumModule() {

    SimpleModule module = new SimpleModule();

    module.addDeserializer(Enum.class, new EnumDeserializer());

    return module;

}

The second pit is finally perfectly solved:

[22:32:28.327] [http-nio-45678-exec-1] [INFO ] [o.g.t.c.e.e.EnumUsedInAPIController:34  ] - result [CREATED, PAID, UNKNOWN]

By doing this, we have solved the problem of using custom fields in serialization and deserialization in enums, as well as the problem of using default values when enum values cannot be found. However, the solution is quite complicated. Therefore, I still recommend using simple data types such as int or String directly in DTOs, rather than using enums with various complex serialization configurations to achieve mapping between enum and enum fields, which will be clearer.

Key Points Review #

Today, based on the two scenarios of input and output of Redis and Web API, I introduced several pitfalls to avoid during serialization and deserialization.

First, it is important to ensure the consistency of the serialization and deserialization algorithms. Different serialization algorithms will produce different outputs, so the same deserialization algorithm must be used to properly handle the serialized data.

Second, Jackson has a lot of serialization and deserialization features that can be used to fine-tune the details of serialization and deserialization. However, be careful not to conflict with the Bean automatically configured by Spring Boot when customizing the ObjectMapper.

Third, when debugging serialization and deserialization issues, three points must be clarified: which component is performing serialization and deserialization, how many times serialization and deserialization occur in the process, and whether it is currently serialization or deserialization.

Fourth, by default, frameworks call the no-argument constructor for deserialization. If you want to call a custom parameterized constructor, you need to inform the framework how to do so. A better approach is to avoid custom constructors for serializable POJOs.

Fifth, it is not recommended to define enumerations in DTOs for cross-service transmission, as it may cause versioning issues and complicate serialization and deserialization, making it prone to errors. Therefore, I only recommend using enumerations internally in the program.

Finally, it is worth noting that if serialization data needs to be used across platforms, not only should the algorithms used on both ends be consistent, but there may also be compatibility issues with data types in different languages. This is also a common pitfall. If you have such requirements, it is advisable to conduct multiple experiments and tests.

I have uploaded the code used today on GitHub, you can click on this link to view it.

Reflection and Discussion #

When discussing the Redis serialization method, we customized the RedisTemplate to use String serialization for the Key and JSON serialization for the Value. This allows the Value obtained from Redis to be directly converted into the desired object type. So, can RedisTemplate be used to store and retrieve data of type Long? Are there any pitfalls involved in this process?

You can take a look at the implementation of the Jackson2ObjectMapperBuilder class (pay attention to the configure method) to analyze what it does besides disabling FAIL_ON_UNKNOWN_PROPERTIES.

Regarding serialization and deserialization, have you encountered any pitfalls? I’m Zhu Ye, and I welcome you to leave a message for me in the comments section to share your experiences. Feel free to share this article with your friends or colleagues for further discussion.