11 Spring Web Body Transformation Common Errors

11 Spring Web Body Transformation Common Errors #

Hello, I’m Fu Jian. In the previous lessons, we learned about URL and Header handling, which are essential in Spring Web development. In this lesson, we will continue to discuss Body handling.

In Spring, many Body conversions are accomplished with the help of third-party encoders and decoders. For example, for JSON parsing, Spring relies on common tools like Jackson and Gson. Therefore, many errors we encounter in Body handling are related to issues with these third-party tools.

Actually, Spring itself doesn’t have many errors, especially with the automatic wrapping provided by Spring Boot and the continuous improvement in handling common issues. This has greatly reduced the errors we can make. However, not every project is directly based on Spring Boot, so there may still be some issues. Let’s review them together.

Case 1: No converter found for return value of type #

When we write web programs using Spring MVC instead of Spring Boot, we usually encounter the error “No converter found for return value of type”. In fact, the code we write is very simple, as shown in the following code:

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Student {
    private String name;
    private Integer age;
}

@RestController
public class HelloController {

    @GetMapping("/hi1")
    public Student hi1() {
        return new Student("xiaoming", Integer.valueOf(12));
    }    
}

And our pom.xml file contains the essential dependencies, with the key configuration as follows:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-webmvc</artifactId>
    <version>5.2.3.RELEASE</version>
</dependency>

But when we run the program and execute the test code, we get the following error:

Error Screenshot

From the above code and configuration, there doesn’t seem to be any obvious errors. So why is this error happening? Is the framework not supported?

Case Analysis #

To understand the reason behind this case, we need to have a preliminary understanding of how responses are handled.

When our request reaches the Controller layer, we get an object, which is new Student("xiaoming", Integer.valueOf(12)) in this case. How should this object be returned to the client?

Should it be returned as JSON, XML, or another encoding type? At this point, a decision is needed. We can find the key code for this decision in the method AbstractMessageConverterMethodProcessor#writeWithMessageConverters:

HttpServletRequest request = inputMessage.getServletRequest();
List<MediaType> acceptableTypes = getAcceptableMediaTypes(request);
List<MediaType> producibleTypes = getProducibleMediaTypes(request, valueType, targetType);

if (body != null && producibleTypes.isEmpty()) {
    throw new HttpMessageNotWritableException(
        "No converter found for return value of type: " + valueType);
}

List<MediaType> mediaTypesToUse = new ArrayList<>();
for (MediaType requestedType : acceptableTypes) {
    for (MediaType producibleType : producibleTypes) {
        if (requestedType.isCompatibleWith(producibleType)) {
            mediaTypesToUse.add(getMostSpecificMediaType(requestedType, producibleType));
        }
    }
}

In fact, we have already shown and analyzed the relevant code in a previous lesson, so here we will only provide a brief analysis of the basic logic of the above code:

  1. Check if the ACCEPT header is present in the request header. If it is not present, any type can be used.
  2. Check the encoding types that can be used for the return type (i.e. the Student instance) currently being processed.
  3. Take the intersection of the results obtained from the above two steps to determine how to return the response.

By comparing the code, we can see that if no suitable encoding method is found in step 2, the error mentioned in the case will be thrown. The specific code line is as follows:

if (body != null && producibleTypes.isEmpty()) {
    throw new HttpMessageNotWritableException(
        "No converter found for return value of type: " + valueType);
}

So how are the available encoding types determined? We can further examine the method AbstractMessageConverterMethodProcessor#getProducibleMediaTypes:

protected List<MediaType> getProducibleMediaTypes(
    HttpServletRequest request, Class<?> valueClass, @Nullable Type targetType) {

Set<MediaType> mediaTypes =
       (Set<MediaType>) request.getAttribute(HandlerMapping.PRODUCIBLE_MEDIA_TYPES_ATTRIBUTE);
if (!CollectionUtils.isEmpty(mediaTypes)) {
   return new ArrayList<>(mediaTypes);
}
else if (!this.allSupportedMediaTypes.isEmpty()) {
   List<MediaType> result = new ArrayList<>();
   for (HttpMessageConverter<?> converter : this.messageConverters) {
      if (converter instanceof GenericHttpMessageConverter && targetType != null) {
         if (((GenericHttpMessageConverter<?>) converter).canWrite(targetType, valueClass, null)) {
            result.addAll(converter.getSupportedMediaTypes());
         }
      }
      else if (converter.canWrite(valueClass, null)) {
         result.addAll(converter.getSupportedMediaTypes());
      }
   }
   return result;
}
else {
   return Collections.singletonList(MediaType.ALL);
}

Assuming that the return type is not explicitly specified (for example, through the produces attribute of GetMapping), all registered HttpMessageConverters will be iterated to see if they support the current type, and ultimately all supported types will be returned. But how are these MessageConverters registered?

After Spring MVC (not Spring Boot) starts, we build a bean of type RequestMappingHandlerAdapter to handle routing and request processing.

Specifically, when we use <mvc:annotation-driven/>, we use AnnotationDrivenBeanDefinitionParser to build this bean. During the construction process, we decide which HttpMessageConverters to use in the future. The relevant code is in AnnotationDrivenBeanDefinitionParser#getMessageConverters:

messageConverters.add(createConverterDefinition(ByteArrayHttpMessageConverter.class, source));
RootBeanDefinition stringConverterDef = createConverterDefinition(StringHttpMessageConverter.class, source);
stringConverterDef.getPropertyValues().add("writeAcceptCharset", false);
messageConverters.add(stringConverterDef);
messageConverters.add(createConverterDefinition(ResourceHttpMessageConverter.class, source));
// Omit other non-relevant code
if (jackson2Present) {
   Class<?> type = MappingJackson2HttpMessageConverter.class;
   RootBeanDefinition jacksonConverterDef = createConverterDefinition(type, source);
   GenericBeanDefinition jacksonFactoryDef = createObjectMapperFactoryDefinition(source);
   jacksonConverterDef.getConstructorArgumentValues().addIndexedArgumentValue(0, jacksonFactoryDef);
   messageConverters.add(jacksonConverterDef);
}
else if (gsonPresent) {
   messageConverters.add(createConverterDefinition(GsonHttpMessageConverter.class, source));
}
// Omit other non-relevant code

Here, we will use some default encoders/decoders, such as StringHttpMessageConverter. However, for types like JSON and XML, we need jackson2Present, gsonPresent, etc., to be true to load the encoder/decoder. Let’s take gsonPresent as an example to see when it becomes true. According to the code:

gsonPresent = ClassUtils.isPresent("com.google.gson.Gson", classLoader);

If we have a dependency on the Gson library, we can add the GsonHttpMessageConverter encoder. Unfortunately, our case does not have any dependencies on JSON libraries, so no JSON-related converters exist in the list of candidates. The final list of candidates is as follows:

As we can see, there are no JSON-related encoders/decoders. For the Student type of return object, none of the above encoders/decoders meet the requirements, so we end up in the following code block:

if (body != null && producibleTypes.isEmpty()) {
   throw new HttpMessageNotWritableException(
         "No converter found for return value of type: " + valueType);
}

This throws a “No converter found for return value of type” error, which matches the actual test results in the case.

Problem Fix #

With this case analysis and source code, we can see that not every type of encoder/decoder is available by default; it depends on the project’s dependencies and whether or not they are supported. To parse JSON, we need to depend on the relevant libraries. So, let’s use Gson as an example to fix this problem:

<dependency>
   <groupId>com.google.code.gson</groupId>
   <artifactId>gson</artifactId>
   <version>2.8.6</version>
</dependency>

We have added the Gson dependency to the pom.xml. Run the program and test the case again, and you will find that the error no longer occurs.

Furthermore, we can also see how the GsonHttpMessageConverter encodes the Student object.

Through this case, we can see that Spring provides us with many useful features, but when these features are intertwined, we may encounter pitfalls. Only by understanding its operation mode can we quickly locate and fix issues.

Case 2: Return Changed Body #

Case 1 helped us solve the parsing problem. However, through continuous practice, we might find that the returned result is no longer the same as before, even though the code remains unchanged. Let’s take a look at the following code:

@RestController
public class HelloController {

    @PostMapping("/hi2")
    public Student hi2(@RequestBody Student student) {
        return student;
    }

}

The above code accepts a Student object as input and returns it as is. We can test it using the following request:

POST http://localhost:8080/springmvc3_war/app/hi2
Content-Type: application/json

{
  "name": "xiaoming"
}

After testing, we will receive the following result:

{
  "name": "xiaoming"
}

However, as the project progresses, we might start returning the following result even though the code remains unchanged:

{
  "name": "xiaoming",
  "age": null
}

In this case, the age field was not initially serialized as part of the response body, but was later serialized as null and returned.

Under what circumstances does this happen? And how can we avoid this problem and maintain consistent behavior?

Case Analysis #

If we encounter the above problem, it is likely caused by the following situation: In the subsequent code development, we directly or indirectly rely on a new JSON parser. For example, the following code snippet introduces a dependency on Jackson:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.6</version>
</dependency>

When multiple Jackson parsers are present, which one will Spring MVC use? The decision can be determined by referring to the following code:

if (jackson2Present) {
    Class<?> type = MappingJackson2HttpMessageConverter.class;
    RootBeanDefinition jacksonConverterDef = createConverterDefinition(type, source);
    GenericBeanDefinition jacksonFactoryDef = createObjectMapperFactoryDefinition(source);
    jacksonConverterDef.getConstructorArgumentValues().addIndexedArgumentValue(0, jacksonFactoryDef);
    messageConverters.add(jacksonConverterDef);
}
else if (gsonPresent) {
    messageConverters.add(createConverterDefinition(GsonHttpMessageConverter.class, source));
}

From the above code, we can see that Jackson takes precedence over Gson. Therefore, without realizing it, our program has switched from Gson encoding/decoding to Jackson. As a result, the behavior may no longer be exactly the same as before.

In terms of the behavior of serializing null fields in this case, let’s examine whether Gson and Jackson behave consistently.

1. For Gson:

By default, GsonHttpMessageConverter constructs a Gson object using new Gson(), and its constructor specifies the relevant configurations:

public Gson() {
    this(Excluder.DEFAULT, FieldNamingPolicy.IDENTITY,
        Collections.<Type, InstanceCreator<?>>emptyMap(), DEFAULT_SERIALIZE_NULLS,
        DEFAULT_COMPLEX_MAP_KEYS, DEFAULT_JSON_NON_EXECUTABLE, DEFAULT_ESCAPE_HTML,
        DEFAULT_PRETTY_PRINT, DEFAULT_LENIENT, DEFAULT_SPECIALIZE_FLOAT_VALUES,
        LongSerializationPolicy.DEFAULT, null, DateFormat.DEFAULT, DateFormat.DEFAULT,
        Collections.<TypeAdapterFactory>emptyList(), Collections.<TypeAdapterFactory>emptyList(),
        Collections.<TypeAdapterFactory>emptyList());
}

From DEFAULT_SERIALIZE_NULLS, we can see that Gson does not serialize null by default.

2. For Jackson:

MappingJackson2HttpMessageConverter constructs an ObjectMapper using Jackson2ObjectMapperBuilder.json().build() by default. It only explicitly specifies the following two configurations:

  • MapperFeature.DEFAULT_VIEW_INCLUSION
  • DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES

By default, Jackson serializes null, so in this case, when age is null, it is still serialized.

From the analysis above, it can be seen that the returned content can indeed change when the dependencies change.

Problem Resolution #

In light of this problem, how can we resolve it? Can we make the behavior of Jackson consistent with Gson when the Jackson dependency is added? This can be achieved by making the following modifications:

@Data
@NoArgsConstructor
@AllArgsConstructor
@JsonInclude(JsonInclude.Include.NON_NULL)
public class Student {
    private String name;
    // or directly on the `age` field: `@JsonInclude(JsonInclude.Include.NON_NULL)`
    private Integer age;
}

We can use the @JsonInclude annotation directly to make the default behavior of Jackson consistent with Gson regarding the handling of null values.

Although the above modification seems simple, what if there are many objects that need to be modified? What if we accidentally miss some objects? Therefore, we can make modifications from a global perspective. The key code for the modification is as follows:

//ObjectMapper mapper = new ObjectMapper();
//mapper.setSerializationInclusion(Include.NON_NULL);

However, how can we modify the ObjectMapper? This object is constructed by MappingJackson2HttpMessageConverter, and it seems that there is no way to modify it. In fact, in a non-Spring Boot program, we can make the following modifications:

@RestController
public class HelloController {

    public HelloController(RequestMappingHandlerAdapter requestMappingHandlerAdapter){
        List<HttpMessageConverter<?>> messageConverters =
                requestMappingHandlerAdapter.getMessageConverters();
        for (HttpMessageConverter<?> messageConverter : messageConverters) {
            if(messageConverter instanceof MappingJackson2HttpMessageConverter ){
                (((MappingJackson2HttpMessageConverter)messageConverter).getObjectMapper()).setSerializationInclusion(JsonInclude.Include.NON_NULL);
            }
        }
    }
    // omitted non-critical code
}

We use auto-injection to obtain the RequestMappingHandlerAdapter, find the Jackson parser, and configure it accordingly.

By using the above two modification approaches, we can ignore the age field when it is null.

Case 3: Required request body is missing #

In Case 1, we were able to parse the request body. However, sometimes we may have some good ideas. For example, for easy querying of problems, we can customize a filter to output the specific request content when a request comes in. The key code is as follows:

public class ReadBodyFilter implements Filter {

    // Omit other non-key code
    @Override
    public void doFilter(ServletRequest request,
                         ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        String requestBody = IOUtils.toString(request.getInputStream(), "utf-8");
        System.out.println("print request body in filter:" + requestBody);
        chain.doFilter(request, response);
    }

}

Then, we can add this filter to web.xml and configure it as follows:

<filter>
  <filter-name>myFilter</filter-name>
  <filter-class>com.puzzles.ReadBodyFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>myFilter</filter-name>
  <url-pattern>/app/*</url-pattern>
</filter-mapping>

Let’s test the interface defined in the Controller layer again:

@PostMapping("/hi3")
public Student hi3(@RequestBody Student student) {
    return student;
}

When running the test, we will find the following log:

print request body in filter:{- “name”: “xiaoming”,- “age”: 10- }- 25-Mar-2021 11:04:44.906 璀﹀憡 [http-nio-8080-exec-5] org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.logException Resolved [org.springframework.http.converter.HttpMessageNotReadableException: Required request body is missing: public com.puzzles.Student com.puzzles.HelloController.hi3(com.puzzles.Student)]

As we can see, the request body is indeed output in the request, but the subsequent operation directly throws an error with the error message: Required request body is missing.

Case analysis #

To understand the root cause of this error, you need to know where this error is thrown from. Re-examining the code related to converting the request body, there is such a key logic (refer to RequestResponseBodyMethodProcessor#readWithMessageConverters):

protected <T> Object readWithMessageConverters(NativeWebRequest webRequest, MethodParameter parameter,
      Type paramType) throws IOException, HttpMediaTypeNotSupportedException, HttpMessageNotReadableException {
   HttpServletRequest servletRequest = webRequest.getNativeRequest(HttpServletRequest.class);
   ServletServerHttpRequest inputMessage = new ServletServerHttpRequest(servletRequest);
   // Read the body and convert it
   Object arg = readWithMessageConverters(inputMessage, parameter, paramType);
   if (arg == null && checkRequired(parameter)) {
      throw new HttpMessageNotReadableException("Required request body is missing: " +
            parameter.getExecutable().toGenericString(), inputMessage);
   }
   return arg;
}
protected boolean checkRequired(MethodParameter parameter) {
   RequestBody requestBody = parameter.getParameterAnnotation(RequestBody.class);
   return (requestBody != null && requestBody.required() && !parameter.isOptional());
}

When @RequestBody is used and it is required, if the parsed body is null, an error will be thrown with the message “Required request body is missing”.

So we need to continue tracing the code to find out under what circumstances the body will be returned as null. The key code is in AbstractMessageConverterMethodArgumentResolver#readWithMessageConverters:

protected <T> Object readWithMessageConverters(HttpInputMessage inputMessage, MethodParameter parameter,
      Type targetType){
   // Omit non-key code
   Object body = NO_VALUE;
   EmptyBodyCheckingHttpInputMessage message;
   try {
      message = new EmptyBodyCheckingHttpInputMessage(inputMessage);
      for (HttpMessageConverter<?> converter : this.messageConverters) {
         Class<HttpMessageConverter<?>> converterType = (Class<HttpMessageConverter<?>>) converter.getClass();
         GenericHttpMessageConverter<?> genericConverter =
               (converter instanceof GenericHttpMessageConverter ? (GenericHttpMessageConverter<?>) converter : null);
         if (genericConverter != null ? genericConverter.canRead(targetType, contextClass, contentType) :
               (targetClass != null && converter.canRead(targetClass, contentType))) {
            if (message.hasBody()) {
               // Omit non-key code: read and convert the body
            else {
               // Process the case where there is no body, and return null by default
               body = getAdvice().handleEmptyBody(null, message, parameter, targetType, converterType);
            }
            break;
         }
      }

}
}
catch (IOException ex) {
throw new HttpMessageNotReadableException("I/O error while reading input message", ex, inputMessage);
}
//省略非关键代码
return body;
}

When there is no body in the message (message.hasBody() is false), the body is considered null. Let's investigate the definition of the message itself, which is a type called EmptyBodyCheckingHttpInputMessage that wraps the request headers and body stream. Its code implementation is as follows:

public EmptyBodyCheckingHttpInputMessage(HttpInputMessage inputMessage) throws IOException { this.headers = inputMessage.getHeaders(); InputStream inputStream = inputMessage.getBody(); if (inputStream.markSupported()) { //省略其他非关键代码 } else { PushbackInputStream pushbackInputStream = new PushbackInputStream(inputStream); int b = pushbackInputStream.read(); if (b == -1) { this.body = null; } else { this.body = pushbackInputStream; pushbackInputStream.unread(b); } } } public InputStream getBody() { return (this.body != null ? this.body : StreamUtils.emptyInput()); }


The condition for determining an empty body is based on the value of pushbackInputStream.read(), which returns -1 when no data can be read.

At this point, you might have a question: If there is a body, doesn't executing read() consume some of the data in the body? Yes, that's correct. That's why I've used pushbackInputStream.unread(b) to return the read data back, so the presence of a body can be determined while ensuring its integrity.

Based on this analysis, along with the previous example, you should be able to understand the reasons for the missing body:

1. The message itself doesn't have a body.
2. There is a body, but the stream it represents has already been read before.

Clearly, our example falls into the second category, where we've already consumed the body in the filter. The key code is as follows:

>//request is a ServletRequest
>String requestBody = IOUtils.toString(request.getInputStream(), "utf-8");

In this case, as a regular stream, there is no data left for subsequent converters to read.

### Problem Resolution

So we can simply remove the code that reads the body in the filter, and subsequent operations will be able to access the data again. But that doesn't satisfy our requirement. So what can we do in such a situation? Here, I will provide the solution directly, which is to define a RequestBodyAdviceAdapter bean:

@ControllerAdvice public class PrintRequestBodyAdviceAdapter extends RequestBodyAdviceAdapter { @Override public boolean supports(MethodParameter methodParameter, Type type, Class> aClass) { return true; } @Override public Object afterBodyRead(Object body, HttpInputMessage inputMessage,MethodParameter parameter, Type targetType, Class> converterType) { System.out.println(“print request body in advice:” + body); return super.afterBodyRead(body, inputMessage, parameter, targetType, converterType); } }


As you can see, the method afterBodyRead is named appropriately, and the body here has already been converted from the data stream.

So how does it work? We can look at the following code reference (based on AbstractMessageConverterMethodArgumentResolver#readWithMessageConverters):

protected Object readWithMessageConverters(HttpInputMessage inputMessage, MethodParameter parameter, Type targetType){ //省略其他非关键代码
if (message.hasBody()) { HttpInputMessage msgToUse = getAdvice().beforeBodyRead(message,parameter, targetType, converterType); body = (genericConverter != null ? genericConverter.read(targetType, contextClass, msgToUse) :((HttpMessageConverter)converter).read(targetClass, msgToUse)); body = getAdvice().afterBodyRead(body, msgToUse, parameter, targetType, converterType); //省略其他非关键代码
} //省略其他非关键代码
return body; }


After a body is parsed, getAdvice() is called to get the RequestResponseBodyAdviceChain, and within this chain, it finds the appropriate advice to execute.

Since we defined PrintRequestBodyAdviceAdapter earlier, its relevant methods are executed. Based on the timing of execution, the body has already been parsed, meaning that the Body object passed to PrintRequestBodyAdviceAdapter is no longer a stream but an already parsed object.

By using the advice approach described above, we satisfy similar requirements and ensure the correct execution of the program. As for other solutions, you can consider them yourself.
## Key Review

Through the study of this lesson, it is believed that you already have an understanding of common errors related to body parsing in Spring Web. Let's review the key points:

1. Different bodies require different encoders and decoders, and the specific usage is negotiated as follows:

   - Check whether the ACCEPT header exists in the request header. If not, any type can be used.
   - Check the encoding types that can be used for the return type (such as the Student instance).
   - Take the intersection of the results obtained from the above two steps to determine the method of return.

2. In non-Spring Boot programs, the JSON encoder and decoder may not be built-in and need to be added with relevant JAR files to automatically depend on them. The automatic dependency is implemented by checking whether the Class exists: when the relevant JAR is added as a dependency, the crucial Class exists and the corresponding encoding and decoding functionality is provided.

3. Different implementations of encoders and decoders (such as JSON tools Jackson and Gson) may have some differences in details. Therefore, you must pay attention to whether adding a new JAR will cause changes in the default encoder and decoder, thus affecting some local behaviors.

4. When trying to read the HTTP Body, you should note that the Body itself is a stream object and cannot be read multiple times.

The above is the main content of this lesson, I hope it will be helpful to you.
## Thought Question

Through the study of Case 1, we know that when using Spring MVC directly instead of Spring Boot, we need to manually add the JSON dependency in order to parse JSON requests or encode JSON responses. So why don't we need to do this when using Spring Boot?

Looking forward to your thoughts in the comments section!