10 Spring Web Header Resolution Common Errors #
Hello, I’m Fu Jian. In this lesson, we will discuss common error cases related to headers in Spring Web development.
In the previous lesson, we reviewed URL-related errors. While the URL is important for an HTTP request, it has limitations in terms of length and the amount of information it can carry. To provide more information, headers are often used. Needless to say, headers are the second most important component, after the URL, as they provide additional information and related capabilities. For example, the Content-Type header specifies the content type of our request or response, making it easier for us to decode. Although the overall process of parsing headers in Spring is similar to that of URLs, headers have their own characteristics. For example, headers are not restricted to appearing only in requests like URLs are. Therefore, handling errors related to headers is different from handling URL errors. Now let’s look at some specific cases.
Case 1: Incorrect Usage of Map Type for Accepting Headers #
In Spring, when parsing headers, we often parse them directly as needed. For example, if we want to use a header called myHeaderName, we would write code like this:
@RequestMapping(path="/hi", method=RequestMethod.GET)
public String hi(@RequestHeader("myHeaderName") String name){
// omitted body processing
};
By defining a parameter and labeling it with @RequestHeader, we specify the name of the header to be parsed. However, if we need to parse multiple headers, the above approach would require more and more parameters. In this case, we generally use a Map to accept all the headers and process them directly. So we might write code like this:
@RequestMapping(path="/hi1", method=RequestMethod.GET)
public String hi1(@RequestHeader() Map map){
return map.toString();
};
After a quick test, everything seems fine. Moreover, the above code conforms to the paradigm of programming to interfaces, by using the Map interface. However, when encountering a specific type of request, the above interface definition will behave unexpectedly. Consider the following request:
GET http://localhost:8080/hi1 - myheader: h1 - myheader: h2
Here, there is a header called myHeader, but it has two values. When we execute the request, we find that the returned result does not include both values. The result looks like this:
{myheader=h1, host=localhost:8080, connection=Keep-Alive, user-agent=Apache-HttpClient/4.5.12 (Java/11.0.6), accept-encoding=gzip,deflate}
How do we understand this common error and the underlying principle behind it? Let’s analyze it in more detail.
Case Analysis #
In fact, when we see this test result, most of us can already deduce the issue. For a multi-value header, in practice, there are generally two ways to represent it. One way is using the following format:
Key: value1, value2
The other way is using the format we see in our test request:
Key: value1 - Key: value2
For the first format, using the Map interface is not a problem. However, if we are using the second format, we will not be able to retrieve all the values. Let’s take a look at how the Map receives all the requests by examining the code.
For parsing a header, there are two main methods implemented in RequestHeaderMethodArgumentResolver and RequestHeaderMapMethodArgumentResolver, both of which inherit from AbstractNamedValueMethodArgumentResolver. However, they are used in different scenarios. We can compare their supportsParameter() methods to understand which scenarios they are suitable for:
In the image above, the method on the left is from RequestHeaderMapMethodArgumentResolver. By comparing them, we can find that for a parameter annotated with @RequestHeader, if its type is a Map, the RequestHeaderMapMethodArgumentResolver is used. Otherwise, RequestHeaderMethodArgumentResolver is generally used.
In our case, it is clear that the parameter type is Map, so the RequestHeaderMapMethodArgumentResolver is naturally used. Let’s continue to see how it parses headers. The key code snippet can be found in resolveArgument():
@Override
public Object resolveArgument(MethodParameter parameter, @Nullable ModelAndViewContainer mavContainer,
NativeWebRequest webRequest, @Nullable WebDataBinderFactory binderFactory) throws Exception {
Class<?> paramType = parameter.getParameterType();
if (MultiValueMap.class.isAssignableFrom(paramType)) {
MultiValueMap<String, String> result;
if (HttpHeaders.class.isAssignableFrom(paramType)) {
result = new HttpHeaders();
}
else {
result = new LinkedMultiValueMap<>();
}
for (Iterator<String> iterator = webRequest.getHeaderNames(); iterator.hasNext();) {
String headerName = iterator.next();
String[] headerValues = webRequest.getHeaderValues(headerName);
if (headerValues != null) {
for (String headerValue : headerValues) {
result.add(headerName, headerValue);
}
}
}
return result;
}
else {
Map<String, String> result = new LinkedHashMap<>();
for (Iterator<String> iterator = webRequest.getHeaderNames(); iterator.hasNext();) {
String headerName = iterator.next();
// Only retrieves one "value"
String headerValue = webRequest.getHeader(headerName);
if (headerValue != null) {
result.put(headerName, headerValue);
}
}
return result;
}
}
For our case, it is not a MultiValueMap, so we enter the else branch. This branch first creates a LinkedHashMap, then puts the headers into the map one by one, and returns it. The actual invocation for retrieving header values is done on line 29, which varies depending on the container. For example, in the Tomcat container, it calls getValue() in MimeHeaders:
public MessageBytes getValue(String name) {
for (int i = 0; i < count; i++) {
if (headers[i].getName().equalsIgnoreCase(name)) {
return headers[i].getValue();
}
}
return null;
}
When there are multiple headers with the same name, as long as we match any one of them, we immediately return. Therefore, in this case, only one value of the header is returned.
In fact, from another perspective, given that we have defined the receiving type as LinkedHashMap, the value’s generic type is String, which is not suitable for organizing multiple values. Therefore, whether from the code or common sense, the code in this case cannot retrieve all the values of myHeader.
Problem Resolution #
Now we need to fix this problem. In the case analysis section, I have actually provided the answer.
In resolveArgument() of RequestHeaderMapMethodArgumentResolver, if our parameter type is MultiValueMap, we generally create a LinkedMultiValueMap and use the following statement to retrieve the header values and add them to the map:
String[] headerValues = webRequest.getHeaderValues(headerName)
Based on the above code snippet, we can conclude that to receive all the headers, we should not use Map directly, but rather use MultiValueMap. We can use the following two approaches to fix this problem:
// Approach 1
@RequestHeader() MultiValueMap map
// Approach 2
@RequestHeader() HttpHeaders map
After running the test again, you will see that the result meets our expectations:
[myheader:"h1", "h2", host:"localhost:8080", connection:"Keep-Alive", user-agent:"Apache-HttpClient/4.5.12 (Java/11.0.6)", accept-encoding:"gzip,deflate"]
Comparatively, Approach 2 is more recommended because it utilizes the commonly used methods for accessing headers, such as directly calling getContentType() to get the Content-Type header value, and so on. It is very convenient to use.
Reflecting on this case, why did we make this mistake? The root cause lies in the fact that we rarely see a header with multiple values, which caused us to inadvertently use the wrong type to accept the header values.
Case 2: Mistakenly Assuming Header Names are Case-Insensitive #
In the HTTP protocol, the names of headers are case-insensitive. When building web applications using various frameworks, we should keep this fact in mind. We can verify this idea. For example, let’s say we have a web service interface as follows:
@RequestMapping(path = "/hi2", method = RequestMethod.GET)
public String hi2(@RequestHeader("MyHeader") String myHeader){
return myHeader;
};
Then, we can test this API endpoint by using the following request:
GET http://localhost:8080/hi2 - myheader: myheadervalue
Furthermore, considering Case 1, we know that we can use a Map to receive all the headers. In this way, can we also ignore the case? Let’s compare it using the following code:
@RequestMapping(path = "/hi2", method = RequestMethod.GET)
public String hi2(@RequestHeader("MyHeader") String myHeader, @RequestHeader MultiValueMap map){
return myHeader + " compare with : " + map.get("MyHeader");
};
Running the previous test request again, we get the following result:
myheadervalue compare with : null
Overall, directly retrieving the header is case-insensitive, but if we retrieve the header from the received Map, it is case-sensitive. If we are not careful, we may easily assume that the header can always be obtained without case differentiation.
So, how should we understand this case?
Case Analysis #
We know that for the definition @RequestHeader("MyHeader") String myHeader, Spring uses the RequestHeaderMethodArgumentResolver for parsing. Referring to the RequestHeaderMethodArgumentResolver#resolveName method:
protected Object resolveName(String name, MethodParameter parameter, NativeWebRequest request) throws Exception {
String[] headerValues = request.getHeaderValues(name);
if (headerValues != null) {
return (headerValues.length == 1 ? headerValues[0] : headerValues);
}
else {
return null;
}
}
From the key invocation request.getHeaderValues(name) to find the header, we can find the fundamental method for searching the header, which is org.apache.tomcat.util.http.ValuesEnumerator#findNext:
private void findNext() {
next=null;
for(; pos< size; pos++ ) {
MessageBytes n1=headers.getName( pos );
if( n1.equalsIgnoreCase( name )) {
next=headers.getValue( pos );
break;
}
}
pos++;
}
In the above method, name is the name of the header to be queried, and we can see that it is case-insensitive here.
As for using a Map to receive all the headers, let’s see if the headers stored and retrieved from the Map are case-insensitive.
Based on the analysis in Case 1, combined with the specific code, we can easily draw the following two conclusions:
1. The headers stored in the Map are case-sensitive.
Referring to the code snippet posted in the analysis section of Case 1, we can see that when storing the header, the required key is obtained from the result of iterating webRequest.getHeaderNames(). The execution process of this method refers to org.apache.tomcat.util.http.NamesEnumerator#findNext:
private void findNext() {
next=null;
for(; pos< size; pos++ ) {
next=headers.getName( pos ).toString();
for( int j=0; j<pos ; j++ ) {
if( headers.getName( j ).equalsIgnoreCase( next )) {
// duplicate.
next=null;
break;
}
}
if( next!=null ) {
// it's not a duplicate
break;
}
}
pos++;
}
Here, we can see that there is no case-insensitivity or conversion work done for the name of the header in the returned result.
2. The headers retrieved from the Map are also case-sensitive.
This can be seen from the fact that the return type is LinkedHashMap, and the get() method of LinkedHashMap is not case-insensitive.
Next, let’s see how to solve the problem.
Issue Resolution #
To retrieve headers from the Map type, we just need to pay attention to the case. The corrected code is as follows:
@RequestMapping(path = "/hi2", method = RequestMethod.GET)
public String hi2(@RequestHeader("MyHeader") String myHeader, @RequestHeader MultiValueMap map){
return myHeader + " compare with : " + map.get("myHeader");
};
Also, you can consider a question: if we use HTTP Headers to receive requests, can we ignore the case when retrieving headers?
You can infer this from its constructor, which looks like this:
public HttpHeaders() {
this(CollectionUtils.toMultiValueMap(new LinkedCaseInsensitiveMap<>(8, Locale.ENGLISH)));
}
Here, it uses LinkedCaseInsensitiveMap instead of a regular LinkedHashMap. Therefore, here case-insensitivity is allowed. Let’s fix it like this:
@RequestMapping(path = "/hi2", method = RequestMethod.GET)
public String hi2(@RequestHeader("MyHeader") String myHeader, @RequestHeader HttpHeaders map){
return myHeader + " compare with : " + map.get("MyHeader");
};
Running the program again, the result will meet our expectations:
myheadervalue compare with : [myheadervalue]
Through this case, we can see that although the HTTP protocol specification allows ignoring case for headers in practice, not all interface methods provided by frameworks are case-insensitive. You must pay attention to this!
Case 3: Attempting to Customize CONTENT_TYPE in Controller #
Unlike headers and URLs, headers can appear in the response. Because of this, some applications try to customize headers for processing. For example, in Spring Boot development using the built-in Tomcat container, there is the following code that sets two headers: one is the commonly used CONTENT_TYPE, and the other is a custom header named “myHeader”.
@RequestMapping(path = "/hi3", method = RequestMethod.GET)
public String hi3(HttpServletResponse httpServletResponse){
httpServletResponse.addHeader("myheader", "myheadervalue");
httpServletResponse.addHeader(HttpHeaders.CONTENT_TYPE, "application/json");
return "ok";
};
If we run the program and test it (accessing GET http://localhost:8080/hi3), we will get the following result:
GET http://localhost:8080/hi3- #
HTTP/1.1 200- myheader: myheadervalue- Content-Type: text/plain;charset=UTF-8- Content-Length: 2- Date: Wed, 17 Mar 2021 08:59:56 GMT- Keep-Alive: timeout=60- Connection: keep-alive
We can see that “myHeader” has been successfully set, but the Content-Type is not set to “application/json” as we wanted, but “text/plain;charset=UTF-8”. Why does this error occur?
Case Analysis #
First, let’s look at the key steps that are executed when attempting to add headers when using the embedded Tomcat container in Spring Boot.
The first step is to look at the org.apache.catalina.connector.Response#addHeader method. The code is as follows:
private void addHeader(String name, String value, Charset charset) {
// omitting other non-critical code
char cc=name.charAt(0);
if (cc=='C' || cc=='c') {
// Check if it's Content-Type. If it is, do not add this header to org.apache.coyote.Response
if (checkSpecialHeader(name, value))
return;
}
getCoyoteResponse().addHeader(name, value, charset);
}
According to the code and comments, a normal header can be added to the header collection. However, if it is a Content-Type, something different happens. It does not add the header in this way, but instead does something else, namely, it calls Response#checkSpecialHeader to set org.apache.coyote.Response#contentType to “application/json”. The key code is as follows:
private boolean checkSpecialHeader(String name, String value) {
if (name.equalsIgnoreCase("Content-Type")) {
setContentType(value);
return true;
}
return false;
}
The final response we receive is as follows:
From the above image, we can see that the Content-Type is not in the Headers, but the Content-Type we set is now a value of the coyoteResponse member. However, this does not mean that it will not be returned later. We can continue to trace the subsequent execution.
After the case code returns “ok”, we need to process the return result. The method executed is RequestResponseBodyMethodProcessor#handleReturnValue, and the key code is as follows:
@Override
public void handleReturnValue(@Nullable Object returnValue, MethodParameter returnType,
ModelAndViewContainer mavContainer, NativeWebRequest webRequest)
throws IOException, HttpMediaTypeNotAcceptableException, HttpMessageNotWritableException {
mavContainer.setRequestHandled(true);
ServletServerHttpRequest inputMessage = createInputMessage(webRequest);
ServletServerHttpResponse outputMessage = createOutputMessage(webRequest);
// Perform encoding conversion for the return value ("ok" in this case) based on the return type
writeWithMessageConverters(returnValue, returnType, inputMessage, outputMessage);
}
In the above code invocation, writeWithMessageConverters converts the return value based on the return value and type, and also does some additional tasks. Here are the key implementation steps:
1. Determine which MediaType to return
Referring to the following key code:
// Determine which MediaType to use for the return value
MediaType selectedMediaType = null;
MediaType contentType = outputMessage.getHeaders().getContentType();
boolean isContentTypePreset = contentType != null && contentType.isConcrete();
// If the contentType header is present, use it as the selectedMediaType
if (isContentTypePreset) {
selectedMediaType = contentType;
}
// If not, calculate which MediaType to use based on the "Accept" header and the return value
else {
HttpServletRequest request = inputMessage.getServletRequest();
List<MediaType> acceptableTypes = getAcceptableMediaTypes(request);
List<MediaType> producibleTypes = getProducibleMediaTypes(request, valueType, targetType);
// omitting other non-critical code
List<MediaType> mediaTypesToUse = new ArrayList<>();
for (MediaType requestedType : acceptableTypes) {
for (MediaType producibleType : producibleTypes) {
if (requestedType.isCompatibleWith(producibleType)) {
mediaTypesToUse.add(getMostSpecificMediaType(requestedType, producibleType));
}
}
}
// omitting other key code
for (MediaType mediaType : mediaTypesToUse) {
if (mediaType.isConcrete()) {
selectedMediaType = mediaType;
break;
}
}
// Omit other key code
}
Here, let me explain. The above code first determines the MediaType of the response based on whether it has the Content-Type header. As analyzed earlier, Content-Type is a special header and is not added to the Header in the Controller layer. Therefore, here, we can only negotiate the final MediaType based on the returned type and the Accept information of the request.
Actually, here the MediaType used is MediaType#TEXT_PLAIN. It is important to note that JSON was not chosen because in cases where both are supported, TEXT_PLAIN has a higher priority by default. The converter has a priority order, as can be seen from the code in WebMvcConfigurationSupport#addDefaultHttpMessageConverters. Therefore, when iterating through the converters using getProducibleMediaTypes() in the above code, the available MediaType is collected in a specific order.
**2. Select the message converter and complete the conversion**
After deciding on the MediaType, we can then select the converter and perform the conversion. The key code is as follows:
```java
for (HttpMessageConverter<?> converter : this.messageConverters) {
GenericHttpMessageConverter genericConverter = (converter instanceof GenericHttpMessageConverter ?
(GenericHttpMessageConverter<?>) converter : null);
if (genericConverter != null ?
((GenericHttpMessageConverter) converter).canWrite(targetType, valueType, selectedMediaType) :
converter.canWrite(valueType, selectedMediaType)) {
// Omit other non-key code
if (body != null) {
// Omit other non-key code
if (genericConverter != null) {
genericConverter.write(body, targetType, selectedMediaType, outputMessage);
} else {
((HttpMessageConverter) converter).write(body, selectedMediaType, outputMessage);
}
}
// Omit other non-key code
}
}
As shown in the code, the decision of which message converter to use is based on three pieces of information: targetType (String), valueType (String), and selectedMediaType (MediaType#TEXT_PLAIN). Common candidate converters can be referred to in the following image:
In this case, the selected converter is StringHttpMessageConverter. When calling the write method in the parent class AbstractHttpMessageConverter, an attempt is made to add the Content-Type. The specific code can be found in AbstractHttpMessageConverter#addDefaultHeaders:
protected void addDefaultHeaders(HttpHeaders headers, T t, @Nullable MediaType contentType) throws IOException {
if (headers.getContentType() == null) {
MediaType contentTypeToUse = contentType;
if (contentType == null || contentType.isWildcardType() || contentType.isWildcardSubtype()) {
contentTypeToUse = getDefaultContentType(t);
} else if (MediaType.APPLICATION_OCTET_STREAM.equals(contentType)) {
MediaType mediaType = getDefaultContentType(t);
contentTypeToUse = (mediaType != null ? mediaType : contentTypeToUse);
}
if (contentTypeToUse != null) {
if (contentTypeToUse.getCharset() == null) {
// Attempt to add charset
Charset defaultCharset = getDefaultCharset();
if (defaultCharset != null) {
contentTypeToUse = new MediaType(contentTypeToUse, defaultCharset);
}
}
headers.setContentType(contentTypeToUse);
}
}
// Omit other non-key code
}
Based on the analysis of the case and the corresponding code, we can see that we are using MediaType#TEXT_PLAIN as the Content-Type header. After all, the previous attempt to add the Content-Type header was not successful. Therefore, the final result is not surprising: “Content-Type: text/plain;charset=UTF-8”.
From this case analysis, we can conclude that although we set the Content-Type in the Controller, it is a special header and may not be successfully set when using Spring Boot with embedded Tomcat. Instead, the Content-Type returned is determined based on the actual return value, type, and other factors.
Issue Fix #
To fix this issue and set the Content-Type successfully, we need to ensure that the returned type is actually JSON so that it can take effect. From the above analysis, we can also make the following modifications:
1. Modify the Accept header in the request to constrain the return type
The modified request would look like this:
GET http://localhost:8080/hi3
Accept: application/json
By including the Accept header, the server will select the value in the Accept header when determining the MediaType. The execution details can be found in the method AbstractMessageConverterMethodProcessor#getAcceptableMediaTypes.
2. Specify the return type explicitly
Explicitly specify the return type in the method like this:
@RequestMapping(path = "/hi3", method = RequestMethod.GET, produces = {"application/json"})
By using the produces attribute, we can specify the MediaType to be returned. Once set, only the specified type will be returned. Refer to the method AbstractMessageConverterMethodProcessor#getProducibleMediaTypes:
protected List<MediaType> getProducibleMediaTypes(HttpServletRequest request, Class<?> valueClass, @Nullable Type targetType) {
Set<MediaType> mediaTypes = (Set<MediaType>) request.getAttribute(HandlerMapping.PRODUCIBLE_MEDIA_TYPES_ATTRIBUTE);
if (!CollectionUtils.isEmpty(mediaTypes)) {
return new ArrayList<>(mediaTypes);
}
// Omit other non-key code
}
These two approaches modify the return value of getAcceptableMediaTypes and getProducibleMediaTypes respectively, which control the final negotiated result to be JSON. This affects the subsequent execution.
However, it is important to note that although we have successfully set the Content-Type header to JSON, the content is still processed using StringHttpMessageConverter. If you are interested, you can further investigate the reason why.
## Key Recap
Through this lesson, we have learned about some common errors when parsing headers in Spring and the underlying reasons behind them. Let's review the key points:
1. To receive all headers completely, we should not directly use a `Map` but rather use a `MultiValueMap`. The two common ways to do this are:
```java
// Way 1
@RequestHeader MultiValueMap<String, String> map
// Way 2: A specialized subtype of MultiValueMap for headers
@RequestHeader HttpHeaders map
Upon closer examination, when Spring parses headers at the underlying level and the received parameter is a Map, only one value is stored when the request header has multiple values.
-
In the HTTP protocol specification, the case of header names is not significant. However, this does not mean that all the ways to obtain headers will consistently return header names in the same case.
-
Not all headers can be freely specified in the response. Even though it may appear to be effective, the value returned to the client is still not the one you specified. For example, in Tomcat, the
CONTENT_TYPEheader is such a case.
The above points are the key knowledge points of this lesson. We hope that you will have more confidence when parsing headers in the future.
Thought-provoking Questions #
In Case 3, we mentioned the example of using the Content-Type in the Controller layer and how sometimes customizing common headers may not work. So is this conclusion universal? That is, does the same problem exist when using other built-in containers or in other development frameworks?
Looking forward to your thoughts in the comments section!