21 Spring Rest Template Common Errors #
Hello, I’m Fu Jian.
In the previous lessons, we discussed common errors that can occur when using a database in a Spring microservice. However, in addition to directly using a database, it is also a common use case to use other microservices to accomplish functionality.
Generally, communication between microservices is mostly done using the HTTP protocol, which naturally involves the use of HttpClient. Before using Spring, we usually use libraries like Apache HttpClient and Ok HttpClient directly. However, once you introduce Spring, you have a better choice, which is the main focus of this lesson - RestTemplate. So, what errors can occur when using RestTemplate? Let’s summarize them below.
Case 1: Parameter type is MultiValueMap #
First, let’s complete an API interface. The code is as follows:
@RestController
public class HelloWorldController {
@RequestMapping(path = "hi", method = RequestMethod.POST)
public String hi(@RequestParam("para1") String para1, @RequestParam("para2") String para2){
return "helloworld:" + para1 + "," + para2;
};
}
Here, we want to create an API that accepts a form request. It reads two parameters, para1 and para2, from the form and returns them as a response to the client.
After defining this interface, we use RestTemplate to send a form request. The code is as follows:
RestTemplate template = new RestTemplate();
Map<String, Object> paramMap = new HashMap<String, Object>();
paramMap.put("para1", "001");
paramMap.put("para2", "002");
String url = "http://localhost:8080/hi";
String result = template.postForObject(url, paramMap, String.class);
System.out.println(result);
The above code defines a Map that contains two form parameters. It then uses RestTemplate’s postForObject method to submit this form.
After testing, you will find that it does not work as expected and returns a 400 error, indicating a request failure:
Specifically, the form parameter para1 is missing. Why does this error occur? What happened to our submitted form?
Analysis #
Before analyzing this problem, let’s take a look at what the submitted request looks like when we use the above RestTemplate to submit the form. I captured the request using the Wireshark network capture tool:
From the above screenshot, we can see that we actually submitted the form data as a JSON request body. Therefore, our API cannot obtain any form parameters.
So why are we submitting the data as a JSON request body? Let’s take a look at the key parts of the code when executing the above code with RestTemplate.
First, let’s look at the call stack of the above code:
Indeed, we can verify that we ultimately use the Jackson library to serialize the form. The key use of JSON lies in the critical part of the code, RestTemplate.HttpEntityRequestCallback#doWithRequest:
public void doWithRequest(ClientHttpRequest httpRequest) throws IOException {
super.doWithRequest(httpRequest);
Object requestBody = this.requestEntity.getBody();
if (requestBody == null) {
// omit unrelated code
}
else {
Class<?> requestBodyClass = requestBody.getClass();
Type requestBodyType = (this.requestEntity instanceof RequestEntity ?
((RequestEntity<?>)this.requestEntity).getType() : requestBodyClass);
HttpHeaders httpHeaders = httpRequest.getHeaders();
HttpHeaders requestHeaders = this.requestEntity.getHeaders();
MediaType requestContentType = requestHeaders.getContentType();
for (HttpMessageConverter<?> messageConverter : getMessageConverters()) {
if (messageConverter instanceof GenericHttpMessageConverter) {
GenericHttpMessageConverter<Object> genericConverter =
(GenericHttpMessageConverter<Object>) messageConverter;
if (genericConverter.canWrite(requestBodyType, requestBodyClass, requestContentType)) {
if (!requestHeaders.isEmpty()) {
requestHeaders.forEach((key, values) -> httpHeaders.put(key, new LinkedList<>(values)));
}
logBody(requestBody, requestContentType, genericConverter);
genericConverter.write(requestBody, requestBodyType, requestContentType, httpRequest);
return;
}
}
else if (messageConverter.canWrite(requestBodyClass, requestContentType)) {
if (!requestHeaders.isEmpty()) {
requestHeaders.forEach((key, values) -> httpHeaders.put(key, new LinkedList<>(values)));
}
The above code may seem complex, but it actually has a simple function: it traverses all the supported encoders and decoders based on the current Body content to find a suitable one, and uses it to convert the Body. Let’s take a look at the judgment of the JSON encoder and decoder to see if it is suitable. Refer to AbstractJackson2HttpMessageConverter#canWrite:
[Code screenshot]
As we can see, when we use a HashMap as the Body, it can be serialized as JSON. So it’s not surprising that this form is serialized as the request Body.
But you may have a question: why can’t the form handling encoder and decoder be used? Let’s continue to look at the implementation of the corresponding encoder and decoder to see if it supports it, that is, FormHttpMessageConverter#canWrite:
[Code screenshot]
From the above code, we can see that in fact, only when the Body we send is a MultiValueMap can we use a form to submit it. By now, you probably understand. It turns out that when using RestTemplate to submit a form, it must be a MultiValueMap, but what we defined in the example is a normal HashMap, which is finally sent as the request Body.
Problem Fix #
In fact, after explaining so much, I believe you already know how to solve this problem. It’s actually very simple, just change the HashMap in the example to a MultiValueMap to store the form data. The corrected code is as follows:
//Error: //Map<String, Object> paramMap = new HashMap<String, Object>(); //paramMap.put(“para1”, “001”); //paramMap.put(“para2”, “002”);
//Corrected code: MultiValueMap<String, Object> paramMap = new LinkedMultiValueMap<String, Object>(); paramMap.add(“para1”, “001”); paramMap.add(“para2”, “002”);
Finally, you will find that after making the above modifications, the form data is finally encoded using the following code, referring to FormHttpMessageConverter#write:
[Code screenshot]
The sent data is shown in the screenshot below:
[Image]
This satisfies our requirements.
In fact, if you carefully read the documentation, you might also be able to avoid this problem. The key lines in the documentation are as follows:
The body of the entity, or request itself, can be a MultiValueMap to create a multipart request. The values in the MultiValueMap can be any Object representing the body of the part, or an HttpEntity.
I believe you can understand what it says without me explaining. Many people make mistakes because they don’t have the patience to read or are too lazy to read, and prefer to “think for themselves.” In the use of Spring, this is a big taboo.
Case 2: When there are special characters in the URL #
Next, let’s take a look at another issue related to using RestTemplate. We will use the same interface definition as before, but simplify it a bit. The code example is as follows:
@RestController
public class HelloWorldController {
@RequestMapping(path = "hi", method = RequestMethod.GET)
public String hi(@RequestParam("para1") String para1){
return "helloworld:" + para1;
};
}
I believe you already have a general idea of what we want to achieve - simply provide an HTTP interface with “parameters”.
Next, let’s test it using the following RestTemplate code:
String url = "http://localhost:8080/hi?para1=1#2";
HttpEntity<?> entity = new HttpEntity<>(null);
RestTemplate restTemplate = new RestTemplate();
HttpEntity<String> response = restTemplate.exchange(url, HttpMethod.GET, entity, String.class);
System.out.println(response.getBody());
When you see this testing code, what do you expect the output to be? You might think it would be:
helloworld:1#2
However, in reality, the actual result is:
helloworld:1
In other words, the server doesn’t consider #2 as part of the content of para1. How do we understand this phenomenon? Let’s analyze it in detail.
Analysis of the Case #
Similar to the approach used in Case 1, before diving into the specific analysis, let’s first get a sense of where the problem lies. We can use the debugging method to examine the parsed URL. The screenshot is as follows:
As you can see, the missing #2 of para1 is actually recorded as part of the Fragment. While we’re at it, let’s take a brief look at what a Fragment is. To understand this, we need to go back to the definition of the URL format:
protocol://hostname[:port]/path/[?query]#fragment
The two key elements involved in this case are explained as follows:
- Query
Parameters needed when requesting data for page loading. Each parameter’s name and value are separated by an = sign and multiple parameters are separated by an & sign.
- Fragment
A string starting with #, used to specify a fragment within a network resource. For example, if a web page contains multiple term explanations, Fragments can be used to directly locate a specific term’s explanation. For example, to locate the scrolling position of a webpage, you can refer to the following examples:
http://example.com/data.csv#row=4 - Selects the 4th row. - http://example.com/data.csv#col=2 - Selects the 2nd column.
After understanding this additional knowledge, we actually know where the issue lies. However, let’s be rigorous and take a look at the source code. First, let’s examine the call stack of URL parsing. The example is as follows:
Referring to the above call stack, the key point of URL parsing lies in the implementation of UriComponentsBuilder#fromUriString:
private static final Pattern URI_PATTERN = Pattern.compile(
"^(" + SCHEME_PATTERN + ")?" + "(//(" + USERINFO_PATTERN + "@)?" + HOST_PATTERN + "(:" + PORT_PATTERN +
")?" + ")?" + PATH_PATTERN + "(\\?" + QUERY_PATTERN + ")?" + "(#" + LAST_PATTERN + ")?");
public static UriComponentsBuilder fromUriString(String uri) {
Matcher matcher = URI_PATTERN.matcher(uri);
if (matcher.matches()) {
UriComponentsBuilder builder = new UriComponentsBuilder();
String scheme = matcher.group(2);
String userInfo = matcher.group(5);
String host = matcher.group(6);
String port = matcher.group(8);
String path = matcher.group(9);
String query = matcher.group(11);
String fragment = matcher.group(13);
// Omitted non-keycode
else {
builder.userInfo(userInfo);
builder.host(host);
if (StringUtils.hasLength(port)) {
builder.port(port);
}
builder.path(path);
builder.query(query);
}
if (StringUtils.hasText(fragment)) {
builder.fragment(fragment);
}
return builder;
}
else {
throw new IllegalArgumentException("[" + uri + "] is not a valid URI");
}
}
From the above code implementation, we can see a few key lines that I extracted:
String query = matcher.group(11);
String fragment = matcher.group(13);
It’s clear that both the Query and Fragment are being processed. In the end, they each find their respective values (1 and 2) based on URI_PATTERN, although this is not in line with our original expectation.
Problem Resolution #
So how do we solve this problem? If you are not familiar with the various URL assembly methods provided by RestTemplate, you might feel a bit desperate. Here is the code correction method I propose, take a look at it first:
String url = "http://localhost:8080/hi?para1=1#2";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
URI uri = builder.build().encode().toUri();
HttpEntity<?> entity = new HttpEntity<>(null);
RestTemplate restTemplate = new RestTemplate();
HttpEntity<String> response = restTemplate.exchange(uri, HttpMethod.GET, entity, String.class);
System.out.println(response.getBody());
The final test result meets our expectations:
helloworld:1#2
If you compare it with the previous example code, you will find that the way the URL is assembled has changed. But in the end, we can achieve the expected result. Refer to the debugging view below:
As you can see, the value of the para1 parameter has become “1#2” as expected.
If you want to learn more, you can also refer to UriComponentsBuilder#fromHttpUrl and compare it with the previously used UriComponentsBuilder#fromUriString:
private static final Pattern HTTP_URL_PATTERN = Pattern.compile(
"^" + HTTP_PATTERN + "(//(" + USERINFO_PATTERN + "@)?" + HOST_PATTERN + "(:" + PORT_PATTERN + ")?" + ")?" +
PATH_PATTERN + "(\\?" + LAST_PATTERN + ")?");
public static UriComponentsBuilder fromHttpUrl(String httpUrl) {
Assert.notNull(httpUrl, "HTTP URL must not be null");
Matcher matcher = HTTP_URL_PATTERN.matcher(httpUrl);
if (matcher.matches()) {
UriComponentsBuilder builder = new UriComponentsBuilder();
String scheme = matcher.group(1);
builder.scheme(scheme != null ? scheme.toLowerCase() : null);
builder.userInfo(matcher.group(4));
String host = matcher.group(5);
if (StringUtils.hasLength(scheme) && !StringUtils.hasLength(host)) {
throw new IllegalArgumentException("[" + httpUrl + "] is not a valid HTTP URL");
}
builder.host(host);
String port = matcher.group(7);
if (StringUtils.hasLength(port)) {
builder.port(port);
}
builder.path(matcher.group(8));
builder.query(matcher.group(10));
return builder;
}
else {
throw new IllegalArgumentException("[" + httpUrl + "] is not a valid HTTP URL");
}
}
As you can see, this method only parses the Query and does not attempt to parse the Fragment. Therefore, the final result matches our expectations.
Through this example, we learned that when there are special characters in the URL, we must be careful with the way the URL is assembled, especially distinguishing between the following two methods:
UriComponentsBuilder#fromHttpUrl - UriComponentsBuilder#fromUriString
Case 3: Be Careful with Multiple URL Encoders #
Next, let’s continue with another case that uses the same interface as before:
@RestController
public class HelloWorldController {
@RequestMapping(path = "hi", method = RequestMethod.GET)
public String hi(@RequestParam("para1") String para1){
return "helloworld:" + para1;
};
}
We can access this interface in a different way, as shown below:
RestTemplate restTemplate = new RestTemplate();
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl("http://localhost:8080/hi");
builder.queryParam("para1", "开发测试 001");
String url = builder.toUriString();
ResponseEntity<String> forEntity = restTemplate.getForEntity(url, String.class);
System.out.println(forEntity.getBody());
We expect the result to be “helloworld:开发测试 001”, but when we run the above code, we find that the result is actually:
helloworld:%E5%BC%80%E5%8F%91%E6%B5%8B%E8%AF%95001
How do we understand this issue?
Case Analysis #
To understand this case, we need a basic understanding of URL handling in the code above. First, let’s look at the code invocation in the case:
String url = builder.toUriString();
This is done by executing UriComponentsBuilder#toUriString:
public String toUriString() {
return this.uriVariables.isEmpty() ?
build().encode().toUriString() :
buildInternal(EncodingHint.ENCODE_TEMPLATE).toUriString();
}
It can be seen that it eventually performs URL encoding:
public final UriComponents encode() {
return encode(StandardCharsets.UTF_8);
}
By examining the call stack, the result is as follows:
And when we convert the URL to a string and send the request using the following statement:
// url is a string- restTemplate.getForEntity(url, String.class);
We find that it will perform encoding again:
By now, you may already understand the problem. That is, according to the code in the case, we perform encoding (Encode) twice, so we end up not getting the desired result.
In addition, we can also check the results after each encoding, as shown below:
After the first encoding:
After the second encoding:
Issue Fix #
How to fix it? Here is the code:
RestTemplate restTemplate = new RestTemplate();
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl("http://localhost:8080/hi");
builder.queryParam("para1", "开发测试 001");
URI url = builder.encode().build().toUri();
ResponseEntity<String> forEntity = restTemplate.getForEntity(url, String.class);
System.out.println(forEntity.getBody());
In fact, this fix is to avoid multiple conversions and multiple encodings. We won’t go into the internal implementation of this correct method, as the correct method is not the focus of this analysis. As long as you realize where the problem lies, we have achieved our goal.
After running the test again, the result matches the expectation:
helloworld:开发测试 001
Key Takeaways #
In this lesson, we learned about 3 typical issues commonly encountered when using RestTemplate. Let’s review the key points again:
- When assembling form data using RestTemplate, we should use MultiValueMap instead of a regular HashMap. Otherwise, the request will be sent as JSON instead of form data. Here’s an example of the correct usage:
MultiValueMap<String, Object> paramMap = new LinkedMultiValueMap<String, Object>();
paramMap.add("para1", "001");
paramMap.add("para2", "002");
String url = "http://localhost:8080/hi";
String result = template.postForObject(url, paramMap, String.class);
System.out.println(result)
- When sending requests with RestTemplate and including query parameters, we need to be cautious of special characters like “#”. To avoid any issues, we can use the following way to assemble the URL:
String url = "http://localhost:8080/hi?para1=1#2";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
URI uri = builder.build().encode().toUri();
- When using URLs in RestTemplate, we should avoid multiple conversions that can result in encoding issues.
These are the key takeaways from this lesson. They are not difficult to grasp, and once you have a good understanding of them, you can be flexible in your usage.
Thought question #
When we compare Case 1 and Case 2, we can see that regardless of whether we are using query parameters or form parameters, our interface definition remains the same. The style is as follows:
@RestController
public class HelloWorldController {
@RequestMapping(path = "hi", method = RequestMethod.GET)
public String hi(@RequestParam("para1") String para1){
return "helloworld:" + para1;
};
}
So, can @RequestParam handle both types of data?
We look forward to your thoughts and comments in the message area!