47 Network Services Based on the HTTP Protocol

47 Network Services Based on the HTTP Protocol #

In the previous article, we briefly discussed network programming and sockets, and mentioned the syscall and net packages in the Go standard library.

I also emphasized the significance of understanding the parameters of the net.Dial and syscall.Socket functions. Since the former indirectly calls the latter, a proper understanding of the latter will greatly benefit the use of the former.

Afterwards, we shifted our focus to the net.DialTimeout function and its handling of operation timeouts, which involves the net.Dialer type. In fact, this type is the underlying implementation of these two “dial” functions in the net package.

Using either the net.Dial or net.DialTimeout function to access network services based on the HTTP protocol is perfectly fine, as the HTTP protocol is built upon the TCP/IP protocol stack and is a text-oriented protocol.

In principle, we can easily write a complete HTTP request message using any text editor. As long as you understand what the header and body of the request message should contain, it will be straightforward. So, in this case, even if you directly use the net.Dial function, you shouldn’t encounter difficulties.

However, convenience is not equal to ease. If we only want to access network services based on the HTTP protocol, it would be much easier to use the entities provided by the net/http package.

The most convenient option is to use the http.Get function. When calling this function, we only need to pass it a URL, like this:

url1 := "http://google.cn"
fmt.Printf("Send request to %q with method GET ...\n", url1)
resp1, err := http.Get(url1)
if err != nil {
    fmt.Printf("request sending error: %v\n", err)
}
defer resp1.Body.Close()
line1 := resp1.Proto + " " + resp1.Status
fmt.Printf("The first line of response:\n%s\n", line1)

The http.Get function returns two values. The first value is of type *http.Response, which represents the structured representation of the response content returned by the network service.

The second value is of type error, which represents any errors that may occur during the creation and sending of the HTTP request, as well as the receiving and parsing of the HTTP response.

The http.Get function internally uses the default HTTP client and calls its Get method to accomplish the task. This default HTTP client is represented by the public variable DefaultClient in the net/http package, which is of type *http.Client. Its underlying type can also be used directly and is ready to use out of the box. The following two lines of code:

var httpClient1 http.Client
resp2, err := httpClient1.Get(url1)

are equivalent to the previous line of code:

resp1, err := http.Get(url1)

http.Client is a struct type, and its fields are all public. The zero value of this type can still be used because either the fields have default values or their zero values represent specific meanings.

Now, I have a question for you regarding the most important field in this type.

Today’s question is: What does the Transport field in the http.Client type represent?

The typical answer to this question is as follows.

The Transport field in the http.Client type represents the process of sending an HTTP request to a network service and receiving the HTTP response from it. In other words, the RoundTrip method of this field should implement all the steps required for a single HTTP transaction (or interaction based on the HTTP protocol).

This field is of type http.RoundTripper, which has a default value represented by the DefaultTransport variable (referred to as DefaultTransport for short). When initializing a value of type http.Client (referred to as Client for short), if this field is not explicitly assigned a value, the Client value will directly use the DefaultTransport.

By the way, the Timeout field of the http.Client type represents the timeout duration for a single HTTP transaction mentioned earlier, and it is of type time.Duration. Its zero value is valid and indicates that no timeout is set.

Problem Analysis #

Now, let’s further understand the Transport field by examining its default value, DefaultTransport.

The actual type of DefaultTransport is *http.Transport, which is the default implementation of the http.RoundTripper interface. This type can be reused and is recommended to be reused. Additionally, it is concurrency-safe. Because of these traits, the http.Client type also possesses the same characteristics.

The http.Transport type internally uses a value of type net.Dialer (referred to as the Dialer value) and sets the Timeout field of this value to 30 seconds.

In other words, if the Dialer value fails to establish a network connection within 30 seconds, it will be considered a timeout. When the value of DefaultTransport is initialized, the DialContext method of this Dialer value is assigned to the DialContext field of the former.

The http.Transport type also includes many other fields, some of which are related to timeouts.

  • IdleConnTimeout: Indicates how long an idle connection should remain open before being closed.
  • DefaultTransport sets the value of this field to 90 seconds. If this value is 0, it means that idle connections will not be closed. Note that this can potentially cause resource leaks.
  • ResponseHeaderTimeout: Indicates the maximum time from when the client completely submits the request to the operating system to when the client receives the response header from the operating system. DefaultTransport does not set a value for this field.
  • ExpectContinueTimeout: Indicates the maximum time to wait after the client submits the request header before receiving the first response header. When the client wants to use the HTTP “POST” method to send a large message body to the server, it can first send a request header that includes “Expect: 100-continue” to ask the server if it is willing to receive the large message body. This field is used to set the timeout in this situation. Note that if the value of this field is less than or equal to 0, any size of request message body will be immediately sent. This can potentially waste network resources. DefaultTransport sets the value of this field to 1 second.
  • TLSHandshakeTimeout: TLS stands for Transport Layer Security. This field represents the timeout during the handshake phase of a TLS-based connection. A value of 0 means there is no time limit for this. DefaultTransport sets the value of this field to 10 seconds.

In addition, there are some fields related to IdleConnTimeout that are worth our attention, namely: MaxIdleConns, MaxIdleConnsPerHost, and MaxConnsPerHost.

The MaxIdleConns field only limits the total number of idle connections, regardless of how many network services the current Transport value has accessed. On the other hand, the MaxIdleConnsPerHost field limits the maximum number of idle connections for each network service accessed by the Transport value.

Each network service has its own network address, may use different network protocols, and may use proxies for some HTTP requests. The Transport value distinguishes different network services based on these three aspects.

The default value of the MaxIdleConnsPerHost field is represented by the http.DefaultMaxIdleConnsPerHost variable, which has a value of 2. In other words, by default, for each network service accessed by a particular Transport value, it can have a maximum of two idle connections. The MaxConnsPerHost field is similar to the MaxIdleConnsPerHost field. However, the former limits the maximum number of connections made to a specific network service for a given Transport value, regardless of whether these connections are idle or not. Additionally, this field does not have a default value, and its zero value indicates no limit.

The DefaultTransport does not explicitly assign values to the MaxIdleConnsPerHost and MaxConnsPerHost fields, but it sets the value of the MaxIdleConns field to 100.

In other words, by default, the maximum number of idle connections is 100, and the maximum number of idle connections per network service is 2. Note that the values of these two fields related to idle connections should be coordinated, so sometimes you need to customize them based on the actual situation.

Of course, this first requires us to customize the value of the Transport field when initializing the Client value. The way to customize this value can be found in the declaration of the DefaultTransport variable.

Finally, let me briefly explain why idle connections occur. As we all know, the HTTP protocol has a request header called “Connection.” In version 1.1 of the HTTP protocol, the default value of this header is “keep-alive.”

In this case, the network connections are persistent connections, and they remain connected even after the current HTTP transaction is completed, making them reusable.

Since connections can be reused, there are two possibilities. One possibility is that for the same network service, a new HTTP request is submitted and the connection is reused. The other possibility is that there are no more HTTP requests for that network service, and the connection remains idle.

Obviously, the latter possibility results in idle connections. Additionally, if too many connections are assigned to a network service, it may also result in idle connections, because each new HTTP request will only use one idle connection. Therefore, it is necessary and essential to limit idle connections and should be carefully considered.

If we want to completely eliminate idle connections, we can set the value of the DisableKeepAlives field to true when initializing the Transport value. This will set the value of the “Connection” header in the HTTP request to “close”. This tells the network service that the network connection does not need to be kept and can be disconnected after the current HTTP transaction is completed.

In this case, a new network connection will be created every time an HTTP request is submitted. Doing so will significantly increase the load on the network service and the client, and each HTTP transaction will take more time. Therefore, in general, we should not set the DisableKeepAlives field.

By the way, in the net.Dialer type, there is also a field called KeepAlive that looks similar. However, it is not the same concept as HTTP persistent connections mentioned earlier. KeepAlive directly affects the underlying socket.

It represents a mechanism for detecting the liveliness of a network connection (more precisely, a TCP connection). Its value indicates the interval at which probe packets are sent. When the value is not greater than 0, it means that this mechanism is not enabled. DefaultTransport sets the value of this field to 30 seconds.

Alright, the above discussed the meaning of the Transport field in the http.Client type and how to customize its values. This involves the http.RoundTripper interface, the http.DefaultTransport variable, the http.Transport type, and the net.Dialer type.

Knowledge Expansion #

Question: What does the ListenAndServe method of the http.Server type do? #

The http.Server type is the counterpart of http.Client. It represents a server based on the HTTP protocol, or in other words, a network service.

The ListenAndServe method of the http.Server type is used to listen on a TCP network address and handle incoming HTTP requests. This method automatically enables keep-alive probes for network connections to ensure persistent connections. The method continuously executes until a serious error occurs or it is shut down from outside. When shut down from outside, it returns an error value represented by the http.ErrServerClosed variable.

For this question, a typical answer may be as follows.

The ListenAndServe method primarily does the following:

  1. It checks the Addr field of the current value of the http.Server type. This field represents the network address the server will use, i.e., the IP address and port number. If the value of this field is an empty string, it is replaced with ":http". In other words, it uses any domain name or IP address that represents the local machine and port number 80.
  2. It starts listening on the determined network address using the TCP protocol by calling the net.Listen function.
  3. It checks the error value returned by the net.Listen function. If the error value is not nil, it is directly returned. Otherwise, it prepares to accept and handle incoming HTTP requests by calling the Serve method of the current value.

This can lead to two common follow-up questions, one being “What does the net.Listen function do?” and the other being “How does the Serve method of the http.Server type accept and handle HTTP requests?”

For the first direct follow-up question, if we summarize it, the answer can be:

  1. The net.Listen function parses the implicitly defined IP address and port number contained in the parameter value.
  2. It determines the listening method based on the given network protocol and starts listening.

Starting from the second step, we can further ask some indirect follow-up questions. This often involves the net.socket function and related socket knowledge.

For the second direct follow-up question, the answer can be:

In a for loop, the Accept method of the network listener is continuously called, which returns two values. The first value is of type net.Conn, representing the network connection that contains the incoming HTTP request. The second value is of type error and represents a possible error.

If this error value is not nil, unless it represents a temporary error, the loop will be terminated. If it is a temporary error, the next iteration of the loop will start after a certain period of time.

If the Accept method does not return a non-nil error value, the program will wrap its first return value into a *http.conn type value, called the conn value. Then, it calls the serve method of this conn value in a new goroutine to handle the current HTTP request.

There are many details in this handling process, so we can still find many indirect follow-up questions. For example, how many states does the conn value have, and what do they represent in terms of the handling stages? What readers and writers are used in the process, and what are their respective roles? How does the program call our custom handler functions, and so on.

There are many questions like these, and I won’t list and explain them all here. Just remember the phrase, “There are no secrets before the source code.” The answers to these questions can all be found in the source code of the Go standard library. If you want to explore this question in depth, be sure to take a look at the net/http package’s source code.

Summary #

Today, we mainly talked about web services based on the HTTP protocol, with a focus on the client side.

After discussing the simple usage of the http.Get function and the http.Client type, we turned our attention to the Transport field of the latter.

This field represents the process of a single HTTP transaction. It is of type http.RoundTripper interface. Its default value is represented by the http.DefaultTransport variable, which is actually of type *http.Transport.

http.Transport contains many fields. We first discussed what value the DialContext field in DefaultTransport receives, and then we explained in detail some fields related to operation timeout.

For example, IdleConnTimeout and ExpectContinueTimeout, as well as related fields like MaxIdleConns and MaxIdleConnsPerHost. Afterwards, I briefly explained the reasons for idle connections and how to customize them.

Finally, as an extension, I briefly outlined the main process of the ListenAndServe method of the http.Server type. However, due to space limitations, I did not go into depth. But this doesn’t mean it’s not worth exploring further. On the contrary, this method is important and worth our careful exploration.

When you need it or are interested, I hope you can take a good look at the related source code in the net/http package. All the secrets are there.

Thought-provoking question #

The thought-provoking question I leave you today is relatively simple: How can we gracefully stop a network service program based on the HTTP protocol?

Click here to view the detailed code accompanying the article on the Go language column.