47 Network Services Based on the HTTP Protocol #
In the previous article, we briefly discussed network programming and sockets, and mentioned the syscall
and net
packages in the Go standard library.
I also emphasized the significance of understanding the parameters of the net.Dial
and syscall.Socket
functions. Since the former indirectly calls the latter, a proper understanding of the latter will greatly benefit the use of the former.
Afterwards, we shifted our focus to the net.DialTimeout
function and its handling of operation timeouts, which involves the net.Dialer
type. In fact, this type is the underlying implementation of these two “dial” functions in the net
package.
Using either the net.Dial
or net.DialTimeout
function to access network services based on the HTTP protocol is perfectly fine, as the HTTP protocol is built upon the TCP/IP protocol stack and is a text-oriented protocol.
In principle, we can easily write a complete HTTP request message using any text editor. As long as you understand what the header and body of the request message should contain, it will be straightforward. So, in this case, even if you directly use the net.Dial
function, you shouldn’t encounter difficulties.
However, convenience is not equal to ease. If we only want to access network services based on the HTTP protocol, it would be much easier to use the entities provided by the net/http
package.
The most convenient option is to use the http.Get
function. When calling this function, we only need to pass it a URL, like this:
url1 := "http://google.cn"
fmt.Printf("Send request to %q with method GET ...\n", url1)
resp1, err := http.Get(url1)
if err != nil {
fmt.Printf("request sending error: %v\n", err)
}
defer resp1.Body.Close()
line1 := resp1.Proto + " " + resp1.Status
fmt.Printf("The first line of response:\n%s\n", line1)
The http.Get
function returns two values. The first value is of type *http.Response
, which represents the structured representation of the response content returned by the network service.
The second value is of type error
, which represents any errors that may occur during the creation and sending of the HTTP request, as well as the receiving and parsing of the HTTP response.
The http.Get
function internally uses the default HTTP client and calls its Get
method to accomplish the task. This default HTTP client is represented by the public variable DefaultClient
in the net/http
package, which is of type *http.Client
. Its underlying type can also be used directly and is ready to use out of the box. The following two lines of code:
var httpClient1 http.Client
resp2, err := httpClient1.Get(url1)
are equivalent to the previous line of code:
resp1, err := http.Get(url1)
http.Client
is a struct type, and its fields are all public. The zero value of this type can still be used because either the fields have default values or their zero values represent specific meanings.
Now, I have a question for you regarding the most important field in this type.
Today’s question is: What does the Transport
field in the http.Client
type represent?
The typical answer to this question is as follows.
The Transport
field in the http.Client
type represents the process of sending an HTTP request to a network service and receiving the HTTP response from it. In other words, the RoundTrip
method of this field should implement all the steps required for a single HTTP transaction (or interaction based on the HTTP protocol).
This field is of type http.RoundTripper
, which has a default value represented by the DefaultTransport
variable (referred to as DefaultTransport
for short). When initializing a value of type http.Client
(referred to as Client
for short), if this field is not explicitly assigned a value, the Client
value will directly use the DefaultTransport
.
By the way, the Timeout
field of the http.Client
type represents the timeout duration for a single HTTP transaction mentioned earlier, and it is of type time.Duration
. Its zero value is valid and indicates that no timeout is set.
Problem Analysis #
Now, let’s further understand the Transport
field by examining its default value, DefaultTransport
.
The actual type of DefaultTransport
is *http.Transport
, which is the default implementation of the http.RoundTripper
interface. This type can be reused and is recommended to be reused. Additionally, it is concurrency-safe. Because of these traits, the http.Client
type also possesses the same characteristics.
The http.Transport
type internally uses a value of type net.Dialer
(referred to as the Dialer
value) and sets the Timeout
field of this value to 30
seconds.
In other words, if the Dialer
value fails to establish a network connection within 30 seconds, it will be considered a timeout. When the value of DefaultTransport
is initialized, the DialContext
method of this Dialer
value is assigned to the DialContext
field of the former.
The http.Transport
type also includes many other fields, some of which are related to timeouts.
IdleConnTimeout
: Indicates how long an idle connection should remain open before being closed.DefaultTransport
sets the value of this field to90
seconds. If this value is0
, it means that idle connections will not be closed. Note that this can potentially cause resource leaks.ResponseHeaderTimeout
: Indicates the maximum time from when the client completely submits the request to the operating system to when the client receives the response header from the operating system.DefaultTransport
does not set a value for this field.ExpectContinueTimeout
: Indicates the maximum time to wait after the client submits the request header before receiving the first response header. When the client wants to use the HTTP “POST” method to send a large message body to the server, it can first send a request header that includes “Expect: 100-continue” to ask the server if it is willing to receive the large message body. This field is used to set the timeout in this situation. Note that if the value of this field is less than or equal to0
, any size of request message body will be immediately sent. This can potentially waste network resources.DefaultTransport
sets the value of this field to1
second.TLSHandshakeTimeout
: TLS stands for Transport Layer Security. This field represents the timeout during the handshake phase of a TLS-based connection. A value of0
means there is no time limit for this.DefaultTransport
sets the value of this field to10
seconds.
In addition, there are some fields related to IdleConnTimeout
that are worth our attention, namely: MaxIdleConns
, MaxIdleConnsPerHost
, and MaxConnsPerHost
.
The MaxIdleConns
field only limits the total number of idle connections, regardless of how many network services the current Transport
value has accessed. On the other hand, the MaxIdleConnsPerHost
field limits the maximum number of idle connections for each network service accessed by the Transport
value.
Each network service has its own network address, may use different network protocols, and may use proxies for some HTTP requests. The Transport
value distinguishes different network services based on these three aspects.
The default value of the MaxIdleConnsPerHost
field is represented by the http.DefaultMaxIdleConnsPerHost
variable, which has a value of 2
. In other words, by default, for each network service accessed by a particular Transport
value, it can have a maximum of two idle connections.
The MaxConnsPerHost
field is similar to the MaxIdleConnsPerHost
field. However, the former limits the maximum number of connections made to a specific network service for a given Transport
value, regardless of whether these connections are idle or not. Additionally, this field does not have a default value, and its zero value indicates no limit.
The DefaultTransport
does not explicitly assign values to the MaxIdleConnsPerHost
and MaxConnsPerHost
fields, but it sets the value of the MaxIdleConns
field to 100
.
In other words, by default, the maximum number of idle connections is 100, and the maximum number of idle connections per network service is 2. Note that the values of these two fields related to idle connections should be coordinated, so sometimes you need to customize them based on the actual situation.
Of course, this first requires us to customize the value of the Transport
field when initializing the Client
value. The way to customize this value can be found in the declaration of the DefaultTransport
variable.
Finally, let me briefly explain why idle connections occur. As we all know, the HTTP protocol has a request header called “Connection.” In version 1.1 of the HTTP protocol, the default value of this header is “keep-alive.”
In this case, the network connections are persistent connections, and they remain connected even after the current HTTP transaction is completed, making them reusable.
Since connections can be reused, there are two possibilities. One possibility is that for the same network service, a new HTTP request is submitted and the connection is reused. The other possibility is that there are no more HTTP requests for that network service, and the connection remains idle.
Obviously, the latter possibility results in idle connections. Additionally, if too many connections are assigned to a network service, it may also result in idle connections, because each new HTTP request will only use one idle connection. Therefore, it is necessary and essential to limit idle connections and should be carefully considered.
If we want to completely eliminate idle connections, we can set the value of the DisableKeepAlives
field to true
when initializing the Transport
value. This will set the value of the “Connection” header in the HTTP request to “close”. This tells the network service that the network connection does not need to be kept and can be disconnected after the current HTTP transaction is completed.
In this case, a new network connection will be created every time an HTTP request is submitted. Doing so will significantly increase the load on the network service and the client, and each HTTP transaction will take more time. Therefore, in general, we should not set the DisableKeepAlives
field.
By the way, in the net.Dialer
type, there is also a field called KeepAlive
that looks similar. However, it is not the same concept as HTTP persistent connections mentioned earlier. KeepAlive
directly affects the underlying socket.
It represents a mechanism for detecting the liveliness of a network connection (more precisely, a TCP connection). Its value indicates the interval at which probe packets are sent. When the value is not greater than 0
, it means that this mechanism is not enabled. DefaultTransport
sets the value of this field to 30
seconds.
Alright, the above discussed the meaning of the Transport
field in the http.Client
type and how to customize its values. This involves the http.RoundTripper
interface, the http.DefaultTransport
variable, the http.Transport
type, and the net.Dialer
type.
Knowledge Expansion #
Question: What does the ListenAndServe
method of the http.Server
type do?
#
The http.Server
type is the counterpart of http.Client
. It represents a server based on the HTTP protocol, or in other words, a network service.
The ListenAndServe
method of the http.Server
type is used to listen on a TCP network address and handle incoming HTTP requests. This method automatically enables keep-alive probes for network connections to ensure persistent connections. The method continuously executes until a serious error occurs or it is shut down from outside. When shut down from outside, it returns an error value represented by the http.ErrServerClosed
variable.
For this question, a typical answer may be as follows.
The ListenAndServe
method primarily does the following:
- It checks the
Addr
field of the current value of thehttp.Server
type. This field represents the network address the server will use, i.e., the IP address and port number. If the value of this field is an empty string, it is replaced with":http"
. In other words, it uses any domain name or IP address that represents the local machine and port number80
. - It starts listening on the determined network address using the TCP protocol by calling the
net.Listen
function. - It checks the error value returned by the
net.Listen
function. If the error value is notnil
, it is directly returned. Otherwise, it prepares to accept and handle incoming HTTP requests by calling theServe
method of the current value.
This can lead to two common follow-up questions, one being “What does the net.Listen
function do?” and the other being “How does the Serve
method of the http.Server
type accept and handle HTTP requests?”
For the first direct follow-up question, if we summarize it, the answer can be:
- The
net.Listen
function parses the implicitly defined IP address and port number contained in the parameter value. - It determines the listening method based on the given network protocol and starts listening.
Starting from the second step, we can further ask some indirect follow-up questions. This often involves the net.socket
function and related socket knowledge.
For the second direct follow-up question, the answer can be:
In a for
loop, the Accept
method of the network listener is continuously called, which returns two values. The first value is of type net.Conn
, representing the network connection that contains the incoming HTTP request. The second value is of type error
and represents a possible error.
If this error value is not nil
, unless it represents a temporary error, the loop will be terminated. If it is a temporary error, the next iteration of the loop will start after a certain period of time.
If the Accept
method does not return a non-nil
error value, the program will wrap its first return value into a *http.conn
type value, called the conn
value. Then, it calls the serve
method of this conn
value in a new goroutine to handle the current HTTP request.
There are many details in this handling process, so we can still find many indirect follow-up questions. For example, how many states does the conn
value have, and what do they represent in terms of the handling stages? What readers and writers are used in the process, and what are their respective roles? How does the program call our custom handler functions, and so on.
There are many questions like these, and I won’t list and explain them all here. Just remember the phrase, “There are no secrets before the source code.” The answers to these questions can all be found in the source code of the Go standard library. If you want to explore this question in depth, be sure to take a look at the net/http
package’s source code.
Summary #
Today, we mainly talked about web services based on the HTTP protocol, with a focus on the client side.
After discussing the simple usage of the http.Get
function and the http.Client
type, we turned our attention to the Transport
field of the latter.
This field represents the process of a single HTTP transaction. It is of type http.RoundTripper
interface. Its default value is represented by the http.DefaultTransport
variable, which is actually of type *http.Transport
.
http.Transport
contains many fields. We first discussed what value the DialContext
field in DefaultTransport
receives, and then we explained in detail some fields related to operation timeout.
For example, IdleConnTimeout
and ExpectContinueTimeout
, as well as related fields like MaxIdleConns
and MaxIdleConnsPerHost
. Afterwards, I briefly explained the reasons for idle connections and how to customize them.
Finally, as an extension, I briefly outlined the main process of the ListenAndServe
method of the http.Server
type. However, due to space limitations, I did not go into depth. But this doesn’t mean it’s not worth exploring further. On the contrary, this method is important and worth our careful exploration.
When you need it or are interested, I hope you can take a good look at the related source code in the net/http
package. All the secrets are there.
Thought-provoking question #
The thought-provoking question I leave you today is relatively simple: How can we gracefully stop a network service program based on the HTTP protocol?
Click here to view the detailed code accompanying the article on the Go language column.