04 How to Use JWT Structured Tokens in OAuth 2 #

Hello, I am Wang Xindong.

In the previous lecture, we talked about how the core of the authorization service is the issuance of access tokens. The OAuth 2.0 specification does not constrain the generation rules for the contents of access tokens, as long as they are unique, non-sequential, and not guessable. This means that we can flexibly choose the format of the tokens, whether they are random strings without internal structure and meaning, or strings with internal structure and meaning.

I won’t introduce the method of using random strings, as we have already used it in previous courses to generate tokens. In terms of structured tokens, the most commonly used one is the JWT token.

Next, I will explain in detail what JWT is, how it works, its advantages, and how to use it. I will also discuss the issue of token lifecycle.

Structured JWT Tokens #

The official definition of what JWT is goes like this:

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way to securely transmit information between parties.

This definition might be confusing. Let’s understand it briefly. JWT is a technology that generates tokens using a structured packaging method. The structured token can have rich meanings, which is the biggest difference from the meaningless and random string-form tokens of the past.

After structuring, the token itself can contain useful information, such as the authorization information that Xiaoming has authorized for Xiaotu Software and the authorized scope information. Alternatively, you can think of it as a “self-encoding” ability, which is exactly what unstructured tokens lack.

This structured JWT can be divided into three components: HEADER, PAYLOAD, and SIGNATURE. The signed JWT structure is divided into three segments separated by periods. The structure is header.payload.signature. For example:

    eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.
    eyJzdWIiOiJVU0VSVEVTVCIsImV4cCI6MTU4NDEwNTc5MDcwMywiaWF0IjoxNTg0MTA1OTQ4MzcyfQ.
    1HbleXbvJ_2SW8ry30cXOBGR9FW4oSWBd3PWaWKsEXE

Note: There are no line breaks inside the JWT. It is just represented in three lines for readability.

You might say that this JWT token still looks like a meaningless and random string. Indeed, directly looking at this string doesn’t make much sense. However, if you copy it to the online verification tool at jwt.io, you can see the decoded data:

Figure 1. JWT Token decoded by the online verification tool

Looking at the decoded data, you can see that it is different from a random string. Clearly, structured content is presented now. Next, I will explain the three parts of JWT in detail.

HEADER represents the type of token and algorithm information, forming the header of JWT. The “typ” field indicates that the second part, PAYLOAD, is a JWT type, and the “alg” field indicates the symmetric signing algorithm of HS256.

PAYLOAD represents the data body of JWT and represents a set of data. The “sub” (token subject, generally set as the unique identifier of the resource owner), “exp” (expiration timestamp of the token), and “iat” (timestamp when the token is issued) are declarative parts of the JWT specification, representing routine operations. For more common declarations, you can refer to the open standard RFC 7519. In a JWT, any valid JSON format data can be included in the PAYLOAD, which means that the set of data represented by PAYLOAD allows for custom declarations.

SIGNATURE represents the signature of the JWT information. So, what is its role? We might think that with the HEADER and PAYLOAD contents, we can carry information in the token and transmit it in the network. But transmitting such information directly in the network is not secure because you are “naked”. Therefore, we need to encrypt and sign it, and the SIGNATURE is the result of signing the information. When the protected resource receives a signature from a third-party software, it needs to verify the signature of the token.

Now that we know the structure of JWT and the meaning of each part, how is the JWT token used in the OAuth 2.0 authorization flow? Before discussing how to use it, let me explain the concept of “token introspection”.

Token introspection #

What is token introspection? Authorization services issue tokens, and protected resource services need to validate these tokens. Moreover, authorization services and protected resource services work together, as I mentioned in Lesson 2 before. Protected resource services call the token introspection service provided by the authorization service. We call this way of validating tokens token introspection.

Sometimes, the authorization service relies on a database, and the protected resource service also relies on this database, which we call a “shared database”. However, in today’s mature distributed and microservices environments, different systems communicate through services instead of databases. For example, the authorization service provides an RPC service to the protected resource service, as shown in the following diagram.

Figure 2: The authorization service provides an interface service for the protected resource to validate tokens.

With JWT tokens, we have another option because the JWT token itself contains information that we previously had to rely on a database or RPC service to obtain, such as the user who authorized a certain software, etc.

Next, let’s take a look at how the overall introspection process changes with JWT tokens.

How is JWT used? #

With the use of JWT tokens, the communication flow is as shown in Figure 3 below. The “Authorization Service” issues a token, and the “Protected Resource Service” receives and parses the token itself to obtain the contained information, without the need to query a database or request an RPC service. This also achieves the internal token checks we mentioned earlier.

Figure 3: The Protected Resource Service directly parses the JWT token

In the above diagram, in order to highlight the position of the JWT token, I simplified the logical relationships. In reality, the Authorization Service issues the JWT token to Little Rabbit Software, and Little Rabbit Software uses the JWT token to request the Protected Resource Service, which is the orders in Jingdong Store for Xiao Ming. It is obvious that the JWT token needs to be transmitted over the public network. Therefore, during the transmission process, the JWT token needs to be Base64 encoded to prevent garbled characters, and it also needs to be signed and encrypted to prevent data information leakage.

If we were to handle these encoding, encryption, and other tasks ourselves, it would increase the additional coding burden. Fortunately, we can rely on some open-source tools to help us with these tasks. For example, in the following demo, I provide the usage of the JJWT (Java JWT) open-source tool.

JJWT is currently the most convenient JWT tool in Java open-source projects. It encapsulates a series of signature algorithms for Base64URL encoding, symmetric HMAC, and asymmetric RSA. With JJWT, we only need to focus on the implementation of the upper-level business logic, without having to worry about the specific implementation of the encoding, decoding, and signature algorithms. These types of open-source tools can be used “out of the box”.

The code for this demo is as follows. With JJWT, we can easily generate a signed JWT token and parse a JWT token.

String sharedTokenSecret = "hellooauthhellooauthhellooauthhellooauth";
Key key = new SecretKeySpec(sharedTokenSecret.getBytes(), SignatureAlgorithm.HS256.getJcaName());
String jwt = Jwts.builder().setHeaderParams(headerMap).setClaims(payloadMap).signWith(key, SignatureAlgorithm.HS256).compact();

Jws<Claims> claimsJws = Jwts.parserBuilder().setSigningKey(key).build().parseClaimsJws(jwt);
JwsHeader header = claimsJws.getHeader();
Claims body = claimsJws.getBody();

When using JJWT to parse the JWT token, it includes the action of verifying the signature. If the signature is incorrect, an exception will be thrown. We can use this to verify the signature and determine whether it is a legitimate JWT token that has not been forged.

The exception message usually looks like this:

JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.

The above is the method of applying the JWT token to the authorization service process using open-source tools. Up to this point, you may have had a question: why go through this extra step and use JWT instead of using a random string with no internal structure or any information? What are the advantages of JWT?

Why use JWT tokens? #

Just wait, I’m summarizing the three main benefits of using JWT tokens for you.

First, the core idea of JWT is to use computation instead of storage, which has a taste of “time for space”. Of course, this method, which involves computation and structured packaging, also reduces the network transmission cost caused by remote calls in the “shared database”, which may save time.

Second, encryption is also an important feature. Because JWT tokens already contain important information, they must be transmitted in ciphertext throughout the entire transmission process. This requirement for encryption guarantees the security of the transmission. The encryption algorithm used here can be symmetric encryption or asymmetric encryption.

Third, using JWT tokens helps enhance the system’s availability and scalability. How should we understand this? As we mentioned earlier, this type of JWT token includes the authentication information needed through “self-encoding” and no longer requires additional storage on the server. Therefore, each request is a stateless session. This adheres to the principle of striving to follow the stateless architecture design, which enhances the system’s availability and scalability.

However, everything has two sides, and JWT tokens also have disadvantages.

The biggest problem with JWT tokens is that once something is done, it cannot be undone. Let’s take the example of Xiao Ming using Xiao Tu software and pause to think about it.

When Xiao Ming is using Xiao Tu software, is it possible for him to change his password on JD.com for some reason, or is it possible for him to suddenly revoke authorization from Xiao Tu? In this case, the token’s state should be changed accordingly, and the original token should be invalidated.

However, when using JWT tokens, each issued token is not stored on the server, so we are powerless to change the token’s state when needed. Because the server does not store this JWT formatted token. This means that within the token’s expiration period, it can “reign indefinitely”.

To solve this problem, can we store the JWT tokens in a remote distributed in-memory database? Obviously not, because this would violate the original intention of JWT (to store information in a structured manner within the token itself). Therefore, we usually have two approaches:

First, we can narrow down the granularity of the key used to generate JWT tokens to the user level, meaning one user has one key. This way, when a user revokes authorization or changes their password, we can also change this key together. Usually, this solution requires a separate key management service.

Second, in an environment where user-initiated revocation of authorization is not provided, if we only consider the scenario of changing passwords, we can use the user’s password as the key for JWT. Of course, this is also at the user level. In this case, when a user changes their password, it is equivalent to changing the key.

Token Lifecycle #

I just discussed the issue of JWT token expiration and its handling of invalidation. In addition, in [Lesson 3], we mentioned that when an authorization service issues an access token, it will set an expiration time. These are all problems related to token lifecycle management. Now, let me talk to you about the lifecycle of tokens.

Everything in the world has a cycle, and tokens are no exception, whether they are structured tokens like JWT or regular tokens. They all have an expiration period. However, JWT tokens can store expiration information in their own data structure.

Specifically for the token lifecycle in OAuth 2.0, there are usually three situations.

The first situation is the natural expiration process of a token, which is also the most common case. This process starts from the authorization service creating a token, to the third-party software using the token, to the protected resource service validating the token, and finally to the token becoming invalid. At the same time, this process does not exclude the occurrence of actively destroying tokens, such as when a token is leaked, the authorization service can invalidate the token.

The second situation in the lifecycle is what I mentioned in the previous lesson, where after an access token becomes invalid, a refresh token can be used to request a new access token to replace the invalidated one, in order to improve the user’s experience with third-party software.

The third situation in the lifecycle is when a third-party software, such as “Little Rabbit,” actively initiates a request to invalidate a token, and the authorization service immediately invalidates the token upon receiving the request. Let’s think about when this mechanism is needed, that is, let’s think about the “motivation” for a third-party software to do this, because in general, “it is difficult for us to give up something we already have.”

For example, sometimes there may be a subscription relationship between the user and the third-party software. For instance, if Xiao Ming purchases “Little Rabbit” software, then there should be a token revocation agreement to support the active invalidation of the token by “Little Rabbit” software when the subscription expires or is canceled, but Xiao Ming’s authorized token has not yet expired. As a platform provider, such as the JD Merchant Open Platform, it is also recommended that responsible third-party software, such as “Little Rabbit” software, adhere to such a token revocation agreement.

I have organized the above three situations into a sequence diagram to help you understand. At the same time, in order to highlight the tokens, I have deliberately used a dark color to identify the access token and refresh token and put them separately as two roles in the entire sequence diagram.

Figure 4. Token Lifecycle

Summary #

The core of OAuth 2.0 is the authorization service, or more specifically, the token. Without a token, there is no OAuth. The token represents the result of the authorization process.

In most cases, the token is a random and opaque string for third-party software. In most situations, when we mention tokens, we are referring to meaningless strings.

However, people are not satisfied with this and have begun to explore alternative ways of generating tokens. This led to the development of JWT tokens, which eliminates the need for a shared database or an authorization service with an interface for token verification. With JWT, we have an additional option for token verification through this structured approach.

In this lesson, I hope you remember the following key points:

We now have a new option for generating tokens, which is the JWT token. This is a structured and informative token where structured means it can organize user authorization information and informative means the token itself contains the authorization information.

Although we focused on JWT tokens in this lesson, whether the token is structured or unstructured, it doesn’t matter to third-party software because tokens are opaque to third-party software in the OAuth 2.0 system. Authorization services and protected resource services are the ones that need to care about tokens.

We need to be aware of the issue of token invalidation with JWT tokens. After using a JWT token, it is no longer stored on the remote server because it is unnecessary. The JWT token itself contains the information. So, how can we control its validity? In this lesson, I suggested two approaches: one is to establish a key management system and reduce the granularity of key generation to the user level, and the other is to use the user’s password as the key.

Now, you have a deeper understanding of JWT and how to use it. When constructing and generating tokens, in addition to using random and arbitrary strings, you can also use structured tokens so that the content information of the token can be parsed directly for verification and processing during token verification.

I have placed the code used today on GitHub, and you can click this link to view it.

Thought question #

Do you know which scenarios are suitable for JWT tokens and which scenarios are not suitable for JWT tokens?

Feel free to share your thoughts in the comments section, and also feel free to share today’s content with other friends. Let’s exchange ideas together.