15 Runtime Reflection How to Convert Between Strings and Structs

15 Runtime Reflection - How to Convert Between Strings and Structs #

In development, we often encounter the need to convert between strings and structs, especially when making API calls. You may need to convert a JSON string returned by an API into a struct for easier manipulation. So how do you convert a JSON string into a struct? This requires knowledge of reflection. In this lesson, I will gradually uncover the mysteries of runtime reflection in Go based on the conversion between strings and structs.

What is Reflection? #

Just like in Java, Go also has runtime reflection, which provides us with the ability to manipulate objects of any type at runtime. For example, you can view the specific type of an interface variable, find out how many fields a struct has, and modify the value of a specific field.

Go is a statically compiled language. For example, when you define a variable, you already know its type. So why do we still need reflection? This is because there are things that can only be known at runtime. For example, if you define a function with an interface{} type parameter, it means that the caller can pass any type of argument to this function. In this case, if you want to know what type of argument the caller passed, you need to use reflection. If you want to know what fields and methods a struct has, you also need reflection.

Again, let’s take the Println function, which I often use, as an example:

src/fmt/print.go

func Println(a ...interface{}) (n int, err error) {

   return Fprintln(os.Stdout, a...)

}

In the example, the source code of fmt.Println has a variadic parameter of type interface{}. This means you can pass zero or more arguments of any type to it, and it will be printed correctly.

reflect.Value and reflect.Type #

In the reflection definition of Go, any interface consists of two parts: the concrete type of the interface and the value corresponding to the concrete type. For example, var i int = 3. Since interface{} can represent any type, the variable i can be converted to interface{}. You can treat the variable i as an interface, and in Go reflection, the representation of this variable is reflect.Value. The Value represents the value of the variable, which is 3, and the Type represents the type of the variable, which is int.

Pro Tip: interface{} is an empty interface that can represent any type. In other words, you can convert any type to an empty interface. It is often used for reflection and type assertion to reduce duplicate code and simplify programming.

In Go reflection, the standard library provides two types reflect.Value and reflect.Type to represent the value and type of any object, respectively. It also provides two functions reflect.ValueOf and reflect.TypeOf to get the reflect.Value and reflect.Type of any object.

Let’s demonstrate this with the following code:

ch15/main.go

func main() {

   i:=3

   iv:=reflect.ValueOf(i)

   it:=reflect.TypeOf(i)

   fmt.Println(iv,it)//3 int

}

This code defines a variable i of type int with a value of 3. Then, by using the reflect.ValueOf and reflect.TypeOf functions, we can obtain the reflect.Value and reflect.Type corresponding to the variable i. After printing with fmt.Println, we can see that the result is 3 int, which also proves that reflect.Value represents the value of the variable, while reflect.Type represents the type of the variable.

reflect.Value #

reflect.Value can be obtained using the function reflect.ValueOf. Let me introduce its structure and usage.

Struct Definition #

In Go, reflect.Value is defined as a struct with the following structure:

type Value struct {

   typ *rtype

   ptr unsafe.Pointer

   flag

}

We can see that the fields of reflect.Value struct are all private, which means we can only use the methods of reflect.Value. Now let’s see what commonly used methods it has:

// Methods for specific types

// The following methods are used to retrieve the corresponding value

Bool
    
Bytes
    
Complex
    
Float
    
Int

String
    
Uint

CanSet // Whether the corresponding value can be modified

// The following methods are used to modify the corresponding value

Set
    
SetBool
    
SetBytes
    
SetComplex
    
SetFloat
    
SetInt
    
SetString
    
Elem // Get the value pointed to by the pointer, usually used to modify the corresponding value

// Field-related methods for struct types

Field
    
FieldByIndex
    
FieldByName
    
FieldByNameFunc
    
Interface // Get the corresponding original type

IsNil // Whether the value is nil

IsZero // Whether the value is the zero value

Kind // Get the type category, such as Array, Slice, Map, etc.

// Methods for obtaining corresponding methods

Method
    
MethodByName
    
NumField // Get the number of fields in a struct type

NumMethod // Get the number of methods in the type's method set

Type // Get the corresponding reflect.Type

It may seem like a lot, but there are actually only three types of methods: methods for retrieving and modifying corresponding values, methods related to struct type fields for getting corresponding fields, and methods related to the type’s method set for obtaining corresponding methods.

Now let me explain how to use them through a few examples.

Get the Original Type #

In the previous example, I used the reflect.ValueOf function to convert any type of object into a reflect.Value. If you want to convert it back, reflect.Value provides the Interface method. Here is the code:

ch15/main.go

func main() {

   i:=3

   // int to reflect.Value

   iv:=reflect.ValueOf(i)

   // reflect.Value to int

   i1:=iv.Interface().(int)

   fmt.Println(i1)

}

This is the conversion between reflect.Value and int types, but it can be done with other types as well.

Modify the Corresponding Value #

Defined variables can be modified at runtime through reflection. For example, in the previous example, the variable i is initially set to 3 and then modified to 4 using reflection. Here is the code:

ch15/main.go

func main() {

   i := 3

   ipv := reflect.ValueOf(&i)

   ipv.Elem().SetInt(4)

   fmt.Println(i)

}

In this example, we use the reflect.ValueOf function to obtain a reflect.Value object representing the variable i. Since reflect.ValueOf returns a copy of the value, we need to pass a pointer to the variable in order to modify its value. By calling the Elem method on the value, we get the value pointed to by the pointer, which allows us to modify it using the SetInt method. Finally, we print the modified value of i.

To modify the value of a field in a struct, we can follow a similar approach. Here are the steps:

  1. Pass a pointer to the struct to obtain a reflect.Value object representing it.
  2. Use the Elem method to get the value pointed to by the pointer.
  3. Use the Field method to get the field that needs to be modified.
  4. Use the Set family of methods to modify the field with the desired value.

Here is an example of modifying the Name field of a person struct to “张三”:

ch15/main.go

func main() {

   p := person{Name: "飞雪无情", Age: 20}

   ppv := reflect.ValueOf(&p)

   ppv.Elem().Field(0).SetString("张三")

   fmt.Println(p)

}

type person struct {

   Name string

   Age  int

}

By following the same rules as modifying a variable, we can modify a field in a struct. The field needs to be exported, meaning it starts with a capital letter, in order to be modified using reflection. Additionally, the Elem method is again used to get the value pointed to by the pointer, allowing us to modify the struct.

To summarize the rules for modifying a value using reflection:

  1. The value should be addressable, meaning we need to pass a pointer to the reflect.ValueOf function.
  2. For modifying a field in a struct, the field needs to be exported (starts with a capital letter).
  3. Use the Elem method to get the value pointed to by a pointer before calling the Set family of methods.

By keeping these rules in mind, you can modify variables and fields at runtime using reflection.

Getting the Underlying Type #

The underlying type refers to the base type of a value, such as interfaces, structs, and pointers. In Go, we can define many new types using the type keyword. For example, in the previous example, the actual type of the variable p is person, but the underlying type of p is the struct type. Similarly, the underlying type of &p is a pointer type. We can verify this using the following code:

ch15/main.go

func main() {

   p := person{Name: "飞雪无情", Age: 20}

   ppv := reflect.ValueOf(&p)

   fmt.Println(ppv.Kind())

   pv := reflect.ValueOf(p)

   fmt.Println(pv.Kind())

}

By calling the Kind method on a reflect.Value object, we can get the underlying type of the value. The Kind method returns a constant of type reflect.Kind. The constants available for use are listed in the source code snippet provided above.

reflect.Type #

While reflect.Value is used for operations related to the value, reflect.Type is preferable when dealing with operations related to the variable’s type itself, such as getting the field names or methods of a struct.

To get the reflect.Type of a variable, we can use the function reflect.TypeOf.

The reflect.Type is an interface, not a struct, so it can only be used through its methods.

Here are commonly used methods of the reflect.Type interface. Most of them have similar functionality to the methods of the reflect.Value interface.

type Type interface {
    ...
}

Although reflect.Type and reflect.Value serve different purposes, they both provide useful capabilities for working with reflection in Go.

    package main

    import (
        "encoding/json"
        "fmt"
        "reflect"
    )

    type person struct {
        Name string `json:"name"`
        Age  int    `json:"age"`
    }

    func main() {
        p := person{Name: "飞雪无情", Age: 20}

        // struct to json
        jsonB, err := json.Marshal(p)
        if err == nil {
            fmt.Println(string(jsonB))
        }

        // json to struct
        respJSON := "{\"name\":\"李四\",\"age\":40}"
        json.Unmarshal([]byte(respJSON), &p)
        fmt.Println(p)
    }

The code you provided shows the usage of the reflect.Type methods in Go.

The reflect.Type interface provides methods to manipulate and retrieve information about a specified type. Some of the methods mentioned in the code are:

  1. Implements(u Type) bool: Checks if the specified type implements the interface u.
  2. AssignableTo(u Type) bool: Checks if the type is assignable to the type u. In other words, it checks if the type can be assigned using the assignment operator =.
  3. ConvertibleTo(u Type) bool: Checks if the type can be converted to the type u. It checks if a type conversion can be performed.
  4. Comparable() bool: Checks if the type is comparable. It checks if the type can be compared using relational operators.

The code provides examples of how to use the reflect.Type methods to iterate through the fields and methods of a struct.

You can use the NumField method to get the number of fields in a struct, and then use a for loop with the Field method to iterate through the fields and print their names. Similarly, you can use the same approach to iterate through the methods of a struct.

The code also shows how to check if a struct implements a specific interface using the Implements method of reflect.Type.

Finally, the code demonstrates how to convert between a struct and JSON. The json package in the Go standard library is used to marshal a struct to a JSON string using the json.Marshal function, and unmarshal a JSON string to a struct using the json.Unmarshal function.

Please note that the code has been formatted and adjusted to maintain the original structure while translating. {“Name”:“飞雪无情”,“Age”:20}

Name is 李四,Age is 40

By carefully observing the above printed JSON string, you will find that the keys in the JSON string are the same as the field names in the struct, such as the Name and Age in the example. Can these keys be changed? For example, change them to lowercase name and age, while keeping the field names as uppercase Name and Age. Of course, it is possible, and this can be achieved using struct tags.

Struct Tag #

As the name suggests, a struct tag is a marker added to a struct field, and it is used to perform additional operations, such as converting between JSON and struct. In the example above, if you want to change the keys in the output JSON string to lowercase name and age, you can do so by adding tags to the struct fields. Here is an example code snippet illustrating this:

type person struct {
   Name string `json:"name"`
   Age int `json:"age"`
}

Adding tags to struct fields is simple. Just enclose a key-value pair in backquotes after the field, as shown in the example above, json:"name". Here, the key before the colon is json, and it allows you to obtain the corresponding name after the colon.

Tip: json as a key is a convention in Go’s json package for parsing JSON. It will use the json key to find the corresponding value to be used as the Key in JSON.

By specifying that name and age can be used as the keys in JSON using struct tags, the code can be modified as shown below:

respJSON:="{\"name\":\"李四\",\"age\":40}"

Yes, JSON strings can now use lowercase name and age. If you run this code again, you will see the following result:

{"name":"张三","age":20}
Name is 李四,Age is 40

The keys in the output JSON string are lowercase name and age, and the JSON strings with lowercase name and age can also be converted to person struct.

As you may have noticed, struct tags are crucial for the conversion between JSON and struct. This tag is like an alias we give to struct fields. But how does the json package obtain this tag? This is where reflection comes in. Let’s look at the following code:

// Traverses the struct fields with a tag named "json"
for i := 0; i < pt.NumField(); i++ {
   sf := pt.Field(i)
   fmt.Printf("The json tag on field %s is %s\n", sf.Name, sf.Tag.Get("json"))
}

To obtain the tag on a field, we first need to reflect and obtain the corresponding field. We can achieve this by using the Field method. This method returns a StructField struct, which has a Tag field that contains all the tags on the field. In the example, to obtain the tag with the key json, we simply call sf.Tag.Get(“json”).

Struct fields can have multiple tags for different purposes, such as json conversion, bson conversion, orm parsing, etc. If there are multiple tags, they should be separated by spaces. Different keys can be used to retrieve different tags, as shown in the following code:

// Traverses the struct fields with tags named "json" and "bson"
for i := 0; i < pt.NumField(); i++ {
   sf := pt.Field(i)
   fmt.Printf("The json tag on field %s is %s\n", sf.Name, sf.Tag.Get("json"))
   fmt.Printf("The bson tag on field %s is %s\n", sf.Name, sf.Tag.Get("bson"))
}

type person struct {
   Name string `json:"name" bson:"b_name"`
   Age int `json:"age" bson:"b_name"`
}

When you run the code, you will see the following result:

The json tag on field Name is name
The bson tag on field Name is b_name

Field Age, with key as json and tag as age.

Field Age, with key as bson and tag as b_name.

As you can see, different tags can be obtained by using the Get method with different keys.

Implementing Struct to JSON #

I believe you already understand what a struct tag is. Now I will demonstrate its use through an example of converting a struct to JSON:

func main() {

    p := person{Name: "飞雪无情", Age: 20}

    pv := reflect.ValueOf(p)

    pt := reflect.TypeOf(p)

    // Custom implementation of struct to JSON

    jsonBuilder := strings.Builder{}

    jsonBuilder.WriteString("{")
    
    num := pt.NumField()

    for i := 0; i < num; i++ {

        jsonTag := pt.Field(i).Tag.Get("json") // Get json tag

        jsonBuilder.WriteString("\"" + jsonTag + "\"")

        jsonBuilder.WriteString(":")

        // Get the value of the field

        jsonBuilder.WriteString(fmt.Sprintf("\"%v\"", pv.Field(i)))

        if i < num-1 {

            jsonBuilder.WriteString(",")

        }

    }

    jsonBuilder.WriteString("}")

    fmt.Println(jsonBuilder.String()) // Print JSON string

}

This is a simple example of converting a struct to JSON, but it already demonstrates the use of structs well. In the above example, the custom jsonBuilder is responsible for concatenating the JSON string, and each field is concatenated into the JSON string through a for loop. By running the above code, you can see the following print result:

{"name":"飞雪无情","age":"20"}

Converting JSON strings is just one application of struct tags. You can treat struct tags as metadata configuration for fields in a struct, and use them to do whatever you want, such as ORM mapping, XML conversion, generating Swagger documentation, and so on.

Laws of Reflection #

Reflection is a way for a program to examine its own structure in computer language. It belongs to a form of metaprogramming. Reflection is flexible and powerful, but it can also be unsafe. It can bypass many static checks of the compiler, leading to chaos if used excessively. To help developers better understand reflection, the authors of the Go language summarized the three laws of reflection on their blog.

  1. Any interface{} value can be reflected into a reflection object, which is the reflect.Value and reflect.Type, by using the reflect.ValueOf and reflect.TypeOf functions.
  2. Reflection objects can also be restored to interface{} variables. This is the reversibility of the first law. It is accomplished by using the Interface method of the reflect.Value struct.
  3. To modify a reflection object, the value must be settable and addressable. This means that it must have an address. See the previous lesson on modifying variable values to understand this.

Pro Tip: Any variable of any type can be converted to an empty interface intferface{}. Therefore, the arguments of the reflect.ValueOf and reflect.TypeOf functions in the first law are interface{}, which means that any variable of any type can be converted to a reflection object. In the second law, the value returned by the Interface method of the reflect.Value struct is also interface{}, indicating that reflection objects can be restored to corresponding type variables.

Once you understand these three laws, you can better understand and use reflection in the Go language.

Conclusion #

In reflection, reflect.Value corresponds to the value of a variable. If you need to perform operations related to the value of a variable, you should use reflect.Value first. This includes getting the value of a variable, modifying the value of a variable, etc. reflect.Type corresponds to the type of a variable. If you need to perform operations related to the type itself, you should use reflect.Type first. This includes getting the fields inside a struct, the method set of a type, etc.

I want to reiterate once again: although reflection is powerful and can simplify programming and reduce duplicate code, excessive use of it can make your code complex and messy. So unless absolutely necessary, try to use reflection as little as possible.

go语言15金句.png

The homework for this lesson is to write code to call methods of a struct using reflection.

Next lesson, I will introduce “Unsafe: The Love-Hate Relationship with Non-Type Safety”. Make sure to come and listen to the class!