21 Log Processing Detail Hand Hold Tutorial From 0 to Writing a Logging Package

21 Log Processing Detail Hand-Hold Tutorial from 0 to Writing a Logging Package #

Hello, I am Kong Lingfei.

In the previous lesson, I introduced how to design a log package. Today, we will have a hands-on session where I will guide you step-by-step to write a log package from scratch.

In practical development, we can choose some excellent open-source log packages and use them directly without any modifications. However, most of the time, we develop or modify a log package based on one or several excellent open-source log packages. To develop or modify a log package, we need to understand how log packages are implemented. In this lesson, I will take you from zero to one to implement a log package with basic functionality, allowing you to get a glimpse of the principles and methods of log package implementation.

Before we start the hands-on session, let’s take a look at some excellent open-source log packages currently available in the industry.

What excellent open source logging packages are there? #

In Go project development, we can implement project logging by modifying some excellent open source logging packages. There are many excellent open source logging packages in the Go ecosystem, such as the standard log package, glog, logrus, zap, seelog, zerolog, log15, apex/log, go-logging, etc. Among them, the more commonly used ones are the standard log package, glog, logrus, and zap.

In order to make you understand the current situation of open source logging packages, I will briefly introduce these commonly used logging packages. For their specific usage methods, you can refer to an article I have written: Log package tutorial using excellent open source libraries (written in Chinese).

Standard log package #

The functionality of the standard log package is very simple, providing only three types of functions: Print, Panic, and Fatal, for logging output. Because it is included in the standard library, we do not need to download and install it separately, making it very convenient to use.

The standard log package consists of less than 400 lines of code. If you want to study how to implement a logging package, reading the standard log package is a good starting point. The Go standard library uses the log package extensively, such as in net/http and net/rpc.

glog #

glog is a logging package introduced by Google. Like the standard log package, it is a lightweight logging package that is easy to use. However, glog provides more features than the standard log package, including the following:

  • Supports four log levels: Info, Warning, Error, Fatal.
  • Supports command-line options, such as -alsologtostderr, -log_backtrace_at, -log_dir, -logtostderr, -v, etc., where each parameter implements a specific function.
  • Supports log file rotation based on file size.
  • Supports logging categorized by log level.
  • Supports V level. The V level feature allows developers to define their own log levels.
  • Supports vmodule. vmodule allows developers to use different log levels for different files.
  • Supports traceLocation. traceLocation can print stack information for specific locations.

The Kubernetes project uses klog, which is an encapsulation based on glog, as its logging library.

logrus #

logrus is currently the most starred logging package on GitHub. It is known for its powerful functionality, high performance, and high flexibility, and also provides the ability to customize plugins. Many excellent open source projects, such as Docker and Prometheus, use logrus. Besides the basic logging functionality, logrus also has the following features:

  • Supports common log levels. logrus supports log levels including Debug, Info, Warn, Error, Fatal, and Panic.
  • Extensible. logrus’s Hook mechanism allows users to distribute logs to any location, such as local files, standard output, Elasticsearch, Logstash, Kafka, etc.
  • Supports custom log formats. logrus provides two built-in formats: JSONFormatter and TextFormatter. In addition, logrus allows users to define their own log formats by implementing the Formatter interface.
  • Structured logging. logrus’s Field mechanism allows users to customize log fields instead of using verbose messages to record logs.
  • Default fields. logrus’s Default Fields mechanism allows adding common log fields to a subset or all logs, such as adding the X-Request-ID field to all logs of a specific HTTP request.
  • Fatal handlers. logrus allows registering one or more handlers that will be called when a log with the Fatal level is produced. This feature is useful when our program needs to gracefully shut down.

zap #

zap is an open source logging package developed by Uber, known for its high performance. Many companies’ logging packages are derived from modification of zap. In addition to the basic logging functions, zap also has many powerful features:

  • Supports common log levels, such as Debug, Info, Warn, Error, DPanic, Panic, Fatal.
  • Very high performance. zap has excellent performance and is suitable for scenarios with high performance requirements.
  • Supports outputting call stacks for specific log levels.
  • Similar to logrus, zap also supports structured logging, default fields, and extensibility due to its support for hooks.

Choosing an open source logging package #

I have introduced many logging packages above, each of which has different usage scenarios. You can choose based on your own requirements and the features of the logging package:

  • Standard log package: The standard log package does not support log levels, log rotation, log formats, etc., so it is rarely used directly in large projects. It is usually used in some short programs, such as in the main.go file used to generate JWT Tokens. The standard log package is also suitable for short pieces of code used for quick debugging and verification.
  • glog: glog implements basic logging functionalities and is suitable for small projects with few logging requirements.
  • logrus: logrus is powerful, not only implementing basic logging functionality but also having many advanced features. It is suitable for large projects, especially those that require structured logging.
  • zap: zap provides powerful logging capabilities with high performance and low memory allocation. It is suitable for projects with high logging performance requirements. In addition, the subpackage zapcore in the zap package provides many low-level log interfaces, which are suitable for secondary encapsulation.

As an example of choosing a logging package for secondary development, when developing a container cloud platform, I found that the Kubernetes source code extensively uses glog, so the logging package needs to be compatible with glog. Therefore, based on zap and zapcore, I encapsulated the github.com/marmotedu/iam/pkg/log logging package, which is well compatible with glog.

In practical project development, you can choose from the above logging packages according to the needs of the project and use them directly. However, most of the time, you will still need to customize and develop based on these packages. In order for you to have a deeper understanding of the design and development of logging packages, next, I will take you from 0 to 1 to develop a logging package.

Writing a Logging Package From Scratch #

Next, I will show you how to quickly write a logging package with basic functionality, so that you can grasp the core design ideas of a logging package through this short example. This logging package mainly achieves the following functionalities:

  • Support custom configurations.
  • Support file names and line numbers.
  • Support log levels: Debug, Info, Warn, Error, Panic, Fatal.
  • Support output to local files and standard output.
  • Support JSON and TEXT format log outputs, and support custom log formats.
  • Support option mode.

The logging package is named cuslog, and the complete code of the example project is stored in the cuslog repository.

The implementation can be divided into the following four steps:

  1. Definition: Define log levels and log options.
  2. Creation: Create a Logger and log printing methods for each log level.
  3. Writing: Output the log to the supported output.
  4. Customization: Customize the log output format.

Define Log Levels and Log Options #

For a basic logging package, log levels and log options need to be defined first. In this example, the code is defined in the options.go file.

Log levels can be defined in the following way:

type Level uint8

const (
    DebugLevel Level = iota
    InfoLevel
    WarnLevel
    ErrorLevel
    PanicLevel
    FatalLevel
)

var LevelNameMapping = map[Level]string{
    DebugLevel: "DEBUG",
    InfoLevel:  "INFO",
    WarnLevel:  "WARN",
    ErrorLevel: "ERROR",
    PanicLevel: "PANIC",
    FatalLevel: "FATAL",
}

When outputting logs, the comparison between the switch level and the output level is used to determine whether to output. Therefore, the log level Level needs to be defined as a numerical type that is easy to compare. Almost all logging packages use the constant counter iota to define log levels.

Furthermore, because readable log levels need to be output in the log output (for example, output “INFO” instead of “1”), a mapping between Level and Level Name is needed. LevelNameMapping will be used during formatting.

Next, let’s look at defining log options. Logs should be configurable to allow developers to set different log behaviors according to different environments. Common configuration options include:

  • Log level.
  • Output destination, such as standard output or file.
  • Output format, such as JSON or Text.
  • Whether to enable file names and line numbers.

The log options in this example are defined as follows:

type options struct {
    output        io.Writer
    level         Level
    stdLevel      Level
    formatter     Formatter
    disableCaller bool
}

To flexibly set log options, you can use the option mode to set log options:

type Option func(*options)

func initOptions(opts ...Option) (o *options) {
    o = &options{}
    for _, opt := range opts {
        opt(o)
    }

    if o.output == nil {
        o.output = os.Stderr
    }

    if o.formatter == nil {
        o.formatter = &TextFormatter{}
    }

    return
}

func WithLevel(level Level) Option {
    return func(o *options) {
        o.level = level
    }
}
...
func SetOptions(opts ...Option) {
    std.SetOptions(opts...)
}

func (l *logger) SetOptions(opts ...Option) {
    l.mu.Lock()
    defer l.mu.Unlock()

    for _, opt := range opts {
        opt(l.opt)
    }
}

A logging package with the option mode can dynamically modify the log options as follows:

cuslog.SetOptions(cuslog.WithLevel(cuslog.DebugLevel))

You can create a setting function WithXXXX for each log option as needed. This example logging package supports the following setting functions:

  • WithOutput(output io.Writer): set the output destination.
  • WithLevel(level Level): set the output level.
  • WithFormatter(formatter Formatter): set the output format.
  • WithDisableCaller(caller bool): set whether to print file names and line numbers.

Create a Logger and Log Printing Methods for Each Log Level #

In order to print logs, we need to create a Logger based on log configurations, and then use the log printing methods of the Logger to complete the output of logs at different levels. The code for creating a Logger is saved in the logger.go file.

A Logger can be created in the following way:

var std = New()

type logger struct {
    opt       *options
    mu        sync.Mutex
    entryPool *sync.Pool
}

func New(opts ...Option) *logger {
    logger := &logger{opt: initOptions(opts...)}
    logger.entryPool = &sync.Pool{New: func() interface{} { return entry(logger) }}
    return logger
}

Above the code, a Logger is defined and the New function for creating a Logger is implemented. Log packages usually have a default global Logger. In this example, a global default Logger is created by calling `var std = New()`. The functions cuslog.Debug, cuslog.Info, cuslog.Warnf, and others, print logs by calling the methods provided by the std Logger.

After defining a Logger, the most essential logging methods need to be added to the Logger to provide support for all levels of logging.

If the log level is Xyz, it usually requires two types of methods: non-formatted method `Xyz(args ...interface{})` and formatted method `Xyzf(format string, args ...interface{})`, for example:

```Go
func (l *logger) Debug(args ...interface{}) {
    l.entry().write(DebugLevel, FmtEmptySeparate, args...)
}

func (l *logger) Debugf(format string, args ...interface{}) {
    l.entry().write(DebugLevel, format, args...)
}

This example implements the following methods: Debug, Debugf, Info, Infof, Warn, Warnf, Error, Errorf, Panic, Panicf, Fatal, and Fatalf. For a more detailed implementation, you can refer to cuslog/logger.go.

Note that Panic, Panicf should call the panic() function, and Fatal, Fatalf functions should call the os.Exit(1) function.

Output Logs to Supported Destinations #

After calling the log printing functions, it is necessary to output these logs to the supported destinations. Therefore, the write function needs to be implemented, and its writing logic is saved in the entry.go file. The implementation is as follows:

type Entry struct {
    logger *logger
    Buffer *bytes.Buffer
    Map    map[string]interface{}
    Level  Level
    Time   time.Time
    File   string
    Line   int
    Func   string
    Format string
    Args   []interface{}
}

func (e *Entry) write(level Level, format string, args ...interface{}) {
    if e.logger.opt.level > level {
        return
    }
    e.Time = time.Now()
    e.Level = level
    e.Format = format
    e.Args = args
    if !e.logger.opt.disableCaller {
        if pc, file, line, ok := runtime.Caller(2); !ok {
            e.File = "???"
            e.Func = "???"
        } else {
            e.File, e.Line, e.Func = file, line, runtime.FuncForPC(pc).Name()
            e.Func = e.Func[strings.LastIndex(e.Func, "/")+1:]
        }
    }
    e.format()
    e.writer()
    e.release()
}

func (e *Entry) format() {
    _ = e.logger.opt.formatter.Format(e)
}

func (e *Entry) writer() {
    e.logger.mu.Lock()
    _, _ = e.logger.opt.output.Write(e.Buffer.Bytes())
    e.logger.mu.Unlock()
}

func (e *Entry) release() {
    e.Args, e.Line, e.File, e.Format, e.Func = nil, 0, "", "", ""
    e.Buffer.Reset()
    e.logger.entryPool.Put(e)
}

In the above code, first, a structure type Entry is defined. This type is used to store all the log information, including log configuration and log content. The writing logic is based on an instance of the Entry type.

The Entry’s write method is used to write the logs. In the write method, it first checks the log output level and the switch level. If the output level is lower than the switch level, it returns without recording anything.

In write, it also checks whether the file name and line number need to be recorded. If so, it calls runtime.Caller() to obtain the file name and line number. When calling runtime.Caller(), it is important to pass the correct stack depth.

The write function calls e.format() to format the log and e.writer() to write the log. In the log configuration passed when creating the Logger, the output location output io.Writer is specified. The type of output is io.Writer. An example is as follows:

type Writer interface {
    Write(p []byte) (n int, err error)
}

io.Writer implements the Write method for writing, so it is only necessary to call e.logger.opt.output.Write(e.Buffer.Bytes()) to write the log to the specified location. Finally, the release method is called to clear the cache and object pool. At this point, recording and writing of logs are completed.

Custom Log Output Format #

The cuslog package supports custom output formats and provides built-in JSON and Text formatters. The Formatter interface is defined as:

type Formatter interface {
    Format(entry *Entry) error
}

cuslog has two built-in formatters: JSON and TEXT.

Testing the Log Package #

After cuslog log package development is complete, test code can be written to test the cuslog package by calling the cuslog package. The code is as follows:

package main

import (
    "log"
    "os"

    "github.com/marmotedu/gopractise-demo/log/cuslog"
)

func main() {
    cuslog.Info("std log")
    cuslog.SetOptions(cuslog.WithLevel(cuslog.DebugLevel))
    cuslog.Debug("change std log to debug level")
    cuslog.SetOptions(cuslog.WithFormatter(&cuslog.JsonFormatter{IgnoreBasicFields: false}))
    cuslog.Debug("log in json format")
    cuslog.Info("another log in json format")

    // Export to a file
    fd, err := os.OpenFile("test.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        log.Fatalln("create file test.log failed")
    }
    defer fd.Close()

    l := cuslog.New(cuslog.WithLevel(cuslog.InfoLevel),
        cuslog.WithOutput(fd),
        cuslog.WithFormatter(&cuslog.JsonFormatter{IgnoreBasicFields: false}),
    )
    l.Info("custom log with json formatter")
}

Save the above code in the main.go file and run:

$ go run example.go
2020-12-04T10:32:12+08:00 INFO example.go:11 std log
2020-12-04T10:32:12+08:00 DEBUG example.go:13 change std log to debug level
{"file":"/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/log/cuslog/example/example.go:15","func":"main.main","message":"log in json format","level":"DEBUG","time":"2020-12-04T10:32:12+08:00"}
{"level":"INFO","time":"2020-12-04T10:32:12+08:00","file":"/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/log/cuslog/example/example.go:16","func":"main.main","message":"another log in json format"}

With this, the log package is complete. The complete package can be found at log/cuslog.

IAM Project Log Package Design #

At the end of this lecture, let’s take a look at how the log package is designed in our IAM project.

First, let’s take a look at the storage location of the log package in the IAM project: pkg/log. It is placed in this location for two main reasons: first, the log package belongs to the IAM project and has customized development content; second, the log package is feature-complete and mature, and can be used by external projects as well.

The log package is based on the go.uber.org/zap package, and has added more features as needed. Next, let’s take a look at the features implemented by the log package through the Options of the log package:

type Options struct {
    OutputPaths       []string `json:"output-paths"       mapstructure:"output-paths"`
    ErrorOutputPaths  []string `json:"error-output-paths" mapstructure:"error-output-paths"`
    Level             string   `json:"level"              mapstructure:"level"`
    Format            string   `json:"format"             mapstructure:"format"`
    DisableCaller     bool     `json:"disable-caller"     mapstructure:"disable-caller"`
    DisableStacktrace bool     `json:"disable-stacktrace" mapstructure:"disable-stacktrace"`
    EnableColor       bool     `json:"enable-color"       mapstructure:"enable-color"`
    Development       bool     `json:"development"        mapstructure:"development"`
    Name              string   `json:"name"               mapstructure:"name"`
}

The meanings of the various configuration options in Options are as follows:

  • development: Whether it is in development mode. If it is in development mode, the stack trace will be recorded for DPanicLevel.
  • name: The name of the logger.
  • disable-caller: Whether to enable the caller. If enabled, the file, function, and line number of the log call will be displayed in the log.
  • disable-stacktrace: Whether to disable printing stack trace information for Panic and higher levels.
  • enable-color: Whether to enable color output. true for yes, false for no.
  • level: Log level, with the following priorities from low to high: Debug, Info, Warn, Error, Dpanic, Panic, Fatal.
  • format: Supported log output formats, currently supports Console and JSON. Console is essentially Text format.
  • output-paths: Supports output to multiple destinations, separated by commas. Supports output to standard output (stdout) and files.
  • error-output-paths: Output paths for internal (non-business) error logs of zap, multiple outputs separated by commas.

The Options structure of the log package supports the following three methods:

  • Build method: The Build method can build a global Logger based on Options.
  • AddFlags method: The AddFlags method can append the fields of Options to the passed-in pflag.FlagSet variable.
  • String method: The String method can return the value of Options as a JSON formatted string.

The log package implements the following three logging methods:

log.Info("This is an info message", log.Int32("int_key", 10))
log.Infof("This is a formatted %s message", "info")
log.Infow("Message printed with Infow", "X-Request-ID", "fbf54504-64da-4088-9b86-67824a7fb508")

Info logs a message with the specified key/value pairs. Infof logs a formatted message. Infow also logs a message with specified key/value pairs. The difference between Infow and Info is that Info requires specifying the type of value. By specifying the log type of the value, the underlying log library does not need to perform reflection operations, so using Info to log messages has the highest performance.

The log package supports a wide range of types, which you can refer to types.go.

The output for the above log statements is:

2021-07-06 14:02:07.070 INFO This is an info message {"int_key": 10}
2021-07-06 14:02:07.071 INFO This is a formatted info message
2021-07-06 14:02:07.071 INFO Message printed with Infow {"X-Request-ID": "fbf54504-64da-4088-9b86-67824a7fb508"}

For each log level, the log package provides three logging methods. For example, assuming the log format is Xyz, it provides Xyz(msg string, fields ...Field), Xyzf(format string, v ...interface{}), and Xyzw(msg string, keysAndValues ...interface{}) three logging methods.

In addition, compared to regular log packages, the log package provides many methods for logging.

The first method is that the log package supports V Level, which allows flexible specification of log levels using integers. The higher the numerical value, the lower the priority. For example:

// Usage of V level
log.V(1).Info("This is a V level message")
log.V(1).Infof("This is a %s V level message", "formatted")
log.V(1).Infow("This is a V level message with fields", "X-Request-ID", "7a7b9f24-4cae-4b2a-9464-69088b45b904")

Note that Log.V only supports Info, Infof, and Infow logging methods.

The second method is that the log package supports the use of WithValues function, for example:

// Usage of WithValues
lv := log.WithValues("X-Request-ID", "7a7b9f24-4cae-4b2a-9464-69088b45b904")
lv.Infow("Info message printed with [WithValues] logger")
lv.Infow("Debug message printed with [WithValues] logger")

The above log output is as follows:

2021-07-06 14:15:28.555 INFO Info message printed with [WithValues] logger {"X-Request-ID": "7a7b9f24-4cae-4b2a-9464-69088b45b904"}
2021-07-06 14:15:28.556 INFO Debug message printed with [WithValues] logger {"X-Request-ID": "7a7b9f24-4cae-4b2a-9464-69088b45b904"}

WithValues can return a logger with specified key-value pairs for later use.

The third method is that the log package provides WithContext and FromContext to add a specified logger to a Context and retrieve a logger from a Context, for example:

// Usage of Context
ctx := lv.WithContext(context.Background())

lc := log.FromContext(ctx)
lc.Info("Message printed with [WithContext] logger")

WithContext and FromContext are very suitable for use in functions passed with context.Context, for example:

func main() {
 
    ...
 
    // Using WithValues
    lv := log.WithValues("X-Request-ID", "7a7b9f24-4cae-4b2a-9464-69088b45b904")
     
    // Using Context
    lv.Infof("Start to call pirntString")
    ctx := lv.WithContext(context.Background())
    pirntString(ctx, "World")  
}
 
func pirntString(ctx context.Context, str string) {
    lc := log.FromContext(ctx)
    lc.Infof("Hello %s", str)
}

The above code will output:

2021-07-06 14:38:02.050 INFO Start to call pirntString {"X-Request-ID": "7a7b9f24-4cae-4b2a-9464-69088b45b904"}
2021-07-06 14:38:02.050 INFO Hello World {"X-Request-ID": "7a7b9f24-4cae-4b2a-9464-69088b45b904"}

By adding the Logger to the Context and passing it between different functions, key-value pairs can be propagated between functions. In the above code, the X-Request-ID is logged in both the main function and the printString function, achieving a kind of call chain effect.

The fourth method allows specific key-value pairs to be extracted from the Context and added to the log output as context, for example, the log call in the internal/apiserver/api/v1/user/create.go file:

log.L(c).Info("user create function called.")

Implemented as follows by calling the Log.L() function:

// L method output with specified context value.
func L(ctx context.Context) *zapLogger {
    return std.L(ctx)
}
 
func (l *zapLogger) L(ctx context.Context) *zapLogger {
    lg := l.clone()
 
    requestID, _ := ctx.Value(KeyRequestID).(string)
    username, _ := ctx.Value(KeyUsername).(string)
    lg.zapLogger = lg.zapLogger.With(zap.String(KeyRequestID, requestID), zap.String(KeyUsername, username))
 
    return lg
}

The L() method extracts the requestID and username from the passed-in Context and appends them to the Logger, returning the modified Logger. When using the Info, Infof, Infow, and other methods of this Logger to record logs, the log output will contain the requestID and username fields, for example:

2021-07-06 14:46:00.743 INFO    apiserver       secret/create.go:23     create secret function called.  {"requestID": "73144bed-534d-4f68-8e8d-dc8a8ed48507", "username": "admin"}

By passing the Context between functions, it is easy to achieve a call chain effect, for example:

// Create add new secret key pairs to the storage.
func (s *SecretHandler) Create(c *gin.Context) {
    log.L(c).Info("create secret function called.")
     
    ...
     
    secrets, err := s.srv.Secrets().List(c, username, metav1.ListOptions{    
        Offset: pointer.ToInt64(0),
        Limit:  pointer.ToInt64(-1),
    })
     
    ...
     
     if err := s.srv.Secrets().Create(c, &r, metav1.CreateOptions{}); err != nil {
        core.WriteResponse(c, err, nil)

        return
    }
 
    core.WriteResponse(c, nil, r)
}

The output of the above code is:

2021-07-06 14:46:00.743 INFO    apiserver       secret/create.go:23     create secret function called.  {"requestID": "73144bed-534d-4f68-8e8d-dc8a8ed48507", "username": "admin"}
2021-07-06 14:46:00.744 INFO    apiserver       secret/create.go:23     list secret from storage.  {"requestID": "73144bed-534d-4f68-8e8d-dc8a8ed48507", "username": "admin"}
2021-07-06 14:46:00.745 INFO    apiserver       secret/create.go:23     insert secret to storage.  {"requestID": "73144bed-534d-4f68-8e8d-dc8a8ed48507", "username": "admin"}

Note that the log.L function by default retrieves the requestID and username keys from the Context, and this is coupled with the IAM project, but it does not affect the use of the log package by third-party projects. This is also why I recommend you to wrap your own log package.

Summary #

When developing a logging package, we often need to build upon existing open-source logging packages. Currently, many projects use the zap logging package for encapsulation. If you have a need for encapsulation, I recommend choosing the zap logging package.

In this lecture, I introduced four commonly used logging packages: the standard log package, glog, logrus, and zap. I then showed you the four steps to develop a logging package, which are as follows:

  1. Define log levels and log options.
  2. Create a Logger and log printing methods for each level.
  3. Output logs to supported destinations.
  4. Customize the log output format.

Finally, I introduced the design and usage of the log package in the IAM project. The log package is encapsulated based on go.uber.org/zap and provides the following powerful features:

  • The log package supports V levels, allowing flexible specification of log levels using integer values.
  • The log package supports the WithValues function, which returns a Logger with specified key-value pairs for later use.
  • The log package provides WithContext and FromContext to add a specified Logger to a Context and retrieve a Logger from a Context.
  • The log package provides the Log.L() function, which conveniently extracts specified key-value pairs from a Context as context to add to log outputs.

Exercise #

  1. Try to implement a new Formatter that can output different log levels in different colors (for example, the string ‘Error’ in red font and the string ‘Info’ in white font)。

  2. Try changing the 2 in the function call to 1 in runtime.Caller(2) and see if there is any difference in the log output compared to before the modification. If there is a difference, think about the reason for the difference.

Feel free to leave a message in the comments section to discuss and exchange ideas. See you in the next lesson.