14 Project Management How to Write High Quality Makefile

14 Project Management How to Write High-Quality Makefile #

Hello, I’m Kong Lingfei. Today, let’s talk about how to write high-quality Makefiles.

In Lesson 10, we learned that to create an elegant Go project, it is not only important to develop an excellent Go application, but also to efficiently manage the project. One effective approach is to use Makefiles to manage our projects, which requires us to write Makefile files for our projects.

When communicating with other developers, I found that everyone recognizes the powerful project management capabilities of Makefiles and can write their own Makefiles. However, some of them do a poor job at project management. After further communication with them, I discovered that these colleagues are simply using the basic syntax of Makefiles to repetitively write low-quality Makefiles, without fully leveraging the capabilities of Makefiles.

Let me give you an example to help you understand what a low-quality Makefile looks like:

build: clean vet
	@mkdir -p ./Role
	@export GOOS=linux && go build -v .

vet:
	go vet ./...

fmt:
	go fmt ./...

clean:
	rm -rf dashboard

The above Makefile has several issues. For example, it has limited functionality and can only perform basic operations such as compiling and formatting code. It lacks advanced features such as building images and generating code. It also lacks extensibility, as it cannot compile binary files that can be run on Mac. Additionally, there is no Help function, making it difficult to use. Moreover, it consists of a single Makefile file with a simple structure, which is not suitable for adding complex management functions.

Therefore, we not only need to write Makefiles, but we also need to write high-quality Makefiles. So, how can we write a high-quality Makefile? I believe that can be achieved through the following four methods:

Lay a solid foundation, which means mastering the syntax of Makefiles.
Do the necessary preparation, that is, plan in advance the functionalities the Makefile needs to achieve.
Plan accordingly and design a reasonable Makefile structure.
Master the techniques and make good use of Makefile writing skills.

Next, let’s take a detailed look at these methods.

Proficient in Makefile Syntax #

To do a good job, you must first sharpen your tools. The first step in writing high-quality Makefiles is to be proficient in the core syntax of Makefiles.

Because Makefile syntax is quite extensive, I have included some suggestions for syntax that you should focus on mastering in the upcoming special edition. This includes Makefile rule syntax, phony targets, variable assignments, conditional statements, and common Makefile functions, among others.

If you want to study the syntax of Makefile in depth and comprehensively, I recommend that you learn from Chen Hao’s book “A Tutorial on Makefile”.

Functionalities to be implemented in Makefile #

Next, we need to plan the functionalities to be implemented in the Makefile. Planning the functionalities in advance is beneficial for designing the overall structure and implementation approach of the Makefile.

Different projects have different functionalities in their Makefiles. Some of these functionalities are achieved through target files, but more functionalities are achieved through pseudo-targets. For Go projects, although the integrated functionalities may vary across projects, most projects need to implement some common functionalities. Now, let’s take a look at the functionalities typically implemented in a large Go project’s Makefile.

Below is a list of functionalities integrated into the IAM project’s Makefile, which may be helpful for your future Makefile design:

$ make help

Usage: make <TARGETS> <OPTIONS> ...

Targets:
  # Code generation commands
  gen                Generate all necessary files, such as error code files.

  # Formatting commands
  format             Gofmt (reformat) package sources (exclude vendor dir if existed).

  # Static code checking
  lint               Check syntax and styling of go sources.

  # Testing commands
  test               Run unit test.
  cover              Run unit test and get test coverage.

  # Build commands
  build              Build source code for the host platform.
  build.multiarch    Build source code for multiple platforms. See option PLATFORMS.

  # Docker image packaging commands
  image              Build docker images for the host architecture.
  image.multiarch    Build docker images for multiple platforms. See option PLATFORMS.
  push               Build docker images for the host architecture and push images to the registry.
  push.multiarch     Build docker images for multiple platforms and push images to the registry.

  # Deployment commands
  deploy             Deploy updated components to development environment.

  # Cleaning commands
  clean              Remove all files that are created by building.

  # Other commands, may vary for different projects
  release            Release IAM.
  verify-copyright   Verify the boilerplate headers for all files.
  ca                 Generate CA files for all IAM components.
  install            Install IAM system with all its components.
  swagger            Generate Swagger document.
  tools              Install dependent tools.

  # Help command
  help               Show this help info.

# Options
Options:
  DEBUG        Whether to generate debug symbols. Default is 0.
  BINS         The binaries to build. Default is all of cmd.
               This option is available when using: make build/build.multiarch
               Example: make build BINS="iam-apiserver iam-authz-server"
  ...

For more detailed commands, you can execute make help in the root directory of the IAM project repository.

Generally, a Makefile for a Go project should implement the following functionalities: code formatting, static code checking, unit testing, code building, file cleaning, help, and so on. If deploying through Docker, it should also have the functionality of Docker image packaging. Since Go is a cross-platform language, the build and Docker packaging commands should also support different CPU architectures and platforms. To have better control over the behavior of Makefile commands, support for options is necessary.

To facilitate the viewing of integrated functionalities in the Makefile, we need to support the help command. It would be best if the help command outputs the integrated functionalities by parsing the Makefile, for example:

## help: Show this help info.
.PHONY: help
help: Makefile
  @echo -e "\nUsage: make <TARGETS> <OPTIONS> ...\n\nTargets:"
  @sed -n 's/^##//p' $< | column -t -s ':' | sed -e 's/^/ /'
  @echo "$$USAGE_OPTIONS"

The help command above parses the ## comments in the Makefile to obtain the supported commands. By using this approach, we don’t need to modify the help command when adding new commands in the future.

You can refer to the aforementioned Makefile management functionality and combine it with the requirements of your own project to compile a list of functionalities to be implemented in the Makefile. Then, you can initially determine the implementation approach and methods. Once you have completed these tasks, your preparation work is basically done.

Design a Reasonable Makefile Structure #

After designing the functionality of the Makefile, we now enter the stage of writing the Makefile. The first step in the writing stage is to design a reasonable Makefile structure.

For large projects, there is a lot of content to manage. If all management functions are integrated into one Makefile, it may become very large and difficult to read and maintain. Therefore, it is recommended to use a layered design approach, where the root Makefile aggregates all the Makefile commands, and the specific implementations are classified and placed in separate Makefiles according to functionality.

We often integrate shell scripts into Makefile commands. However, if the shell script is too complex, it can also result in a Makefile with too much content that is difficult to read and maintain. Integrating complex shell scripts into Makefile also leads to a poor writing experience. In this case, you can encapsulate the complex shell commands in a shell script, which can be directly called by the Makefile, while simple commands can be directly integrated into the Makefile.

Therefore, the final recommended Makefile structure is as follows:

In the above Makefile organization, the root Makefile aggregates all the management functions of the project using pseudo targets. At the same time, these pseudo targets are classified and placed in separate Makefiles in the same category, making the Makefile easier to maintain. For complex commands, they are written in separate shell scripts and called by the Makefile commands.

As an example, here is an excerpt of the Makefile organization structure for the IAM project:

├── Makefile
├── scripts
│   ├── gendoc.sh
│   ├── make-rules
│   │   ├── gen.mk
│   │   ├── golang.mk
│   │   ├── image.mk
│   │   └── ...
    └── ...

We put operations of the same category under the scripts/make-rules directory. The name of the Makefile is named after the category, such as golang.mk. Finally, in the /Makefile, include these Makefiles.

To match the hierarchical structure of the Makefile, all targets in golang.mk are named in the format go.xxx. With this naming convention, we can easily distinguish what a target does and which file it belongs to, which is especially useful in complex Makefiles. Here is an excerpt of the contents of the Makefile in the root directory of the IAM project, which you can refer to:

include scripts/make-rules/golang.mk
include scripts/make-rules/image.mk
include scripts/make-rules/gen.mk
include scripts/make-rules/...

## build: Build source code for host platform.
.PHONY: build
build:
	@$(MAKE) go.build

## build.multiarch: Build source code for multiple platforms. See option PLATFORMS.
.PHONY: build.multiarch
build.multiarch:
	@$(MAKE) go.build.multiarch

## image: Build docker images for host arch.
.PHONY: image
image:
	@$(MAKE) image.build

## push: Build docker images for host arch and push images to registry.
.PHONY: push
push:
	@$(MAKE) image.push

## ca: Generate CA files for all iam components.
.PHONY: ca
ca:
	@$(MAKE) gen.ca

Furthermore, a reasonable Makefile structure should be forward-looking. That is, it should accommodate future functionalities without changing the existing structure. This requires you to organize the current functionalities that the Makefile needs to implement, the upcoming functionalities, and the possible future functionalities, and then, based on these functionalities, write an extensible Makefile using Makefile programming techniques.

Here, note that the above Makefile defines a large number of pseudo targets using .PHONY. When defining pseudo targets, be sure to use .PHONY to indicate them. Otherwise, when there are files with the same name, the pseudo targets may not be executed.

Mastering Makefile Writing Techniques #

Finally, you also need to master some writing techniques in Makefile, which can make your Makefile more extensible and powerful.

Next, I will share with you some Makefile writing techniques that I have accumulated during my long-term development process. You need to practice more in actual writing and form writing habits.

Technique 1: Make Good Use of Wildcards and Automatic Variables #

Makefile allows you to perform regular expression-like matching on targets, mainly using the wildcard %. By using wildcards, you can make different targets use the same rules, making the Makefile more extensible and concise.

In our IAM project, a large number of wildcards % are used, such as go.build.%, ca.gen.%, deploy.run.%, tools.verify.%, tools.install.%, etc.

Here, let’s take a look at a specific example, tools.verify.% (located in the scripts/make-rules/tools.mk file), defined as follows:

tools.verify.%:
  @if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi

make tools.verify.swagger, make tools.verify.mockgen, etc. can all use the rules defined above, where % represents swagger and mockgen respectively.

If we don’t use %, we would need to define separate rules for tools.verify.swagger and tools.verify.mockgen, which would be cumbersome and difficult to modify later.

In addition, the benefits of naming in the format of tools.verify.% can also be seen here: the definition of the dependency tools is in the Makefile file scripts/make-rules/tools.mk; verify indicates that the pseudo-target tools.verify.% belongs to the verify category and is mainly used to verify tool installation. With this naming convention, you can easily know which Makefile file the target is in and what functionality you want to achieve.

In addition, the definition above also uses the automatic variable $*, which represents the matched values swagger and mockgen.

Technique 2: Make Good Use of Functions #

The built-in functions in Makefile can help us achieve powerful functionality. Therefore, when we write Makefile, if there is a functional requirement, we can prioritize using these functions. I have listed the commonly used functions and their implementation functions in Makefile Common Function List, which you can refer to.

The IAM Makefile files use a large number of the above functions. If you want to see the specific usage methods and scenarios of these functions, you can refer to the Makefile files in the IAM project make-rules.

Technique 3: Depend on the Tools Needed #

If a command in Makefile requires a certain tool, you can include that tool as a dependency of the target. In this way, when executing the target, you can check if the system has installed the tool and automatically install it if it is not installed, achieving a higher level of automation. For example, in the Makefile file, the format pseudo-target is defined as follows:

.PHONY: format
format: tools.verify.golines tools.verify.goimports
  @echo "===========> Formating codes"
  @$(FIND) -type f -name '*.go' | $(XARGS) gofmt -s -w
  @$(FIND) -type f -name '*.go' | $(XARGS) goimports -w -local $(ROOT_PACKAGE)
  @$(FIND) -type f -name '*.go' | $(XARGS) golines -w --max-len=120 --reformat-tags --shorten-comments --ignore-generated .

You can see that format depends on tools.verify.golines tools.verify.goimports. Let’s take a look at the definition of tools.verify.golines:

tools.verify.%:
  @if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi

And let’s take a look at the tools.install.$* rule:

.PHONY: install.golines
install.golines:
  @$(GO) get -u github.com/segmentio/golines

With the tools.verify.% rule defined, we can know that tools.verify.% first checks if the tool is installed. If it is not installed, it will execute tools.install.$* to install it. In this way, when we execute the tools.verify.% target, if the system does not have the golines command installed, it will automatically call go get to install it, improving the automation level of the Makefile.

Technique 4: Put Common Functions in the /Makefile, and Less Common Functions in Categorized Makefiles #

For a project, especially a large project, there are many areas that need to be managed, most of which can be automated using Makefile. However, in order to keep the /Makefile file clean, we cannot add all commands to the /Makefile file.

A good suggestion is to put commonly used functions in the /Makefile, and less common functions in categorized Makefiles, and include these categorized Makefiles in the /Makefile. For example, the format, lint, test, build, and other common commands are integrated into the /Makefile of the IAM project, while less commonly used functions such as gen.errcode.code and gen.errcode.doc are placed in the scripts/make-rules/gen.mk file. Of course, we can also directly execute make gen.errcode.code to execute the gen.errcode.code pseudo-target. In this way, we can ensure that the /Makefile is simple and easy to maintain, and we can also run pseudo-targets through the make command, making it more flexible.

Tip 5: Writing an Extensible Makefile #

What does it mean to write an extensible Makefile? In my opinion, an extensible Makefile has two aspects:

It allows new functionality to be added without changing the structure of the Makefile.
When expanding the project, the new functionality can be automatically incorporated into the existing logic of the Makefile.

To achieve the first aspect, we can design a reasonable structure for the Makefile. To achieve the second aspect, we need to use some techniques when writing the Makefile, such as using wildcards, automatic variables, and functions. Let’s take a look at an example to help you understand better.

In the golang.mk of our IAM project, when executing make go.build, all components under the cmd/ directory can be built. In other words, when new components are added, make go.build can still build the newly added components. This achieves the second aspect mentioned above.

Here’s the specific implementation:

COMMANDS ?= $(filter-out %.md, $(wildcard ${ROOT_DIR}/cmd/*))
BINS ?= $(foreach cmd,${COMMANDS},$(notdir ${cmd}))

.PHONY: go.build
go.build: go.build.verify $(addprefix go.build., $(addprefix $(PLATFORM)., $(BINS)))
.PHONY: go.build.%               

go.build.%:             
  $(eval COMMAND := $(word 2,$(subst ., ,$*)))
  $(eval PLATFORM := $(word 1,$(subst ., ,$*)))
  $(eval OS := $(word 1,$(subst _, ,$(PLATFORM))))           
  $(eval ARCH := $(word 2,$(subst _, ,$(PLATFORM))))                         
  @echo "===========> Building binary $(COMMAND) $(VERSION) for $(OS) $(ARCH)"
  @mkdir -p $(OUTPUT_DIR)/platforms/$(OS)/$(ARCH)
  @CGO_ENABLED=0 GOOS=$(OS) GOARCH=$(ARCH) $(GO) build $(GO_BUILD_FLAGS) -o $(OUTPUT_DIR)/platforms/$(OS)/$(ARCH)/$(COMMAND)$(GO_OUT_EXT) $(ROOT_PACKAGE)/cmd/$(COMMAND)

When executing make go.build, it will execute the dependency of go.build, which is $(addprefix go.build., $(addprefix $(PLATFORM)., $(BINS))). The addprefix function returns a string like go.build.linux_amd64.iamctl go.build.linux_amd64.iam-authz-server go.build.linux_amd64.iam-apiserver ..., and then the go.build.% pseudo-target will be executed.

In the go.build.% pseudo-target, the values of COMMAND, PLATFORM, OS, and ARCH are calculated using the eval, word, and subst functions. Finally, the component to be built is located using $(ROOT_PACKAGE)/cmd/$(COMMAND).

There are two techniques in the above implementation that you should pay attention to. First, obtaining the names of all components in the cmd/ directory with:

COMMANDS ?= $(filter-out %.md, $(wildcard ${ROOT_DIR}/cmd/*))
BINS ?= $(foreach cmd,${COMMANDS},$(notdir ${cmd}))

Then, using wildcards and automatic variables to automatically match and build targets like go.build.linux_amd64.iam-authz-server.

As you can see, to write an extensible Makefile, it is not only important to be proficient in using Makefile, but also requires us to think about how to write the Makefile.

Tip 6: Place all output in one directory for easy cleaning and searching #

During the execution of the Makefile, various files will be outputted, such as Go compiled binary files and test coverage data. I suggest that you put these files in a unified directory to facilitate cleaning and searching in the future. Usually, we can put them in a directory named _output, making it easy to clean by simply deleting the _output folder, for example:

.PHONY: go.clean
go.clean:
  @echo "===========> Cleaning all build output"
  @-rm -vrf $(OUTPUT_DIR)

Here, note that you should use -rm instead of rm to prevent an error when executing make go.clean without the _output directory.

Tip 7: Use hierarchical naming for targets #

By using hierarchical naming, for example, tools.verify.swagger, we can achieve target grouping management. There are many benefits to doing this. First, when there are a large number of targets in the Makefile, grouping can help us better manage these targets. Second, grouping is also convenient for understanding, as the group name can quickly identify the functional category of the target. Finally, this approach greatly reduces the probability of target name conflicts. For example, the Makefile of the IAM project uses the following naming convention extensively.

.PHONY: gen.run
gen.run: gen.clean gen.errcode gen.docgo

.PHONY: gen.errcode
gen.errcode: gen.errcode.code gen.errcode.doc

.PHONY: gen.errcode.code
gen.errcode.code: tools.verify.codegen
    ...
.PHONY: gen.errcode.doc
gen.errcode.doc: tools.verify.codegen
    ...

Tip 8: Splitting Targets #

Another practical tip is to split targets properly. For example, we can split the installation of tools into two targets: verifying if the tools are already installed and installing the tools. By doing this, we can achieve greater flexibility in our Makefile. For example, we can selectively execute one of the operations based on our needs or execute both operations together.

Here’s an example:

gen.errcode.code: tools.verify.codegen

tools.verify.%:
  @if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi  

.PHONY: install.codegen
install.codegen:
  @$(GO) install ${ROOT_DIR}/tools/codegen/codegen.go

In the above Makefile, gen.errcode.code depends on tools.verify.codegen. tools.verify.codegen checks if the codegen command exists and if it doesn’t, it calls install.codegen to install the codegen tool.

If our Makefile design was:

gen.errcode.code: install.codegen

Then every time we execute gen.errcode.code, it would require reinstalling the codegen command. This operation is unnecessary and would slow down the execution of make gen.errcode.code.

Tip 9: Setting OPTIONS #

When writing a Makefile, we also need to control some variable functionalities through OPTIONS. To help you understand, let’s take the IAM project’s Makefile as an example.

Suppose we need to control whether detailed information is printed when executing the Makefile through an option called V. This can be achieved with the following steps.

First, define USAGE_OPTIONS in the /Makefile file. Defining USAGE_OPTIONS allows developers to be aware of this OPTION when executing make help and set it as needed.

define USAGE_OPTIONS    
                         
Options:
  ...
  BINS         The binaries to build. Default is all of cmd.
               ...
  V            Set to 1 enable verbose build. Default is 0.    
endef    
export USAGE_OPTIONS

Then, in the scripts/make-rules/common.mk file, we choose different behaviors depending on whether the V option is set:

ifndef V    
MAKEFLAGS += --no-print-directory    
endif

Alternatively, we can use the following method to use V:

ifeq ($(origin V), undefined)                                
MAKEFLAGS += --no-print-directory              
endif

In the above examples, I introduced the V OPTION, which allows us to perform different operations in the Makefile based on whether V is defined. There’s another type of OPTION that we directly use in the Makefile, such as BINS. We can use the following approach for such an OPTION:

BINS ?= $(foreach cmd,${COMMANDS},$(notdir ${cmd}))
...
go.build: go.build.verify $(addprefix go.build., $(addprefix $(PLATFORM)., $(BINS)))

This means that the value of the BINS variable is determined by checking whether it has been assigned a value using ?=, and if it hasn’t, it assigns the value after the equal sign. Then, we can use it in the Makefile rules.

Tip 10: Defining Environment Variables #

We can define environment variables in the Makefile, for example:

GO := go                                          
GO_SUPPORTED_VERSIONS ?= 1.13|1.14|1.15|1.16|1.17    
GO_LDFLAGS += -X $(VERSION_PACKAGE).GitVersion=$(VERSION) \    
  -X $(VERSION_PACKAGE).GitCommit=$(GIT_COMMIT) \       
  -X $(VERSION_PACKAGE).GitTreeState=$(GIT_TREE_STATE) \                          
  -X $(VERSION_PACKAGE).BuildDate=$(shell date -u +'%Y-%m-%dT%H:%M:%SZ')    
ifneq ($(DLV),)                                                                                                                              
  GO_BUILD_FLAGS += -gcflags "all=-N -l"    
  LDFLAGS = ""      
endif                                                                                   
GO_BUILD_FLAGS += -tags=jsoniter -ldflags "$(GO_LDFLAGS)" 
...
FIND := find . ! -path './third_party/*' ! -path './vendor/*'    
XARGS := xargs --no-run-if-empty

These environment variables serve the same purpose as macro definitions in programming: by modifying them in one place, we can make many locations reflect the changes simultaneously, avoiding repetitive work.

Typically, we define variables like GO, GO_BUILD_FLAGS, FIND as environment variables.

Tip 11: Calling Yourself #

While writing a Makefile, you may encounter a situation where an A-Target command needs to perform an operation B-Action, which you have already implemented as a pseudo target B-Target. To achieve maximum code reusability, it’s best to execute B-Target within the command of A-Target. Here’s how to do it:

tools.verify.%:
  @if ! which $* &>/dev/null; then $(MAKE) tools.install.$*; fi

Here, we are calling the pseudo target tools.install.$* using $(MAKE). Note that by default, Makefile outputs the following information when switching directories:

$ make tools.install.codegen
===========> Installing codegen
make[1]: Entering directory `/home/colin/workspace/golang/src/github.com/marmotedu/iam'
make[1]: Leaving directory `/home/colin/workspace/golang/src/github.com/marmotedu/iam'

If you find the information like Entering directory annoying, you can disable the printing of these messages by setting MAKEFLAGS += --no-print-directory.

Summary #

If you want to efficiently manage a project, using Makefile is currently the best practice. We can write a high-quality Makefile by following these methods.

First, you need to be proficient in the syntax of Makefile. I suggest you focus on the following syntax: Makefile rule syntax, phony targets, variable assignment, special variables, automatic variables.

Next, we need to plan ahead for the functionalities the Makefile should achieve. A large Go project usually requires the following functionalities: code generation commands, formatting commands, static code analysis, testing commands, building commands, Docker image packaging commands, deployment commands, cleaning commands, etc.

Then, we also need to design a reasonable Makefile structure through functionalities classification, file separation, scripting complex commands, etc.

Finally, we need to master some Makefile writing techniques, such as utilizing wildcards, automatic variables, and functions; writing extensible Makefiles; using hierarchical naming conventions, and so on. These techniques can further ensure that we write a high-quality Makefile.

Exercise #

Go through the Makefile implementation of the IAM project and see how the IAM project installs all the features with one command make tools.install, and how it specifies the installation of the xxx tool with make tools.install.xxx.
When you write a Makefile, what writing techniques have you used? Feel free to share your experience or any pitfalls you have encountered.

Looking forward to seeing your thoughts and answers in the comments section. Feel free to discuss any issues related to Makefile. See you in the next lesson!