04 Norm Design How to Standardize Project Development Amidst Chaos

04 Norm Design How to Standardize Project Development amidst Chaos #

Hello, I am Kong Lingfei. Today, let’s talk about the standards that are needed in application development.

Rules are important, both in life and in software development. An application is usually developed by multiple people, each with their own development habits and approaches. Without a unified set of standards, many problems can arise, such as:

  • Inconsistent code styles: Different code styles exist in the code repository, making it painful to read or modify other people’s code, and the entire codebase appears messy.
  • Disorganized directories: The same functionality is placed in different directories, or you have no idea what a certain directory is meant for. It can be difficult to determine where to place new code. These issues greatly reduce the maintainability of the code.
  • Inconsistent APIs: The public API interfaces lack uniformity. For example, while modifying the user interface, you use /v1/users/colin, but for modifying the secret interface, you use /v1/secret?name=secret0. This inconsistency makes it difficult to understand and remember.
  • Inconsistent error codes: Error codes are directly exposed to users and are primarily used to identify error types for troubleshooting. Inconsistent error codes make it difficult to distinguish between error types or assign different error codes to the same error, increasing the difficulty of understanding.

Therefore, at the design stage and before coding, we need good standards to guide developers in ensuring that we are developing " one application “. Good standards not only improve software quality but also increase development efficiency, reduce maintenance costs, and even reduce the number of bugs. They can also make your development experience smooth and effortless. Therefore, it is necessary to spend some time discussing and establishing standards with your team members before coding.

So, where do we need to establish standards, and how should these standards be established?

What places need to establish specifications? #

A Go project involves many aspects, so there will be multiple specifications, and similar specifications may vary due to team differences. Therefore, in this course, I will only introduce some commonly used specifications in development. For easy memory, I divide them into non-coding specifications and coding specifications based on whether they are related to code:

  • Non-coding specifications mainly include open source specifications, documentation specifications, version specifications, Commit specifications, and release specifications.
  • Coding specifications mainly include directory specifications, code specifications, interface specifications, log specifications, and error code specifications.

For easy memory, I have organized these specifications into the following diagram:

In this lesson, let’s start with open source specifications, documentation specifications, and version specifications. Since there are many Commit specifications, we will cover them in the next lesson. As for other specifications, they will be introduced in the subsequent content. For example, log specifications will be covered together with log package design because they are closely related.

Open Source Guidelines #

First of all, let’s introduce the concept of open source guidelines.

In fact, there is no official open source guideline in the industry, and it is rare for people to mention it in actual development. So why do we need to know about open source guidelines?

There are primarily two reasons: First, open source projects require higher standards for code quality, code formatting, documentation, and other aspects compared to non-open source projects. Following the requirements of open source projects in the development of our own projects can drive the improvement in project quality. Second, some large companies require their teams to open source their projects in order to avoid duplicating efforts. Therefore, driving the development of Go projects according to open source standards in advance can save us a lot of trouble when open sourcing our code in the future.

Every open source project needs an open source license, which defines your rights and responsibilities when using open source software. In other words, it determines what you can and cannot do. Therefore, the first guideline of open source standards is to choose a suitable open source license. So, what are the available open source licenses and how to choose one? I will explain in detail next.

Overview of Open Source Licenses #

First of all, it should be noted that only open source projects need open source licenses. If your project is not intended to be open source, you do not need an open source license. However, it is always good to understand the concept, as it may come in handy in the future.

There are hundreds of open source licenses in the industry, and each license has different requirements. Some licenses have stricter conditions while others are more permissive. You don’t need to remember all of them, just knowing the six commonly used open source licenses is sufficient. Those licenses are GPL, MPL, LGPL, Apache, BSD, and MIT. For more information on these licenses, you can refer to the Introduction to Open Source Licenses (in Chinese).

Now, how do you choose the right open source license for yourself? You can refer to the following diagram created by Ukrainian programmer Paul Bagwell:

In the diagram, licenses on the right side are more permissive than those on the left side. When making a choice, you can follow the selection criteria in the diamond-shaped box from top to bottom. In order to use the source code provided by the IAM project without any burden, I chose the most permissive license, which is MIT.

Furthermore, because Apache is a license that is friendly to commercial applications, users can modify the code to meet their needs when necessary, and release/sell it as open source or commercial products. Therefore, open source projects of large companies usually adopt the Apache 2.0 open source license.

What Are the Characteristics of Open Source Guidelines? #

When participating in open source projects or following the requirements of open source projects to standardize code, what aspects of guidelines should we pay attention to?

In my opinion, any guideline that can make a project better should be considered as part of open source guidelines.

In addition to following the coding and non-coding guidelines mentioned earlier, the code of an open source project should also adhere to the following guidelines.

First, an open source project should have a high unit test coverage. This ensures that third-party developers can easily perform detailed unit testing on the entire project after completing their code, and also guarantees the quality of code submissions.

Second, the entire codebase and commit history should not contain internal IP addresses, internal domain names, passwords, keys, or any sensitive information. Otherwise, it may lead to the exposure of sensitive information and could pose security risks to our internal business.

Third, when our open source project receives pull requests, issues, or comments from other developers, we should handle them promptly. This not only ensures that the project keeps getting updated but also stimulates the enthusiasm of other developers to contribute code.

Fourth, a good open source project should continuously update features and fix bugs. For open source projects that have been completed and no longer maintained, they should be archived in a timely manner and explained in the project description.

In my opinion, the above points are the more important aspects of open source guidelines. If you want to know the detailed contents of the open source guidelines, you can refer to this material that I have put on GitHub.

Lastly, I would like to remind you of two things: First, if possible, you can promote and operate open source projects to make more people aware of them, use them, and contribute code. For example, you can publish articles on platforms such as Juejin and Jianshu, or create QQ and WeChat discussion groups. These are all good ways to promote your project. Second, if you are proficient in English and have the time, it is best to have documentation in both Chinese and English, with English being prioritized, so that developers from around the world can understand, use, and participate in your project.

Document Specification #

In my work, I have found that many developers focus heavily on code output, but not on document output. They think that even without software documentation, it doesn’t matter much and doesn’t affect software delivery. I want to say that this view is wrong! Because documentation is an important part of software delivery, projects without documentation are difficult to understand, deploy, and use.

Therefore, writing documentation is an essential part of development work. So what documents need to be written for a project, and how should they be written? I believe the three most needed types of documents in a project are README documents, project documents, and API interface documents.

Below, we will discuss their writing specifications one by one.

README Specification #

The README document is the facade of the project. It is the first document that developers read when they learn about the project and is placed in the root directory of the project. Because it is mainly used to introduce the project’s functionality, installation, deployment, and usage, it can be standardized.

Below, let’s directly look at the content of the README specification through a README template:

# Project Name

<!-- Write a short statement describing the project -->

## Features

<!-- Describe the core features of the project -->

## Software Architecture (optional)

<!-- You can describe the project's architecture -->

## Quick Start

### Dependency Check

<!-- Describe the project's dependencies, such as dependencies on packages, tools, or any other dependencies -->

### Build

<!-- Describe how to build the project -->

### Run

<!-- Describe how to run the project -->

## User Guide

<!-- Describe how to use the project -->

## How to Contribute

<!-- Tell other developers how to contribute source code to the project -->

## Community (optional)

<!-- If necessary, introduce some community-related content -->

## About the Author

<!-- Write the project author here -->

## Who's Using (optional)

<!-- List other influential projects that use this project, as a form of advertising -->

## License

<!-- Link to the project's open-source license here -->

For more specific examples, you can refer to the README.md file of the IAM system.

Here, there is an online README generation tool that you can also refer to: readme.so.

Project Documentation Specification #

Project documentation includes all content that needs to be documented, and they are usually centrally located in the /docs directory. When we create documentation for a team project, we usually plan and create directories in advance to store different documents. Therefore, before starting development of a Go project, we also need to establish a software documentation specification. Good documentation specifications have two advantages: readability and quick document positioning.

Different projects have different documentation needs, so when formulating documentation specifications, you can consider including two types of documents.

  • Development documentation: Used to explain the project’s development process, such as how to set up the development environment, build binary files, testing, deployment, etc.
  • User documentation: Software usage documentation, usually targeting software users, with content that can be added as needed. For example, it can include API documentation, SDK documentation, installation documentation, feature introduction documentation, best practices, operation guides, FAQs, etc.

To facilitate global developers and users, development documentation and user documentation can be pre-planned in both English and Chinese versions. To deepen your understanding, let’s take a look at the directory structure of the practical project:

docs
├── devel                            # Development documentation, both English and Chinese versions can be planned in advance
│   ├── en-US/                       # English version, file structure can be organized as needed
│   └── zh-CN                        # Chinese version, file structure can be organized as needed
│       └── development.md           # Development manual, explaining how to compile, build, and run the project
├── guide                            # User documentation
│   ├── en-US/                       # English version, file structure can be organized as needed
│   └── zh-CN                        # Chinese version, file structure can be organized as needed
│       ├── api/                     # API documentation
│       ├── best-practice            # Best practices, storing important practice articles
│       │   └── authorization.md
│       ├── faq                      # Frequently asked questions
│       │   ├── iam-apiserver
│       │   └── installation
│       ├── installation             # Installation documentation
│       │   └── installation.md
│       ├── introduction/            # Product introduction documentation
│       ├── operation-guide          # Operation guide, which can be further divided into subdirectories based on RESTful resources to store operation manuals for system core/all functions
│       │   ├── policy.md
│       │   ├── secret.md
│       │   └── user.md
│       ├── quickstart               # Quick start
│       │   └── quickstart.md
│       ├── README.md                # User documentation entry file
│       └── sdk                      # SDK documentation
│           └── golang.md
└── images                           # Directory for storing images
    └── 部署架构v1.png

API Documentation Specification #

API documentation, also known as API interface documentation, is usually written by backend developers to describe the API interfaces provided by components and how to call these API interfaces.

In the early stages of a project, API documentation can decouple frontend and backend development, allowing them to develop in parallel: the frontend only needs to implement the calling logic based on the API documentation, and the backend only needs to provide the functionality according to the API documentation.

After both frontend and backend development is completed, we can proceed directly to integration testing, which improves development efficiency. In the later stages of the project, the API documentation can be provided to users, reducing the entry barrier for component usage and minimizing communication costs. Obviously, a well-structured and comprehensive interface document with a fixed format is very important. So how do we write an interface document and what are the specifications?

There are four ways to write an interface document, including writing a Word document, using tools, generating through comments, and writing a Markdown document. The specific implementation methods are shown in the table below:

Among them, the two most commonly used methods are generating through comments and writing in Markdown format. In this column, I choose to write in Markdown format for the following reasons:

  • Compared to generating through comments, writing the interface document in Markdown format can express richer content and format, without adding a large number of comments in the code.
  • Compared to Word documents, Markdown documents take up less space, can be published together with code repositories, and facilitate the distribution and retrieval of API documents.
  • Compared to online API document writing tools, Markdown documents eliminate the dependence on third-party platforms and limitations of the network.

What specifications should an API interface document follow? In fact, a standardized API interface document usually needs to include a complete API interface introduction document, API interface change history document, general instructions, data structure instructions, error code description, and API interface usage document. The API interface usage document needs to include interface description, request method, request parameters, output parameters, and request examples.

Of course, depending on the requirements of different projects, the API interface document may have different formats and content. I will use the API interface document specification used in the practical project of this course as an example to explain to you.

The interface document is divided into several Markdown files and is placed in the directory docs/guide/zh-CN/api:

  • README.md: API interface introduction document, which categorically introduces the API interfaces supported by IAM and provides links to relevant API interface documents for developers to view.
  • CHANGELOG.md: API interface document change history, which facilitates historical traceability and allows callers to decide whether to update functions and versions.
  • generic.md: Used to explain common request parameters, response parameters, authentication methods, and request methods.
  • struct.md: Lists the data structures used in the interface document. These data structures may be used by multiple API interfaces and will be referenced in the user.md, secret.md, and policy.md files.
  • user.md, secret.md, policy.md: API interface documents, interfaces of the same REST resource are stored in one file, named after the REST resource name.
  • error_code.md: Error code description, generated automatically by a program.

Here, taking the user.md interface document as an example, let me explain how the interface document is written. The user.md file records the interfaces related to users, and each interface is arranged in order, including the following 5 parts:

  • Interface Description: Describes what functionality the interface implements.
  • Request Method: The request method of the interface, in the format of HTTP method request path, for example, POST /v1/users. In the General Instructions section under Request Method, the protocol and address for the interface request will be explained.
  • Input Parameters: The input fields of the interface, which are divided into Header parameters, Query parameters, Body parameters, and Path parameters. Each field is described using 4 attributes: Parameter Name, Required, Type, and Description. If the parameter has restrictions or default values, it can be indicated in the description.
  • Output Parameters: The return fields of the interface, each field is described using 3 attributes: Parameter Name, Type, and Description.
  • Request Example: A real API interface request and response example.

If you have mastered these contents and want to learn more about the detailed API interface document specification, you can refer to this link.

Versioning Specification #

When developing Go projects, I recommend that you introduce version control for all components. There are two main reasons for this: first, with version numbers, we can clearly identify the version of a component and locate its functionality and code, which makes it easier to troubleshoot. Second, by including version numbers when releasing components, users can know the progress of the project, as well as the differences in functionality between the current version and the previous version.

The mainstream versioning specification in the industry is Semantic Versioning (SemVer), which is also the versioning specification adopted by the IAM system. So, what is Semantic Versioning?

What is Semantic Versioning (SemVer)? #

Semantic Versioning (SemVer) is a meaningful and standardized version numbering scheme drafted by GitHub. It defines the format, incrementing rules, and meanings of different version numbers.

Under this specification, the version number and how it increases contain information about the underlying code and modifications between adjacent versions. The format of a semantic version is Major.Minor.Patch (X.Y.Z), where X, Y, and Z are non-negative integers and leading zeros are prohibited.

The version number can increment according to the following rules:

  • Major version (MAJOR): When making incompatible API changes.
  • Minor version (MINOR): When making backward-compatible functional additions and modifications. There is an unwritten convention to note that even numbers indicate stable versions, while odd numbers indicate development versions.
  • Patch version (PATCH): When making backward-compatible bug fixes.

For example, v1.2.3 is a semantic version number, and the specific meaning of each number in the version number is shown in the following figure:

You might have also come across a version number like this: v1.2.3-alpha. This is adding a pre-release version and version metadata as extensions to the Major.Minor.Patch format, following the format X.Y.Z[-pre-release][+version metadata], as shown in the figure below:

Now let’s look at what the pre-release version and version metadata mean.

A pre-release version implies that the version is unstable and may have compatibility issues. Its format is X.Y.Z-[a series of dot-separated identifiers]. Here are a few examples:

1.0.0-alpha
1.0.0-alpha.1
1.0.0-0.3.7
1.0.0-x.7.z.92

Version metadata, generally generated by the compiler during the compilation process, is only defined in format and not manually controlled. Here are some examples of version metadata:

1.0.0-alpha+001
1.0.0+20130313144700
1.0.0-beta+exp.sha.5114f85

Note that both the pre-release version and the version metadata can only consist of letters and numbers, and there must not be any spaces.

Semantic Versioning Control Specification #

There are many Semantic Versioning control specifications, and I will introduce a few important ones here. If you need more detailed specifications, you can refer to the content in this link (note: the link is in Chinese).

  • After a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications must be released as a new version.
  • Software in the 0.y.z version range is for initial development and everything is subject to change. No public API should be considered stable. Version 1.0.0 is defined as the first stable version, and all subsequent version number updates are based on this version.
  • Patch version Z (x.y.Z | x > 0) MUST be incremented when backwards-compatible bug fixes are introduced. This is referred to as a patch.
  • Minor version Y (x.Y.z | x > 0) MUST be incremented when backwards-compatible new features are introduced or marked as deprecated. It can also be incremented for significant improvements that may or may not break backward compatibility. Any increment of the minor version MUST reset the patch version to 0.
  • Major version X (X.y.z | X > 0) MUST be incremented when backward-incompatible changes are introduced to the public API. It may also include minor and patch level changes. Any increment of the major version MUST reset both the minor and patch versions to 0.

How to Determine Version Numbers? #

After discussing so much, how should we actually determine version numbers?

Here are a few suggestions:

First, when developing, I recommend using 0.1.0 as the first development version and incrementing the minor version with each subsequent release.

Second, when our version is a stable version and is being published for the first time, we can set the version number to 1.0.0.

Third, when strictly following the Angular commit message convention, we can determine the version number as follows:

  • For a fix-type commit, increment the patch version by 1.
  • For a feat-type commit, increment the minor version by 1.
  • For a commit with a BREAKING CHANGE, increment the major version by 1.

Summary #

A good set of conventions is like a set of “rules” for project development. It can ensure the maintainability and readability of the entire project, and reduce the number of bugs.

The design of project conventions mainly includes coding conventions and non-coding conventions. Today, we have learned about open source conventions, documentation conventions, and versioning conventions. Now let’s review the key points together.

  1. It is best to standardize new projects according to open source standards to drive them to become high-quality projects.
  2. Before development, it is best to standardize the document directory in advance and choose an appropriate way to write API documentation. In the hands-on project of this course, I use the Markdown format, and I recommend you to use this method as well.
  3. Projects should follow versioning conventions. The currently mainstream versioning convention in the industry is semantic versioning, which is also the convention I recommend.

The detailed version of the project convention examples used today can be found here for easy access: Open Source Convention, README Convention, API Interface Documentation Convention.

Exercise #

  1. Besides the non-coding specifications we introduced today, what other specifications have you used in your development?
  2. Try using the API documentation specifications introduced in this lecture to write an API interface for your current project.

Looking forward to seeing your thoughts and answers in the comments. I also welcome discussions about specification design. See you in the next lecture!