30 What Should a Good Project Automation Look Like

30 What should a good project automation look like? #

Hello, I’m Zheng Ye.

In this module on automation, I’m going to start with the daily work of programmers. When introducing “Iteration 0”, I mentioned that build scripts are an important part of project preparation, but I didn’t specifically discuss what a build script should look like.

Today, let’s take a typical Java REST service as an example to see what the most basic build script should do. I’m using Spring Boot, the most common framework in Java technology, as the base framework, and for the build tool, I have chosen Gradle.

The first question that many Java programmers might have is, why use Gradle instead of Maven? After all, Maven is the most classic build tool in the Java community. The answer is that Maven is not flexible enough.

Think about how many times you have used Maven to implement specific requirements. I guess most people’s answer would be none. With the rise of continuous integration and continuous delivery, the customization capability of build scripts becomes more and more important, and Maven falls short in this aspect.

In fact, as early as 2012, Maven was put on Hold in the ThoughtWorks Technology Radar, which means, if you can avoid using it, then don’t use it.

To accompany this lecture, I have written a demo and put it on GitHub. Its functionality is very simple:

  • By sending a POST request to /users, it achieves user registration;
  • Accessing /users allows you to view the registered users.

If it’s convenient, it would be best for you to clone this project for reference. Here, I mainly focus on explaining what automation should look like. If you want to learn about the specific implementation, you can refer to the code in the demo.

Alright, let’s get started!

Basic Preparation #

Start by cloning this project from Github.

git clone  https://github.com/dreamhead/geektime-zero.git

Next, navigate to the directory where the project is located.

cd geektime-zero

Once you’re ready, we can further explore this project.

Typically, when we want to understand a project, we open it in an IDE. Here, I recommend using IntelliJ IDEA, which is currently the best Java IDE in the industry. Since the community edition of IntelliJ IDEA has become free, it has become my preferred choice to recommend to others.

I understand that development tools, apart from programming languages, can be a topic that easily sparks “religious wars”. If you prefer another IDE, feel free to use your favorite one, but make sure to adjust the configuration in the build script.

How do you open this project? Let’s start by generating an IDEA project using the Gradle command.

./gradlew idea

This command will generate an .ipr file, which is the IDEA project file. You can open it with IntelliJ IDEA.

Here are two points to note.

First, we are using gradlew here, which is a wrapper for Gradle commands. It will automatically download the Gradle version required to build this project. The key point is that by using this command, we lock down the Gradle version to avoid situations where “it works for you but not for me” due to differences in the build scripts.

Second, the IDE project is generated by Gradle. Many people tend to intuitively open the project directly in the IDE. However, in some team projects, there may be multiple build files, and you won’t know which one to open without asking someone. This is very unfriendly for newcomers to the project.

The approach we use here is similar to the Gradle wrapper, and it avoids issues that can arise from having different IDE versions installed.

Additionally, since the IDE project is generated, if there are any new program library dependencies added to the project, you just need to rerun the previous command. Modern IDEs have good automatic loading capabilities, and when they detect changes to the project file, they will reload accordingly.

Alright, now you can open the project in your IDE, and we can further explore it.

Getting Started with the Project #

First, let’s get to know a bit about Gradle configuration files, as they are the key to automating our project.

  • build.gradle is the Gradle configuration file. Because Gradle is written in Groovy, build.gradle is essentially a Groovy script, and the configuration within it is Groovy code. This is the foundation that allows Gradle to be flexible and customized.

  • settings.gradle is another Gradle configuration file used to support multi-module projects. While each module within a project can have its own build.gradle, the entire project has only one settings.gradle.

In Gradle, many capabilities are provided in the form of plugins. For example, the previous step of generating an IDEA project was just one line of code in the configuration file:

apply plugin: 'idea'

Therefore, if you are a loyal fan of another IDE, you can replace this line with your preferred IDE.

(Note: This project uses Lombok to simplify the code. In order to compile and run the code in IntelliJ IDEA, you can install the Lombok plugin and enable annotation processing in “Build, Execution, Deployment” -> “Compiler” -> “Annotation Processors”.)

Now that we have the basic knowledge, let’s learn about code organization.

First, let’s talk about modularization. Unless your codebase is very small, modularization is almost inevitable. A proper way to divide code is based on different business functionalities. For example, put user-related content in one module, transaction and order information in another module, and logistics information in yet another module.

If you plan to build microservices in the future, each module can become an independent service.

In our project, I have exemplarily divided it into two modules:

  • zero-identity is the module for user information.
  • zero-bootstrap is the module that packages multiple modules into a deployable application.

Information about these two modules is configured in settings.gradle:

include 'zero-bootstrap'
include 'zero-identity'

Next is the directory structure. The specific way to organize code has become a convention in the Java world.

Put your source code under src/main/java, configuration files under src/main/resources, and test code under src/test/java. This embodies the principle of “Convention over Configuration”. If the tools you’re using don’t have a convention, you’ll need to establish your own and require others to follow it.

Check #

In the automation process, a fundamental task is to perform checks. In our project, this is accomplished through a check task.

./gradlew check

What does this check do? It depends on the configuration. In this project, we have applied the Java plugin, which compiles Java files, checks if the code compiles correctly, runs tests, checks if the code functions properly, and more. But I require more.

In the “Iteration 0” section, I mentioned that basic code style checks should be included in the build script. Here, I use CheckStyle to perform this task. By default, you just need to apply the Checkstyle plugin.

apply plugin: 'checkstyle'

In this project, I have made some customizations, such as specifying certain files to be excluded from the checks.

style.excludePackages = [
]
    
style.excludeClasses = [
]

Code coverage should also be included in the build script. Here, I use JaCoCo. Similarly, you just need to apply the JaCoCo plugin by default.

apply plugin: 'jacoco'

I have made some customizations here as well, such as generating HTML reports for the results and excluding certain files from the checks.

coverage.excludePackages = [
]
    
coverage.excludeClasses = [
]

The most unique feature here is that I have set the test coverage to be fixed at 1.0, which means 100% test coverage. This is the default configuration for new projects and is my requirement for the team.

If a new project can pass all of these checks, the rate of bugs should be lower. Of course, you can add more checks according to your needs.

Database Migration #

In “Iteration 0”, I mentioned database migration, which is about how to modify the database. In the example project, I chose the database migration tool - Flyway.

plugins {
    id "org.flywaydb.flyway" version "5.2.4"
}

First, some basic configurations need to be done to ensure that the database can be connected to. (Note: If you want to use the configuration here directly, you can create a user named “zero” with the password “geektime” on your local MySQL database, and then create a database called “zero_test”.)

flyway {
    url = 'jdbc:mysql://localhost:3306/zero_test?useUnicode=true&characterEncoding=utf-8&useSSL=false'
    user = 'zero'
    password = 'geektime'
    locations = ["filesystem:$rootDir/gradle/config/migration"]
}

So how do you modify the database? First, add a database migration file. For example, in the sample project, I created a migration file (gradle/config/migration/V2019.02.15.07.43__Create_user_table.sql), and created a User table in it.

CREATE TABLE zero_users(
    id bigint(20) not null AUTO_INCREMENT,
    name varchar(100) not null unique,
    password varchar(100) not null,
    primary key(id)
);

For the migration file version, I chose to name it with a timestamp. Another option is to use a version number, such as V1, V2.

The advantage of using the timestamp naming method is that different people can develop at the same time, and the probability of naming conflicts is very low. However, using the version number naming method may have a higher probability of naming conflicts.

After adding the database migration file, just execute the following command:

./gradlew flywayMigrate

This way, the modifications to the database will be applied to the database, and you can open the database to take a look.

Building the Application #

With the basic checks in place and the database ready, it’s time to build our application.

The first step is to generate the build artifacts, which can be done with a single command:

./gradlew build

This command will generate an executable JAR file in zero-bootstrap/build/libs, which is our final build artifact. Additionally, the build task depends on the check task, which means that the code will be checked before the build process.

In the past, Java programs were simply packaged and deployed to an application server. Thanks to advancements in infrastructure, we can now skip the deployment step and execute the JAR file directly. This can be done with the following command:

java -jar zero-bootstrap/build/libs/zero-bootstrap-*-boot.jar

During development, there’s no need to generate the JAR file every time. Instead, we can run the application directly using Gradle:

./gradlew bootRun

However, the most common way to run the application is to find the Bootstrap entry class in your IDE and run it directly.

Now that the program is running, let’s test it. You can use tools like Postman or Curl to send the following content as a POST request to http://localhost:8080/users:

{
    "username": "foo",
    "password": "bar"
}

After that, you can access http://localhost:8080/users in your browser, and you should see the user we just registered.

Summary #

Let’s summarize today’s content. Today, we demonstrated a basic project automation process through a specific example, which includes:

  • Generating an IDE project;
  • Compiling;
  • Packaging;
  • Running tests;
  • Code style checking;
  • Test coverage;
  • Database migration;
  • Running the application.

But is this all there is to automation? Obviously not. What I have presented here is just a basic example. In fact, almost every repetitive or tedious task should be automated. We should not waste time and energy on tasks that machines can do well for us.

Today’s infrastructure has made our automation work much easier than before. For example, executable JAR packages have simplified deployment to application servers compared to the past. Gradle has also greatly reduced the difficulty of customizing build scripts.

The project automation mentioned here is also the foundation of continuous integration. The commands executed on a continuous integration service should be written in our build scripts, such as:

./gradlew build

In 2011, I published an article on InfoQ titled “Software Development Foundation,” discussing what a project’s build script should look like. Although some of the tools used in the article are no longer popular today, some of the basic ideas are still relevant. If you’re interested, you can take a look.

If there’s only one thing you can remember from today’s content, please remember this: Automate your workflow.

Finally, I would like to ask you to share which processes you have automated in your daily development work. Feel free to write your thoughts in the comments.

Thank you for reading, and if you found this article helpful, please feel free to share it with your friends.