21 Custom Metrics How to Implement Custom Metrics and Actuator Endpoints

21 Custom Metrics - How to Implement Custom Metrics and Actuator Endpoints #

In Lesson 20, we introduced the Spring Boot Actuator component to fulfill the system monitoring functionality of Spring Boot applications, and we focused on how to extend the implementation of common Info and Health monitoring endpoints.

In this lesson, we will continue the discussion on how to extend Actuator endpoints, but with more emphasis on metrics-related content. Additionally, we will provide methods for creating custom Actuator implementations to meet the needs of application scenarios where the default endpoints are not sufficient.

Metrics in Actuator #

Metrics are an important dimension for system monitoring. In Spring Boot 2.X, the Actuator component primarily uses the built-in Micrometer library to collect and analyze metrics.

Micrometer Metrics Library #

Micrometer is a library for measuring monitoring metrics which provides a universal API for collecting performance data on the Java platform. In an application, we only need to use the common API provided by Micrometer to collect metrics.

Let’s start by briefly introducing several core concepts included in Micrometer.

Firstly, we need to introduce the Meter interface, which represents the performance metric data to be collected. The definition of Meter is as follows:

public interface Meter extends AutoCloseable {

    // Unique identifier of the Meter, which is a combination of name and tags
    
    Id getId();

    // A set of measurements
    
    Iterable<Measurement> measure();

    // Enumeration values of Meter's type

    enum Type {
        COUNTER,
        GAUGE,
        LONG_TASK_TIMER,
        TIMER,
        DISTRIBUTION_SUMMARY,
        OTHER
    }
}

From the above code, we can see that Meter contains an Id object, which is used to define the name and tags of the Meter. Looking at the enumeration values of Type, we can easily see all the types of meters included in Micrometer.

Next, let’s explain two concepts.

Meter’s name: For a Meter, each Meter has its own name, and when created, a series of tags can be specified. Meter’s tags: The purpose of tags is to allow the monitoring system to classify and filter metrics using these tags.

In daily development, the commonly used meter types mainly include Counter, Gauge, and Timer.

  • Counter: The purpose of this meter is the same as its name - an ever-increasing accumulator. We can implement the accumulation logic through its increment method.
  • Gauge: Unlike Counter, the value measured by Gauge is not necessarily cumulative. We can specify a value through its gauge method.
  • Timer: This meter is relatively simple - it is used to record the duration of an event.

Now that we have clarified the commonly used meters and their use cases, how can we create these meters?

In Micrometer, we provide a MeterRegistry, which is a meter registry responsible for creating and maintaining various meters. The creation method of MeterRegistry is shown in the following code:

public abstract class MeterRegistry implements AutoCloseable {

    protected abstract <T> Gauge newGauge(Meter.Id id, @Nullable T obj, ToDoubleFunction<T> valueFunction);

    protected abstract Counter newCounter(Meter.Id id);

    protected abstract Timer newTimer(Meter.Id id, DistributionStatisticConfig distributionStatisticConfig, PauseDetector pauseDetector);

    ...
}

The above code is only one way to create a Meter. From it, we can see that MeterRegistry provides corresponding creation methods for different meters.

Another way to create a Meter is by using the specific builder method of a particular meter. Taking Counter as an example, its definition includes a builder method and a register method, as shown in the following code:

public interface Counter extends Meter {
    
    static Builder builder(String name) {
        return new Builder(name);
    }
    
    default void increment() {
        increment(1.0);
    }
    
    void increment(double amount);
    
    double count();
    
    @Override
    default Iterable<Measurement> measure() {
        return Collections.singletonList(new Measurement(this::count, Statistic.COUNT));
    }
    
    ...
    
    public Counter register(MeterRegistry registry) {
        return registry.counter(new Meter.Id(name, tags, baseUnit, description, Type.COUNTER));
    }
}

Note that the register method at the end is used to register the current Counter to the MeterRegistry, so we need to create a Counter. Typically, we would use the following code to create it:

Counter counter = Counter.builder("counter1")
        .tag("tag1", "value1")
        .register(registry);

Now that we have an understanding of the basic concepts of the Micrometer framework, let’s return to Spring Boot Actuator and take a look at the Metrics endpoint that it provides specifically for managing metrics.

Extending the Metrics Endpoint #

In Spring Boot, it provides a Metrics endpoint for us to implement production-level metric tools. When we access the actuator/metrics endpoint, we will get a series of metrics as shown below:

{
    "names": [
        "jvm.memory.max",
        "jvm.threads.states",
        "jdbc.connections.active",

…

"jvm.gc.memory.promoted",
"jvm.memory.used",
"jvm.gc.max.data.size",
"jdbc.connections.max",
"jdbc.connections.min",
"jvm.memory.committed",
"system.cpu.count",
"logback.events",
"http.server.requests",
"jvm.buffer.memory.used",
"tomcat.sessions.created",
"jvm.threads.daemon",
"system.cpu.usage",
"jvm.gc.memory.allocated",
"hikaricp.connections.idle",
"hikaricp.connections.pending",
"jdbc.connections.idle",
"tomcat.sessions.expired",
"hikaricp.connections",
"jvm.threads.live",
"jvm.threads.peak",
"hikaricp.connections.active",
"hikaricp.connections.creation",
"process.uptime",
"tomcat.sessions.rejected",
"process.cpu.usage",
"jvm.classes.loaded",
"hikaricp.connections.max",
"hikaricp.connections.min",
"jvm.gc.pause",
"jvm.classes.unloaded",
"tomcat.sessions.active.current",
"tomcat.sessions.alive.max",
"jvm.gc.live.data.size",
"hikaricp.connections.usage",
"hikaricp.connections.timeout",
"jvm.buffer.count",
"jvm.buffer.total.capacity",
"tomcat.sessions.active.max",
"hikaricp.connections.acquire",
"process.start.time"
]
}

The above code contains various metrics including total system memory, free memory, number of processors, system uptime, and heap information, as well as database connection information after introducing JDBC and HikariCP data source components. If we want to obtain detailed information about a particular metric, we can add the corresponding metric name after the actuator/metrics endpoint.

For example, if we want to obtain the current memory usage, we can access the actuator/metrics/jvm.memory.used endpoint, as shown in the following code:

{
   "name":"jvm.memory.used",
   "description":"The amount of used memory",
   "baseUnit":"bytes",
   "measurements":[
      {
         "statistic":"VALUE",
         "value":115520544
      }
   ],
   "availableTags":[
      {
         "tag":"area",
         "values":[
            "heap",
            "nonheap"
         ]
      },
      {
         "tag":"id",
         "values":[
            "Compressed Class Space",
            "PS Survivor Space",
            "PS Old Gen",
            "Metaspace",
            "PS Eden Space",
            "Code Cache"
         ]
      }
   ]
}

As mentioned earlier when introducing Micrometer, the Metrics system includes support for Counter and Gauge measurement metrics. By injecting Counter or Gauge into our business code, we can record the desired measurement metrics. Counter provides an increment() method for exposure, while Gauge provides a value() method.

Next, we will use Counter as an example to introduce how to embed custom Metrics metrics in business code, as shown in the following code:

@Component
public class CounterService {
    public CounterService() {
        Metrics.addRegistry(new SimpleMeterRegistry());
    }
}


    
public void counter(String name, String... tags) {

    Counter counter = Metrics.counter(name, tags);

    counter.increment();

}

In this code snippet, we built a public service CounterService and exposed a Counter method for business systems to use. Of course, you can also implement similar utility classes to encapsulate various meters.

In addition, Micrometer also provides a MeterRegistry utility class for creating metrics. Therefore, we highly recommend using MeterRegistry to simplify the process of creating custom metrics.

Using MeterRegistry #

Returning to the example of SpringCSS, let’s go to CustomerTicketService in the customer-service.

For example, we want to count every time a customer ticket is created and use it as a runtime metric of the system. The implementation for this effect is shown in the following code:

@Service
public class CustomerTicketService {
    @Autowired
    private MeterRegistry meterRegistry;

    public CustomerTicket generateCustomerTicket(Long accountId, String orderNumber) {
        CustomerTicket customerTicket = new CustomerTicket();
        ...
        meterRegistry.summary("customerTickets.generated.count").record(1);
        return customerTicket;
    }   
}

In the generateCustomerTicket method above, we automatically add a counter every time a CustomerTicket is created using MeterRegistry.

Moreover, MeterRegistry provides some utility methods to create custom metrics. These utility methods include not only the regular methods for creating specific meters such as counter, gauge, and timer, but also the summary method mentioned in the above code. The summary method returns a DistributionSummary object, which is defined as follows:

public interface DistributionSummary extends Meter, HistogramSupport {
    static Builder builder(String name) {
        return new Builder(name);
    }

    // Record data
    void record(double amount);

    // Record the number of operations executed
    long count();

    // Record the total amount of data
    double totalAmount();

    // Record the average value of the data
    default double mean() {
        return count() == 0 ? 0 : totalAmount() / count();
    }

    // Record the maximum value of the data
    double max();
    ...
}

Because the purpose of DistributionSummary is to record a series of events and process them, the line of code meterRegistry.summary("customertickets.generated.count").record(1) added in CustomerTicketService is equivalent to recording each call to the generateCustomerTicket method.

Now, if we visit the actuator/metrics/customertickets.generated.count endpoint, we can see the increasing metrics information as the service is called continuously:

{
    "name":"customertickets.generated.count",
    "measurements":[
        {
            "statistic":"Count",
            "value":1
        },
        {
            "statistic":"Total",
            "value":19
        }
    ] 
}

It is apparent that using MeterRegistry makes it easier to use custom metrics. Here, you can also try different functionalities of MeterRegistry according to your business requirements.

Next, let’s look at a relatively more complex usage. In the customer-service, we also want to have a metric that records the total number of newly added CustomerTickets. The example code for this is as follows:

@Component
public class CustomerTicketMetrics extends AbstractRepositoryEventListener<CustomerTicket> {
    private MeterRegistry meterRegistry;

    public CustomerTicketMetrics(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    @Override
    protected void onAfterCreate(CustomerTicket customerTicket) {
        meterRegistry.counter("customerTicket.created.count").increment();   
    }
}

Firstly, here we use the Counter method of MeterRegistry to initialize a counter, and then call its increment method to increase the measurement count (we are already familiar with this part).

Note that here we also introduce an AbstractRepositoryEventListener abstract class, which can monitor the events triggered by Repository layer operations in Spring Data, such as BeforeCreateEvent and AfterCreateEvent events before and after entity creation, and BeforeSaveEvent and AfterSaveEvent events before and after entity saving.

For these events, AbstractRepositoryEventListener can capture and call the corresponding callback functions. The implementation of part of the AbstractRepositoryEventListener class is shown in the following code:

public abstract class AbstractRepositoryEventListener<T> implements ApplicationListener<RepositoryEvent> {

 

    public final void onApplicationEvent(RepositoryEvent event) {

        

        …

        Class<?> srcType = event.getSource().getClass();

        

        if (event instanceof BeforeSaveEvent) {

            onBeforeSave((T) event.getSource());

        } else if (event instanceof BeforeCreateEvent) {

            onBeforeCreate((T) event.getSource());

        } else if (event instanceof AfterCreateEvent) {

            onAfterCreate((T) event.getSource());

        } else if (event instanceof AfterSaveEvent) {

            onAfterSave((T) event.getSource());

              }

        …

	}

}

In this code snippet, we can see that AbstractRepositoryEventListener directly implements the ApplicationListener listener interface in the Spring container, and triggers the callback function based on the event type passed in the onApplicationEvent method.

Taking the requirement scenario in the example as an example, we can perform measurement operations after creating the Account entity. That is, the code for the measurement operation can be placed in the onfterCreate callback function, as shown in the example code.

Now, we perform the operation of generating a customer work order and access the corresponding Actuator endpoint, and we can also see that the measurement data is continuously increasing.

Custom Actuator Endpoint #

In the daily development process, extending existing endpoints may not always meet business requirements, and customizing Spring Boot Actuator monitoring endpoints is a more flexible approach.

Suppose we need to provide a monitoring endpoint to obtain the current system’s user information and computer name. This can be achieved by implementing an independent MySystemEndPoint, as shown in the following code:

@Configuration
@Endpoint(id = "mysystem", enableByDefault=true)
public class MySystemEndpoint { 

    @ReadOperation
    public Map<String, Object> getMySystemInfo() {
        Map<String,Object> result= new HashMap<>();
        Map<String, String> map = System.getenv();
        result.put("username",map.get("USERNAME"));
        result.put("computername",map.get("COMPUTERNAME"));
        return result;
    }
}

In this code snippet, we can see that MySystemEndpoint mainly obtains the required monitoring information through system environment variables.

Note that we introduce a new annotation @Endpoint here, which is defined as shown in the following code:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Endpoint {
    //Endpoint ID
    String id() default "";
    //Enable by default flag
    boolean enableByDefault() default true;
}

The @Endpoint annotation in this code snippet is mainly used to set the endpoint ID and the enable flag by default. In the example, we specify the ID as “mysystem” and the enableByDefault flag as true.

In fact, in Actuator, there is also a batch of endpoint annotations similar to @Endpoint. Among them, endpoints annotated by @Endpoint can be accessed by JMX and web to access the application, endpoints annotated by @JmxEndpoint can only be accessed through JMX, and endpoints annotated by @WebEndpoint can only be accessed through the web.

In the example code, we also see an @ReadOperation annotation, which is applied to a method to indicate a read operation. In Actuator, in addition to providing the @ReadOperation annotation, it also provides the @WriteOperation and @DeleteOperation annotations, corresponding to write and delete operations.

Now, by accessing http://localhost:8080/actuator/mysystem, we can get the following monitoring information.

{
    "computername":"LAPTOP-EQB59J5P",
    "username":"user"
}

Sometimes, in order to obtain specific metric information, we need to pass parameters to an endpoint. Actuator specifically provides a @Selector annotation to indicate input parameters. The code snippet is as follows:

@Configuration
@Endpoint(id = "account", enableByDefault = true)
public class AccountEndpoint {

    @Autowired
    private AccountRepository accountRepository;

    @ReadOperation
    public Map<String, Object> getMySystemInfo(@Selector String arg0) {
        Map<String, Object> result = new HashMap<>();
        result.put(accountName, accountRepository.findAccountByAccountName(arg0));
        return result;
    }
}

The logic of this code snippet is very simple, which is to get user account information based on the passed accountName.

Please note that using the @Selector annotation, we can trigger measurement operations using endpoint addresses like http://localhost:8080/actuator/account/account1.

Summary and Preview #

Measurement is the core means of observing the runtime state of an application. In this lecture, we introduced the Micrometer measurement library introduced in Spring Boot, as well as various measurement components provided in the library. At the same time, based on the MeterRegistry core utility class in Micrometer, we completed the implementation process of embedding measurement indicators in business systems. Finally, we briefly introduced the development method of how to customize an Actuator endpoint.