02 Triggers How to Build Event Sources and Function Computation Throughput

02 Triggers How to build event sources and function computation throughput #

Hello, I’m Jingyuan.

In the previous lesson, I introduced you to the lifecycle of Function Compute through the development and invocation process of a function. I explained the execution flow and basic concepts of Function Compute at different stages from both the user and platform perspectives.

If you have been paying attention, you may have noticed that cloud functions uploaded to the Function Compute platform generally need to be executed through a trigger. If you have not previously been exposed to event-driven models and Serverless-related knowledge, the concept of triggers may be unfamiliar to you.

After completing today’s lesson, I believe you will not only understand the concept and principles of triggers but also have a certain understanding of trigger-related knowledge points, such as events and invocations. Finally, I will guide you in implementing a custom trigger.

Introduction to Triggers #

In the previous lesson, we learned that functions hosted on a platform generally need an event to trigger them to run. But before understanding triggers, let’s first understand events.

What are Events? #

To put it simply, an event is an action or occurrence that takes place during the runtime of a system. Function computation provides an event-driven computing model. In relation to this concept, there are three things we need to understand.

Firstly, the definition and intention of events by the CNCF. CloudEvents aims to provide interoperability across services, platforms, and systems by providing a common format to describe event data. When developing a function computing platform, we can design it more based on this specification. When selecting an open-source framework or developing business code for a function computing platform, interoperability can also be considered as a reference factor for technology selection.

Secondly, the extent to which domestic cloud providers adhere to event specifications. Currently, major third-party cloud providers hope to promote and implement the CloudEvents specification. However, each platform has some unique attributes and parameters included in their event specification definition to varying degrees.

If you develop a cloud function on one of these cloud products, you generally need to do some simple adaptation to migrate. Fortunately, some cloud providers currently have the ability to support script migration and adaptation, and you can contact them through support tickets, greatly reducing migration costs.

Thirdly, which Serverless-related products define specifications by the CNCF. In function computing, we deal with single function event triggers, events triggered by multiple simple functions through asynchronous calls, and event interactions orchestrated through workflows in complex scenarios.

The CNCF Serverless working group has defined corresponding format specifications or primitives for both functions and workflows. Currently, many traditional enterprises and cloud providers are trying to adhere to this standard, especially in complex scenarios involving interactions between multiple functions. When selecting, you can pay attention to whether it is compatible with this standard, making it easier to migrate functions and orchestration rules across different platforms.

What are Triggers? #

After understanding the triggering condition of “events,” let’s look at the “trigger” action before the function starts running. We usually refer to the combination of this relationship that connects upstream and downstream services driven by events as triggers, which is one of the main ways to trigger function execution.

As we know, function computing consists of cloud functions and triggers. For function computing, triggers describe a set of relationships and rules, including the event source, target function, and trigger conditions as the three core elements.

Among these, the event source is the event producer, and the target function is the event handler. When the trigger conditions are met, the function computing engine receives a notification and schedules the corresponding target function for execution.

The metadata of triggers can be persistently stored by the service provider or jointly held by the function hosting platform and the service provider. Triggers also have different classifications based on different integration and invocation methods.

Trigger Types #

Based on the integration principles of triggers, they are generally classified into one-way integration triggers, two-way integration triggers, and proxy integration triggers. Their differences lie in where the event source and event rules are stored and triggered from.

When designing triggers, we mainly consider the upstream and downstream relationship between the event source and function computing and the convenience of operations.

For example, the object triggers seen in cloud providers can be created and configured on both the object storage platform and the function computing platform. This is known as a two-way integration trigger. As for message listening and triggering scenarios, since they mostly involve computing processing of asynchronous messages and the on-demand execution characteristics of function computing, we usually design them as one-way integration triggers, where you can directly create and configure triggers on the function computing platform.

Based on the invocation method, triggers can also be classified into synchronous triggers and asynchronous triggers.

Synchronous invocation: The triggered event is processed by function computing, and the result is returned directly. For example, HTTP triggers and CDN triggers provided by public cloud providers are mostly synchronous triggers.

Asynchronous invocation: After the event is triggered and requested by function computing, the function computing service returns a successful receipt status code, and function computing ensures that the event is reliably processed. For example, scheduled triggers and message queue triggers are all asynchronous triggers.

When using triggers, you can prioritize your own needs and find the most suitable trigger type from different classifications.

By now, I believe your understanding of triggers is not just based on a superficial outline from the previous lesson. Next, let’s experience a simple trigger and lay the foundation for implementing triggers hands-on.

Hands-on Experience #

Let’s start by experiencing Baidu Intelligent Cloud’s Function Compute platform. Assuming we have already created a function named test_geekbang, the next step is to create a trigger.

As shown in the picture, we have created a BOS trigger (BOS is Baidu Intelligent Cloud’s Object Storage Service) and set the target object to a Bucket (bos:/cfc-sns) under BOS. The event types that can trigger the function execution include file uploads and appends.

Once the trigger is created, we can upload a file to BOS to test it. Since we have not set any additional filter rules, uploading any file will trigger the execution of the test_geekbang function.

The overall operation is not difficult, and you can also refer to Baidu Intelligent Cloud’s Function Compute (CFC) tutorial to experience other specific settings. You will find that triggers make the connection between function computation and external services very fast and efficient.

By now, you are probably eager to know how this connection is implemented. Next, let’s get our hands dirty and implement a trigger to understand the underlying principles.

Implementation #

We cannot see the detailed implementation of the object storage triggers of cloud vendors, so it is possible to use an open-source storage service with similar and comprehensive functionality as the base for implementing triggers.

Here, I have chosen to use MinIO, an open-source object storage service similar to Baidu Intelligent Cloud BOS, for demonstration. MinIO is implemented in Golang, and it has client libraries available for various programming languages such as Golang, Java, Python, JavaScript, etc. If you have some free time, you can also try to reproduce this experiment using different programming languages.

Let’s assume we need to fulfill the following functional scenario:

Trigger the execution of function computing by listening to MinIO events (object creation, access, and deletion in MinIO storage). For example, when a file is stored in a MinIO bucket, the monitoring service detects this event and triggers the execution of a predefined business logic by querying and determining the existence of a set of relationships and rules.

So, for this functional requirement, can you break it down into an implementation flow? Take your time to think before answering.

I have roughly divided the process into 5 steps:

The user binds the relevant cloud functions to custom triggers on the function computing platform and sets the event source identifier.
MinIO performs an action, let’s say uploading a file, which associates this action with a unique path under a specific bucket, creating a triggering event.
The event listener (Event Monitor Server) runs as a daemon process and captures this event.
The listener queries and retrieves the corresponding triggers from a MySQL database based on the unique identifier formed by the “action + bucket + path.”
The listener triggers the retrieved triggers one by one, triggering remote function calls on the cloud functions.

Next, let’s define this trigger based on this scenario and flow. The overall implementation requires four steps: determining the event types, defining the event protocol, agreeing on the metadata storage conventions, and binding the events.

Determining Event Types #

Since MinIO already implements an API for event monitoring, we can directly select the three types of events we need from the current event types supported by the MinIO API: s3:ObjectCreated:, s3:ObjectRemoved:, and s3:ObjectAccessed:*, which satisfy the requirements of CRUD operations on an object and can be listened to.

Among them, the first event type satisfies the need to create an object in different ways; the second event type supports deleting specific objects from a storage bucket; and the third event type is used to listen for object access events.

Once the event types are selected, we can use the pre-packaged MinIO client available on GitHub to simplify our development. By introducing mino-go, we can quickly implement the listener for an event type:

func (m *EventMonitor) work() error {
    miClient := m.miClient
    for miInfo := range miClient.ListenNotification(context.Background(), "", "", []string{
        "s3:ObjectCreated:*",
        "s3:ObjectAccessed:*",
        "s3:ObjectRemoved:*",
    }) {
        // TODO: Implement the business logic
    }
}

Defining the Event Protocol #

Next, we need to define the event protocol to transmit trigger requests between MinIO and the cloud functions. Here, we adopt HTTP+JSON. You can refer to the following format for event definition:

{
    "eventId": "Event ID",
    "source": "Event source, in this case, represents MiniIO",
    "specversion": "Protocol version",
    "type": "Event type",
    "timestamp": "Request timestamp",
    "target": "Function unique identifier",
    "data": {
        // Data to be processed
    }
}

}

However, it is not enough to define the protocol for passing requests. In order to identify a trusted event source, the function computing platform usually requires authorization and authentication after a trigger is created to ensure the security of service requests. Next, let’s take a look at the interface design. In this scenario, we need to create four functions to meet the needs of trigger console operations, namely trigger creation, modification, deletion, and query. You can also build the following interface protocol using REST.

func OperateMinioTriggers(request *restful.Request, response *restful.Response)

In the above function, the request parameter includes the action parameter to identify the CRUD operation.

Here you need to note that if a function is deleted, all its related features and triggers should be removed completely. This is also a common oversight in the initial design.

Metadata storage convention #

According to common operation habits, triggers can be implemented on both the MinIO and function computing service sides. Event binding rules can also be created when configuring buckets or when creating a function to listen for request events.

The creation process for these two bindings, except for the differences in UI and platform parameter validation during creation, are quite similar. In order to reduce the difficulty of the initial attempt, we choose to integrate on the MinIO service side.

MinIO, as the event source service, notifies the execution of the trigger function. Therefore, the associated relationship information only needs to be stored on the MinIO service side, while the authentication and rules are stored on the function computing platform side. Specifically:

Relationship creation: Determines the specific trigger relationship, such as the specific bucket and trigger conditions.
Rule creation: Ensures that the trigger source is triggered according to the established rules, such as identifying MinIO actions and trigger forms.

Based on the above description, you can design the solution. Here, I also provide a simple data structure for the object to be stored. You can convert it into a database table for storage. Usually, multiple relationships can share one rule. You can create two tables accordingly: tb_rule to store rules (usually stored on the function computing platform side) and tb_relation to store relationships based on the type of the trigger, stored on the event trigger source side.

Rule:

type Rule struct {
    ID        uint
    Sid       string    // Rule unique ID
    Resource  string    // Function identification
    Source    string    // Trigger source
    ……                    // You can add fields based on business complexity, such as creation time, user ID, and other information.
}

Relation:

type Relation struct{
    RelationId string      // Relationship ID
    RuleId     string      // ID of the rule associated with the trigger
    Source     string      // Trigger source, can be specific to the bucketName
    Target     string      // Unique identifier of the function
    Condition  string      // Trigger condition
    ……                       // You can add fields based on business complexity, such as trigger custom parameters.
}

Event binding #

Based on the above agreed design solution, let’s complete the event binding process from the MinIO service side. It requires three steps.

Step 1: Request the MinIO interface defined above to create a custom trigger configuration, for example, to listen for an action of storing an object in a bucket.

Step 2: After receiving the request, the MinIO service side (such as the API-Server) stores the relationship in the database and notifies the function computing platform synchronously to create the corresponding rule.

Step 3: After receiving the request from the MinIO service side, the function computing platform checks whether the same rule already exists in the current database. If it does, it directly returns a successful response. If it doesn’t, it records the rule for verification during triggering and finally returns a successful notification to the MinIO service side.

With these steps, we have completed the binding process of the event relationship with MinIO as the event source and the function computing platform as the processing side.

Trigger Invocation Process #

After defining and binding a trigger, we can take uploading a file as an example to have an overview of the event triggering process in MinIO:

User performs an operation to upload a file on MinIO.
MinIO’s event monitoring service (EventMonitorServer) captures the triggered event.
EventMonitorServer retrieves the trigger relationship that matches the event from the relationship table (tb_relation) in the database.
Based on the relationship, it initiates a request to the function computing service.
The function computing service on the platform receives the request, performs authentication and rule verification, executes the corresponding function for processing, and finally returns the result based on the agreed asynchronous or synchronous manner.

At this point, we have defined a trigger for object storage in MinIO and described the interaction process of the invocation time based on the temporal relationship.

As mentioned before, MinIO has already provided relevant event monitoring APIs. In this case, you only need to develop a simple service, such as the event monitoring service EventMonitorServer, to listen for changes in these API interfaces. You can further practice and demonstrate according to this method.

If you need to deploy to a production environment, you also need to add additional capabilities such as caching and retry mechanism based on the requirements of performance and stability to ensure the SLA of the production environment.

Summary #

Today, I introduced you to the link between event sources and function computing - triggers, as well as the knowledge related to events and triggers. At the same time, we also got hands-on experience with cloud storage triggers provided by cloud vendors. Finally, we built a custom trigger based on MinIO to delve into the execution principle of triggers.

Among all these, the most important part is to use MinIO as the core foundation to build a custom trigger. I hope you can remember three key pieces of information about implementing triggers.

First, we chose MinIO as the purpose of this hands-on implementation in order to better understand the implementation principles related to cloud vendors’ object storage. When you encounter similar storage events triggering function execution in the future, you can also use the methods mentioned in this section to deal with them.

Second, the implementation process of triggers roughly includes four core steps: event type determination, event protocol definition, metadata storage agreement, and event binding.

Finally, the MinIO client introduced today, the listening of the three major event types of MinIO, and the given code examples can be learned and used immediately. You can also try to quickly get started with practical exercises to implement a trigger.

I hope that through the above introduction, you will have a deeper understanding of triggers and related events and invocation principles. When using or customizing triggers in the future, you will be able to handle them more accurately.

Discussion Question #

Alright, we’ve come to the end of this lesson. Finally, I’d like to leave you with a question to ponder.

In your work scenario, what other scenarios have you come across that can be implemented as triggers? During the transformation process, what technical considerations should we be mindful of?

Feel free to share your thoughts and answers in the comments section. Let’s engage in discussion and exchange ideas. Thank you for reading, and please feel free to share this lesson with more friends for further discussions and progress.