24 Advanced Practical V How to transform from a Serverless engine to a Serverless platform #

Hello, I am Jingyuan.

In the previous two lessons, I provided a detailed introduction to the characteristics and selection methods of privatizing the core serverless engine, and deployed it based on the popular Knative engine. I believe that by now, you are already proficient in the serverless foundation.

However, studying only the core engine is far from enough to construct a serverless platform. There is still a certain distance to go in building a complete platform. In today’s lesson, I will continue to focus on the theme of “privatization,” and explain to you in detail how to build a complete platform based on a serverless engine foundation.

I hope that through this lesson, you will have the ability to construct a platform and be more adept at choosing the “most suitable” ToB provider to collaborate on a serverless platform.

Overview: What’s the difference between an engine and a platform? #

Let’s first clarify the focus of “engine” and “platform” in the context of Serverless.

The goal of an engine or framework is mainly to provide a system or architecture with specific functionality. For example, Knative, OpenFaaS, Fission, and other open-source Serverless engines provide a system for running FaaS-based products.

When researching and deploying an engine, you don’t need any business or architectural experience. By going through the official website and following the usage documentation, you can quickly set up a Serverless system for testing and experimentation. However, it is not suitable for direct use in enterprise production lines, as it lacks capabilities such as permission control, metadata management, and BaaS integration.

On the other hand, a platform needs to consider the practical problems encountered in a production environment. From initial design to full-scale production use, it requires overall planning, iterative upgrades, and a process from internal testing to public testing to full openness. Throughout the service process, considerations must be made for SLA commitments and more. This requires designers to possess rich domain expertise, architectural design capabilities, and long-term planning and iteration thinking.

Public cloud and private cloud platforms, although they may differ in appearance—for example, public clouds need to adapt to cloud service providers’ container services, IAM, cloud product integration, multi-tenancy, etc., while private clouds need to adapt to their own container infrastructure, middleware, and various BaaS services—share the same principles: Private cloud Serverless platforms mainly enjoy control and security by shifting the maintenance responsibilities from cloud vendors to enterprise operation and maintenance teams. The only difference is that private clouds may be less concerned about billing issues for business users.

So, what aspects do we need to consider when building a Serverless platform?

How to Build a Serverless Platform? #

Based on the relationship between users, platforms, and functions, I have summarized 7 key points from the perspectives of security, ease of use, operability, and scalability.

The overall platform is built on top of a Kubernetes cluster with containers as the foundation. For the core engine, you can choose the serverless engine framework introduced in the previous lesson or develop your own engine. Let’s discuss the key points for each function.

Function Management #

Most open-source engines come with some built-in function management capabilities, such as CRUD operations through CLI or integrated console operations. However, neither the visual interface nor the CLI operations meet the requirements of a production-level platform.

From the perspective of function management, it is necessary to integrate functions’ creation, configuration, deployment and release, testing, deletion, and online/offline functionalities, while also supporting the selection of underlying resource specifications.

From the perspective of function code development, it is necessary to provide development support for multiple programming languages (such as Java, Python, Node.js, Golang, etc.). In contrast to the built-in templates and language support of the function computing engine, the platform needs to emphasize ease of use, such as code package upload and deployment, persistent storage, and integration with commonly used triggers.

In addition, there is one point that is often overlooked, which is the inconsistency of different operating system versions when creating function computing base images, which needs to be considered due to different enterprise operating systems.

Platform Governance #

This part includes user isolation, authentication verification, user roles, and permissions management, which are all closely related to security. Let’s discuss them one by one.

• User isolation: This includes not only the control of function entities, such as namespace, concurrency setting, and persistent storage isolation, but also the isolation of running containers. Regarding containers, you need to consider either adding a security container or allocating based on Node nodes. I have compared these two modes in the Scaling lesson, you can review it to deepen your understanding.
• Authentication verification: It is usually integrated with the enterprise’s public IAM system. However, from the perspective of high availability, you can design two sets of systems: one is the normal access mode of the enterprise IAM system, and the other is the emergency mode of the local system’s IAM.
• User roles and permissions management: The design of most mainstream permission management systems is based on the RBAC model or variations of RBAC, and you can reuse your previous experience in designing microservice systems.

Resource Control #

This functionality can be included in platform governance, but it is discussed separately to emphasize another important feature of the platform: metering and billing. For public and private clouds, the difference lies in the fact that public clouds are oriented towards numerous external customers, and cloud providers focus on revenue, so they not only need metering but also billing, requiring the serverless platform to interface with the enterprise’s billing system. For private clouds, it is mainly used within the enterprise, so metering is more important and billing is not a very urgent function.

The second feature is resource usage restrictions. For example, the usage of memory, CPU, and GPU instances, as well as the maximum usage in a single region, are all points that the platform needs to consider.

Development Toolchain #

A good development tool not only makes development, debugging, and deployment more convenient but also facilitates the migration of old services to new technology platforms, which is the original intention of designing a new platform.

Therefore, in addition to the CLI commands provided by the serverless engine itself, we should also develop a set of CLI command tools that meet our own enterprise needs. For general development, we should have Web IDE tools for fast online development, and IDE plugins for developers who are accustomed to IDEs like VSCode.

High Availability #

Enterprises generally have requirements in terms of server failure, abnormal traffic, synchronous/asynchronous failures, cold start, scaling, and other aspects. We find that apart from cold start and scaling, which are unique features of serverless, other aspects are the same as the knowledge points we have discussed in microservice governance.

In general, these requirements can be addressed through multi-AZ deployment, remote backup, etc. By introducing service governance middleware (such as circuit breakers, registries, etc.), traffic abnormal scheduling and synchronous request failures can be addressed, but asynchronous call failures need to be handled in combination with middleware. Recall that we mentioned asynchronous invocation before, the platform needs strategies such as delayed loading and retry to ensure the reliability of user requests.

Regarding cold start and scaling, you can review them again, and I believe you will have the answers.

So far, the main functions of a serverless platform are almost complete. But to make it truly user-friendly, you also need to have the capabilities of observability, orchestration, asynchronous invocation, layers, templates, single-instance multi-concurrency, and reserved instances. Can’t you find the answers to the implementation and use of these core capabilities from the previous lessons?

Next, based on the previous lesson and with the knowledge points of observability, I will go into detail with you about the steps to design a serverless platform.

Building Observable Capabilities #

You may wonder: why take an observable approach? First, observability introduces open-source components that have low coupling with specific enterprises or businesses, making it easier for you to understand and implement. Secondly, you can apply the theoretical knowledge learned from observability to practice through this platform perspective.

In the Observable course, I mentioned the three major elements in observability: metrics, traces, and logs. However, mastering these three elements alone is far from enough. You can further consider the following five aspects to improve the construction of observability and fill in any gaps:

• Clearly identify the collected data: Whether it is metrics, traces, or logs, you should clearly understand the types of data you need, instead of collecting all data aimlessly. Each type of data should be considered from the perspectives of platform operation and user concerns. For example, metrics should include the number of calls and execution duration that users care about, as well as the resource usage of each instance.
• Component selection: It is best to choose components from mature products. Personally, I recommend using some CNCF-recommended and mature products, such as Prometheus and fluent-bit. In addition, you also need to consider the learning cost of introducing components and the subsequent maintenance work. Some observable components, although feature-rich, may require a significant amount of time to become familiar with and deploy.
• Resource cost: Observable components also consume resources. For example, a powerful log cleansing component like Logstash may consume several GB of memory for a single instance, and storing observable data on disk requires a large amount of storage space. These need to be evaluated in conjunction with specific resource budgets. If the budget allows, you can fully use services on public clouds to avoid subsequent component maintenance work.
• Adaptation: In industries like finance and telecommunications, to improve system security, the operating system used is often a domestically developed operating system like Kylin, which has its own PaaS infrastructure. Therefore, to adapt to the PaaS infrastructure, it may be necessary to perform secondary development on these observable components. This is also a consideration in the selection phase.
• Scalability: The scalability of observable components should also be considered from the beginning of the design, which requires a long-term estimation of the platform’s usage. For example, for Prometheus, deploying it in a federated cluster can be considered. After all, if there is significant growth in the future, deploying a single instance will encounter bottlenecks.

Before starting the practical implementation, you can first take a look at the overall architecture diagram, which covers the observable solution built around the three main pillars using Knative. Next, we will look at the specific implementation paths separately.

Metrics #

First, clearly identify the metrics data to be collected. In Lecture 12, I proposed specific items from both the platform and user perspectives, including the number of calls, execution duration, and resource usage. Knative Engine already provides partial support for metric collection, including business metrics and module metrics.

Business metrics are provided by queue-proxy, including 5 types of metrics closely related to the business, such as revision_request_count, revision_request_latencies, and revision_app_request_count.

In terms of module metrics, we can collect metrics reported by Activator, Autoscaler, Controller, etc. These include Activator-level concurrent statistics related to overall cold start concurrency and the number of function pods related to current resource usage.

So which component should we use to collect metrics? Here I recommend using Prometheus. There are two reasons for this:

• First, Knative officially recommends Prometheus as the metric collection tool, and the mentioned modules each have their own exporters for reporting, making it very convenient to use.
• Second, Prometheus belongs to the mature metric collection tools in the CNCF observability series, and the community is also very active, which is very helpful for later maintenance work.

After determining Prometheus as the metric collection tool, the recommended visualization tool is Grafana, which is almost always used together by default and is also recommended on the Knative official website.

Tracing #

Tracing provides information on the time consumption of key processes. For a Serverless platform, the two most critical stages are cold start and the overall execution time of the business code.

Returning to Knative, cold start time includes the process from receiving the request from the Activator to the completion of preparing the user’s container and environment, while the execution time of the business code is the entire runtime of the function. In the collection process, we often calculate the time consumption of the tracing by subtracting the duration of spans. For example, the cold start time can be obtained by subtracting the duration of the Activator service from the duration of the User Container’s execution completion.

In addition, in addition to the Activator and User Pod nodes, we also need to consider support for user-defined tracing, allowing users to mark and report on the critical path within the function using the SDK.

For tracing, I still use Opentelemetry and Jaeger, which are recommended by Knative. Opentelemetry is used for collecting and reporting tracing information, while Jaeger is used for visualizing tracing information.

As you can see from the diagram above, I have deployed the Otel-agent as a DaemonSet (you can also deploy it through Sidecar or SDK injection). It is used to report the tracing information of the function. Then, the Otel-collector collects these information, and finally, we can access the data in the collector through Jaeger-Query to view it. This forms a complete closed-loop for tracing information.

Logging #

For the logging part, Knative officially recommends using fluent-bit. This is mainly because fluent-bit is super fast, lightweight, and highly scalable. It is the preferred tool for containerized log collection and processing and can interface with various log storage downstream systems such as ElasticSearch and Kafka.

The overall log processing flow in the diagram uses fluent-bit, with ElasticSearch responsible for storage, and Kibana for visualization.

Well, now let’s combine the Knative we set up in the previous class with the design plan above for hands-on practice.

Observable Functionality Hands-on #

I will guide you through several key steps in setting up the components and provide step-by-step instructions. In particular, I will explain the more complex parts: metrics and links. After learning this, you will be able to apply the same principles to setting up logs.

Metrics #

First, let’s set up the relevant components for collecting metrics. Observable construction based on metrics includes three parts: data reporting, Prometheus data collection, and Grafana data visualization.

As mentioned earlier, the Knative engine itself exposes many metrics. Modules such as Autoscaler and Activator in Serving, as well as the queue-proxy in User Pods, implement data reporting and expose dedicated ports for Prometheus to collect. For example, Activator’s request_concurrency and request_latencies are statistics on request concurrency and latency, which means Activator itself acts as an exporter.

I will use Activator as an example to demonstrate the configuration of Prometheus and Grafana. First, let’s configure the Prometheus scrape rules using the config yaml provided by the Knative official website. The configuration for scraping Activator metrics is as follows:

# Activator pods
- job_name: activator
  scrape_interval: 3s
  scrape_timeout: 3s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  # Scrape only the targets matching the following metadata
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
    action: keep
    regex: knative-serving;activator;metrics-port
  # Rename metadata labels to be reader friendly
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    regex: (.*)
    target_label: namespace
    replacement: $1
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    regex: (.*)
    target_label: pod
    replacement: $1
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    regex: (.*)
    target_label: service
    replacement: $1

In this configuration, scrape_interval and scrape_timeout represent the interval at which data is collected and the timeout for collecting data, respectively. The most important part is from line 9 to line 11, where Prometheus defines how to scrape Activator data.

source_labels represent the original labels of the data, and regex represents the values that match the source_labels labels. They are one-to-one correspondence. Finally, I use action: keep to indicate that the labels specified by the regex field should be retained. For example, the value of the label __meta_kubernetes_pod_label_app needs to be Activator in order to be scraped.

Here, we need to create the configmap in the knative-monitoring namespace. If it does not exist, you need to create it in advance:

kubectl create ns knative-monitoring
kubectl apply -f 100-scrape-config.yaml

Next, create K8s resources such as Service, Deployment, and Role using the 200-prometheus.yaml:

kubectl apply -f 200-prometheus.yaml

After successfully executing the above command, you can see that a Prometheus pod is running in the knative-monitoring namespace:

$ kubectl get pod -n knative-monitoring
NAME                    READY   STATUS    RESTARTS   AGE
prometheus-system-0     1/1     Running   0          4d23h

Then, we can use the Prometheus UI to view the metrics. If your K8s is not deployed locally, you can use port forwarding to map the port locally. Let’s see the port exposed by the Prometheus Service:

$ k get svc -n knative-monitoring
NAME                     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
prometheus-system-np     NodePort   10.0.133.199   <none>        8080:30609/TCP   4d23h

You can see that the exposed port is 8080, and then we use the port-forward command to access it locally:

kubectl -n knative-monitoring port-forward svc/prometheus-system-np 8080

After opening localhost:8080, you can enter the Prometheus homepage. We use the curl command to access the function deployed in the previous section, and search for the activator_request_count metric on the Prometheus homepage. Afterwards, you can see that the metrics reported by Activator have been collected by Prometheus, and the labels of this metric will also include the revision name of the accessed function:

Then we deploy Grafana in the same way. You can refer to the official provided grafana yaml:

kubectl apply -f 100-grafana-dash-knative-efficiency.yaml
kubectl apply -f 100-grafana-dash-knative.yaml
kubectl apply -f 100-grafana.yaml

Note that in the 100-grafana.yaml file, many config maps need to be associated as Grafana’s view configurations. Here, I only used grafana-dashboard-definition-knative-efficiency and grafana-dashboard-definition-knative which are related to Knative, excluding other config maps configurations.

After the configuration is completed, go to knative-monitoring and you can see that there are already two running pods:

kubectl get pod -n knative-monitoring

NAME                       READY   STATUS    RESTARTS   AGE
grafana-cbb657d6-twc2l     1/1     Running   0          4d7h
prometheus-system-0        1/1     Running   0          4d23h

Then, as before, view the service in the same way and use port-forward for local access:

kubectl get svc -n knative-monitoring

NAME                      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
grafana                   NodePort   10.0.63.40      <none>        30802:30870/TCP   4d8h
prometheus-system-np      NodePort   10.0.133.199    <none>        8080:30609/TCP    4d23h

kubectl -n knative-monitoring port-forward svc/grafana 30802

After logging in with the default username, password, and admin, create a dashboard and you can see the collected metrics:

Tracing #

After setting up the components for collecting metrics, we will now start building the components for collecting tracing information, mainly including two parts: Opentelemetry and Jaeger. The detailed process of the entire setup can be found in this article provided by the official. Next, let’s start setting it up.

Step 1: Before deploying the Opentelemetry components, we need to install the Certificate Manager:

# Create the namespace
kubectl create ns cert-manager 
# Deploy the Certificate Manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml

After completion, you can see three running pods in the namespace:

kubectl get pod -n cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-bfcbf79f6-jvwnr               1/1     Running   0          16s
cert-manager-cainjector-65c9c8779f-wsk8v   1/1     Running   0          16s
cert-manager-webhook-848ff6dc66-cmq8t      1/1     Running   0          16s

Step 2: After the certificate installation is completed, we will deploy the Opentelemetry operator. The operator is responsible for managing the Opentelemetry collector, which is the Otel-collector shown in the screenshots of the metrics part:

# Create the namespace
kubectl create ns opentelemetry-operator-system
# Deploy the operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.40.0/opentelemetry-operator.yaml

After successful deployment, you can see the corresponding running pods:

$ kubectl get pod -n opentelemetry-operator-system
NAME                                                         READY   STATUS    RESTARTS   AGE
opentelemetry-operator-controller-manager-596866cc59-7p6cr   2/2     Running   0          19s

Step 3: Deploy the Jaeger Operator to control Jaeger instances.

kubectl create namespace observability &&
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/v1.28.0/deploy/crds/jaegertracing.io_jaegers_crd.yaml &&
kubectl create -n observability \
    -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/v1.28.0/deploy/service_account.yaml \
    -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/v1.28.0/deploy/role.yaml \
    -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/v1.28.0/deploy/role_binding.yaml \
    -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/v1.28.0/deploy/operator.yaml

After deployment, check the pod status:

$ kubectl get pod -n observability
NAME                               READY   STATUS    RESTARTS   AGE
jaeger-operator-7576fbc794-7gr8n   1/1     Running   0          16s

Step 4: Create a Jaeger instance.

kubectl apply -n observability -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
EOF

We check the pod and service status and can see that the above Jaeger type created multiple services, such as simplest-agent, simplest-collector, and simplest-query.

$ kubectl get pod -n observability
NAME                               READY   STATUS    RESTARTS   AGE
jaeger-operator-7576fbc794-7gr8n   1/1     Running   0          11m
simplest-746f54765d-zdb7d          1/1     Running   0          9s

$ k get svc -nobservability | grep simplest
simplest-agent                   ClusterIP  None           <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP      9d
simplest-collector               ClusterIP  10.0.94.222    <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP  9d
simplest-collector-headless      ClusterIP  None           <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP  9d
simplest-query                   ClusterIP  10.0.239.242   <none>        16686/TCP,16685/TCP                    9d

Then, use port-forward to access the simplest-query service, and open localhost:16686. You will see the following interface:

kubectl -n observability port-forward service/simplest-query 16686

Step 5: Deploy the Opentelemetry Collector and Opentelemetry Agent. Both of them are collectors for tracing data, but they have different data sources and directions. The Opentelemetry Agent receives trace data from instances and forwards it to the Opentelemetry Collector, and the Opentelemetry Collector receives trace data sent by the Otel Agent or the application instance itself and forwards it to Jaeger.

According to the previous design, we can configure it according to the Opentelemetry Collector example provided by the Knative website. Pay attention to the jaeger.endpoint field in the exporters section, which needs to be configured as the downstream endpoint, i.e., the Jaeger collector we created earlier. Also, specify the exporter type as Jaeger on the last line.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: observability
spec:
  config: |
    receivers:
      zipkin:
    exporters:
      logging:
      jaeger:
        endpoint: "simplest-collector.observability.svc.cluster.local:14250"
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [zipkin]
          processors: []
          exporters: [logging, jaeger]

Similarly, since the downstream of the Opentelemetry Agent is the Opentelemetry Collector, the otlp.endpoint in the exporters needs to be set to the endpoint of the Opentelemetry Collector created earlier, and the last line also needs to specify the exporter type as otlp (Opentelemetry Protocol).

.....
    exporters:
      otlp:
        endpoint: "otel-collector.observability.svc.cluster.local:4317"
        tls:
          insecure: true
.....          
    service:
      pipelines:
        traces:
          receivers: [zipkin, otlp]
          processors: []
          exporters: [otlp]

After configuring, execute the apply command:

kubectl apply -f opentelemetry-collector.yaml
kubectl apply -f opentelemetry-agent.yaml

After successful deployment, there will be 5 running pods in the observability namespace, including Jaeger Operator, Jaeger Instance, Otel Collector, and two Otel Agents.

☁  opentelemetry  kubectl get pod -n observability
NAME                               READY   STATUS    RESTARTS   AGE
jaeger-operator-7576fbc794-7gr8n   1/1     Running   0          28m
otel-agent-collector-4j9sc         1/1     Running   0          9s
otel-agent-collector-xd52d         1/1     Running   0          9s
otel-collector-6f49dfc856-ntwpp    1/1     Running   0          48s
simplest-746f54765d-zdb7d          1/1     Running   0          16m

Step 6: Configure tracing information for Knative Serving. The Knative Serving’s tracing configuration is stored in the config-tracing ConfigMap in the knative-serving namespace. Here, you need to set the zipkin-endpoint to the endpoint of the Opentelemetry Agent created in the previous step. After that, all tracing information reported by the Knative Serving components will be sent to the Opentelemetry Agent.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-tracing
  namespace: knative-serving
data:
  backend: "zipkin"
  zipkin-endpoint: "http://otel-agent-collector.observability.svc.cluster.local:9411/api/v2/spans"
  debug: "true"

Finally, to verify, open localhost:16886 in a local browser and access the configured function. Then, you will see the activator-service keyword in the interface:

After clicking on one of the trace entries, you can see a complete chain of events:

Summary #

Finally, let’s summarize today’s content. Through this lesson, I believe you have learned that there is still some distance to master a serverless engine and build a serverless platform. We need to combine underlying technologies with rich domain experience, architectural design capabilities, and long-term planning thinking to iteratively build it.

Today, I introduced to you the concept of building a platform based on an engine through 7 core functional points: function management, platform control, resource control, development toolchain, high availability, observability, and extensibility. It is like adding monochrome sketches to an outline drawing, giving it a clear image and space.

Is this enough? In the actual running process, we also need to consider: how can new and old businesses generate value based on this platform? How to easily build applications? How to meet the constantly extending scene requirements?

Therefore, we need to add colors on top of the sketch to make the painting more aesthetically pleasing. These “colors” include convenient migration tools and framework integration, professional training and practical manuals, scheduling of underlying heterogeneous resources, and the ability to customize runtime, etc.

You can also imagine if there is still room for improvement in the existing serverless platform, and where is its future road? What does the ultimate form look like? For example, whether there will be a unified cross-platform toolchain in the industry, and the extension of serverless will no longer be limited to the manifestation of a product platform but become a design paradigm. I will discuss these questions with you in the conclusion.

Homework #

Alright, the class has come to an end. Finally, I have prepared some homework for you.

Based on the solution mentioned above, try to complete the part related to log collection. Think about which open-source components you need to use and how to integrate and display them.

Thank you for reading, and feel free to share this class with more friends for further discussion and exchange of ideas.