50 Service Orchestration Deployment Practices Based on Helm

50 Service Orchestration Deployment Practices Based on Helm #

Hello, I’m Kong Lingfei.

In the previous lesson, I introduced the basics of Helm and guided you through deploying a simple application. After mastering the fundamentals of Helm, today we will move on to a practical exercise and deploy an IAM (Identity and Access Management) application using Helm.

To deploy the IAM application with Helm, we first need to create an IAM Chart package. Then, we can use this Chart package to deploy the IAM application with just one command. In actual development, we often need to deploy applications in different environments, so I will also demonstrate how to deploy the IAM application in multiple environments.

Creating IAM Chart Package #

Before deploying the IAM application, we need to create an IAM Chart package.

Assuming the IAM project source code is located at ${IAM_ROOT}, navigate to the ${IAM_ROOT}/deployments directory and create the Chart package. The creation process consists of four steps, which I will explain in detail below.

Step 1, create a template Chart.

A Chart is a collection organized in a file directory, where the directory name represents the Chart name (without version information). You can refer to this Chart Development Guide, which explains how to develop your own Chart.

Alternatively, you can use the helm create command to quickly create a template Chart and make modifications based on that Chart to create your own Chart. The creation command is as follows:

$ helm create iam

Executing helm create iam will generate an iam directory in the current directory, which contains the Chart files. The Chart directory structure and files are as follows:

$ tree -FC iam/
├── charts/                            # [Optional]: This directory contains other Charts that the current Chart depends on
├── Chart.yaml                         # YAML file that describes the basic information of the Chart, including the name and version
├── templates/                         # [Optional]: Directory that contains deployment file templates. The template uses values from values.yaml and values provided by Tiller
│   ├── deployment.yaml                # Kubernetes Deployment object
│   ├── _helpers.tpl                   # Template used to modify Kubernetes object configurations
│   ├── hpa.yaml                       # Kubernetes HPA object
│   ├── ingress.yaml                   # Kubernetes Ingress object
│   ├── NOTES.txt                      # [Optional]: Provides instructions for using the Chart
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests/                         # Defines some testing resources
│       └── test-connection.yaml
└── values.yaml                        # Default configuration file for the Chart

In the above directory, there are two important files:

  • Chart.yaml file
  • templates directory

Now, let’s take a closer look at the Chart.yaml file.

Chart.yaml is used to describe the basic information of the Chart, including the name, version, etc. The contents are as follows:

apiVersion: Chart API version (required)
name: Chart name (required)
version: Semantic version (required)
kubeVersion: Semantic version of compatible Kubernetes (optional)
description: One-sentence description of the project (optional)
type: Chart type (optional)
keywords:
  - A group of keywords about the project (optional)
home: URL of the project's home page (optional)
sources:
  - List of URLs for the project source code (optional)
dependencies: # List of necessary Chart dependencies (optional)
  - name: Chart name (nginx)
    version: Chart version ("1.2.3")
    repository: (optional) Repository URL ("https://example.com/charts") or alias ("@repo-name")
    condition: (optional) YAML path that evaluates to a boolean value, used to enable/disable the Chart (e.g. subchart1.enabled)
    tags: # (optional)
      - Tag used to enable/disable a group of Charts at once
    import-values: # (optional)
      - Mapping of ImportValue to save source values to imported parent keys. Each item can be a string or a pair of child/parent list items
    alias: (optional) Alias used within the Chart. Useful when adding the same Chart multiple times
maintainers: # (optional)
  - name: Maintainer's name (required for each maintainer)
    email: Maintainer's email (optional for each maintainer)
    url: Maintainer's URL (optional for each maintainer)
icon: URL of an SVG or PNG image used as an icon (optional)
appVersion: Included application version (optional). It doesn't need to be semantic, but it is recommended to use quotes
deprecated: Whether the Chart is deprecated (optional, boolean value)
annotations:
  example: List of annotations entered by name (optional).

Let’s also take a look at the templates directory.

The templates directory contains the YAML-formatted resource definition templates for various Kubernetes resources in the application. For example:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: {{ .Values.pump.name }}
  name: {{ .Values.pump.name }}
spec:
  ports:
  - name: http
    protocol: TCP
    {{- toYaml .Values.pump.service.http| nindent 4 }}
  selector:
    matchLabels:
      app: {{ .Values.pump.name }}

The above snippet is a template for a Kubernetes Service resource. app: {{ .Values.pump.name }} sessionAffinity: None type: {{ .Values.serviceType }}

The value {{ .Values.pump.name }} will be replaced with the value of pump.name from the deployments/iam/values.yaml file. The template syntax above extends the syntax of the Go text/template package:

# This template definition removes all trailing empty lines in the test template
{{- define "test"}}
Template content
{{- end}}

# This template definition removes the first empty line at the beginning of the test template
{{- template "test" }}

The following is the syntax for indenting YAML files:

# This template definition removes all leading and trailing empty lines in the test template
{{- define "test" -}}
Template content
{{- end -}}

# You can add 4 spaces at the beginning of each line in the test template for YAML alignment
{{ include "test" | indent 4}}

Finally, there are three points to note here:

  • Chart names must be lowercase letters and numbers. Words can be separated by a hyphen -, but uppercase letters, underscores, and dots (.) cannot be used.
  • Use SemVer 2 to represent version numbers whenever possible.
  • YAML files should be indented with two spaces (never use the tab key).

Step 2: Edit the Chart file under the iam directory.

We can build our own Chart package based on the template Chart generated by helm create. Here, we added YAML formatted Kubernetes resource file templates for creating the iam-apiserver, iam-authz-server, iam-pump, and iamctl services:

$ ls -1 iam/templates/*.yaml
iam/templates/hpa.yaml                                   # Kubernetes HPA template file
iam/templates/iam-apiserver-deployment.yaml              # iam-apiserver service deployment template file
iam/templates/iam-apiserver-service.yaml                 # iam-apiserver service service template file
iam/templates/iam-authz-server-deployment.yaml           # iam-authz-server service deployment template file
iam/templates/iam-authz-server-service.yaml              # iam-authz-server service service template file
iam/templates/iamctl-deployment.yaml                     # iamctl service deployment template file
iam/templates/iam-pump-deployment.yaml                   # iam-pump service deployment template file
iam/templates/iam-pump-service.yaml                      # iam-pump service service template file

You can view the specific content of the templates in the deployments/iam/templates/ directory.

When editing the Chart, we can use helm lint to verify the format is correct. For example:

$ helm lint iam
==> Linting iam

1 chart(s) linted, 0 chart(s) failed

If 0 chart(s) failed is displayed, it means the current Iam Chart package has passed the validation.

Step 3: Modify the configuration file of the Chart and add custom configurations.

You can edit the deployments/iam/values.yaml file to customize your own configuration. You can refer to deployments/iam/values.yaml for specific configurations.

When modifying the values.yaml file, you can refer to the following best practices:

  • Variable names should start with lowercase letters, and words should be separated by camel case. For example, chickenNoodleSoup.
  • Add quotes to all string values.
  • To avoid integer conversion issues, it is better to store integers as strings and use {{ int $value }} in the template to convert the string back to an integer.
  • Each property defined in values.yaml should be documented. The documentation string should start with the attribute it describes and provide at least one sentence of description. For example:
# serverHost is the host name for the webserver
serverHost: example
# serverPort is the HTTP listener port for the webserver
serverPort: 9191

Here, it is important to note that all Helm built-in variables start with an uppercase letter to distinguish them from user-defined values. For example, .Release.Name, .Capabilities.KubeVersion, etc.

For security reasons, only configure items related to Kubernetes resources in values.yaml, such as deployment replicas and service ports. As for the configuration files of the iam-apiserver, iam-authz-server, iam-pump, and iamctl components, we create separate ConfigMaps and reference them in the Deployment.

Step 4: Package the Chart and upload it to the Chart repository.

This is an optional step and can be chosen based on your actual needs. If you want to learn more about the specific operation, you can refer to the Helm chart repository for more information.

Finally, the Chart package for the IAM application can be found in deployments/iam.

IAM Chart Deployment #

Above, we created the Chart package for the IAM application. Next, we will use this Chart package to create the IAM application with just one click. The deployment of the IAM Chart consists of a total of 10 steps, and you can follow along with me step by step.

Step 1: Configure scripts/install/environment.sh.

The scripts/install/environment.sh file contains various custom configurations. You mainly need to configure the following variables related to the database, while using the default values for other configurations.

  • MariaDB configuration: Variables in the environment.sh file starting with MARIADB_.
  • Redis configuration: Variables in the environment.sh file starting with REDIS_.
  • MongoDB configuration: Variables in the environment.sh file starting with MONGO_.

Step 2: Create the configuration files for the IAM application.

$ cd ${IAM_ROOT}
$ make gen.defaultconfigs # Generate default configuration files for the iam-apiserver, iam-authz-server, iam-pump, iamctl components
$ make gen.ca # Generate CA certificates

The above commands will store the IAM configuration files in the ${IAM_ROOT}/_output/configs/ directory.

Step 3: Create the iam namespace.

We will create all the resources related to the IAM application in the iam namespace. By creating IAM resources in a separate namespace, it not only facilitates maintenance but also effectively avoids impacting other Kubernetes resources.

$ kubectl create namespace iam

Step 4: Save the configuration files for the IAM services as ConfigMap resources in the Kubernetes cluster.

$ kubectl -n iam create configmap iam --from-file=${IAM_ROOT}/_output/configs/
$ kubectl -n iam get configmap iam
NAME   DATA   AGE
iam    4      13s

Step 5: Save the certificate files used by the IAM services as ConfigMap resources in the Kubernetes cluster.

$ kubectl -n iam create configmap iam-cert --from-file=${IAM_ROOT}/_output/cert
$ kubectl -n iam get configmap iam-cert
NAME       DATA   AGE
iam-cert   14     12s

Step 6: Create the Docker registry access key.

In the preparation phase, we opened the Tencent Cloud Container Registry Service and created a user 10000099xxxx with the password iam59!z$.

Next, we can create the docker-registry secret. When Kubernetes downloads Docker images, it requires a docker-registry secret for authentication. The creation command is as follows:

$ kubectl -n iam create secret docker-registry ccr-registry --docker-server=ccr.ccs.tencentyun.com --docker-username=10000099xxxx --docker-password='iam59!z$'

Step 7: Build the Docker images and push them to the container registry.

$ make push REGISTRY_PREFIX=ccr.ccs.tencentyun.com/marmotedu VERSION=v1.1.0

Step 8: Install the IAM Chart package.

In the 49th article, I introduced 4 methods to install a Chart package. Here, we will install using the path to the unpackaged IAM Chart. The installation method is as follows:

$ cd ${IAM_ROOT}
$ helm -n iam install iam deployments/iam
NAME: iam
LAST DEPLOYED: Sat Aug 21 17:46:56 2021
NAMESPACE: iam
STATUS: deployed
REVISION: 1
TEST SUITE: None

After executing helm install, Kubernetes will automatically deploy the application. When all the IAM application pods are in the Running state, it means that the IAM application has been successfully installed:

$ kubectl -n iam get pods|grep iam
iam-apiserver-cb4ff955-hs827         1/1     Running     0          66s
iam-authz-server-7fccc7db8d-chwnn    1/1     Running     0          66s
iam-pump-78b57b4464-rrlbf            1/1     Running     0          66s
iamctl-59fdc4995-xrzhn               1/1     Running     0          66s

Step 9: Test the IAM application.

We created a test Deployment called iamctl in the iam namespace using helm install. You can log in to the Pod created by the iamctl Deployment to perform operational tasks and smoke tests. The login command is as follows:

$ kubectl -n iam exec -it `kubectl -n iam get pods -l app=iamctl | awk '/iamctl/{print $1}'` -- bash

After logging into the iamctl-xxxxxxxxxx-xxxxx Pod, you can perform operational tasks and smoke tests.

Let’s start with operational tasks. The iamctl tool provides various functionalities through subcommands, which you can use to perform different tasks, as shown in the following image:

Image

Now let’s look at the smoke test:

# cd /opt/iam/scripts/install
# ./test.sh iam::test::smoke

If the output of the ./test.sh iam::test::smoke command ends with the string congratulations, smoke test passed!, it means that the IAM application has been successfully installed. As shown in the following image:

Image

Step 10: Destroy the resources of the EKS cluster.

$ kubectl delete namespace iam

You can choose to delete the EKS cluster as well if you no longer need it.

IAM Application Multi-Environment Deployment #

In practical project development, we need to deploy the IAM application to different environments with different configuration files. So how does the IAM project accomplish multi-environment deployment?

The IAM project creates multiple Helm values files (in the format values-{envName}-env.yaml) under the configs directory:

  • values-test-env.yaml: Helm values file for the testing environment.
  • values-pre-env.yaml: Helm values file for the pre-production environment.
  • values-prod-env.yaml: Helm values file for the production environment.

When deploying the IAM application, we specify the -f parameter in the command line, for example:

$ helm -n iam install -f configs/values-test-env.yaml iam deployments/iam # Install to the testing environment.

Summary #

In this lesson, we created a template Chart using helm create iam and developed the IAM application’s Helm Chart package based on this template. The final IAM Helm Chart package can be found at: deployments/iam.

With the Helm Chart package, we can deploy the entire IAM application with a single command helm -n iam install iam deployments/iam. When all the Pods in the IAM application are in the Running state, it indicates that the IAM application has been successfully deployed.

Finally, we can log in to the iamctl container and execute the command test.sh iam::test::smoke to perform smoke testing on the IAM application.

Practice Exercises #

  1. Try adding MariaDB, MongoDB, and Redis templates to the Helm Chart and use Helm to deploy the entire IAM application with one click.
  2. Try using the helm command to upgrade, rollback, and delete the IAM application.

Feel free to leave a message in the comments section to discuss and exchange ideas. See you in the next session.