31 How do programmers learn operation and maintenance knowledge? #

Hello, I’m Zheng Ye.

In the previous lecture, we talked about the automation of the development process. Our focus was on how to build an effective deployment package that will eventually be deployed online. Now let’s turn our attention to the related work of deployment.

Scattered Knowledge of Operations and Maintenance #

In some moderately-sized companies, a dedicated position called “Operations and Maintenance” (O&M) is set up for deployment work. Of course, the responsibilities of this position go far beyond just deployment and also involve maintaining the stability of the online systems. However, if your team is not large or your project is in the initial stage, these tasks often have to be completed by programmers themselves.

For a programmer, it is very important to understand how their own software is deployed and put into production. We not only need to understand the logic of a software, but also its physical deployment. Only in this way can we know how to fix it when problems occur.

More importantly, when designing, we can try to avoid the problems caused by deployment. And deployment is exactly where automation skills can be best utilized.

Okay, even if you are determined to learn the knowledge related to operations and maintenance, how do you plan to learn it? Let me ask you a question first, what comes to your mind when you hear about operations and maintenance?

If you are a programmer who has just entered this industry, you may think of Docker and Kubernetes; if you entered the industry a little earlier, you may also think of Chef, Puppet, and Ansible; if you entered even earlier, you may think of Shell scripts. Yes, all these things are related to operations and maintenance. So should I learn each of them one by one?

Based on my personal learning experience, if all the knowledge is scattered and not organized into a system, your existing knowledge cannot help you learn new knowledge, and this learning method is highly inefficient and extremely painful.

If it is structured knowledge, so-called learning new knowledge is just about learning incrementally, and there are not many new things that need to be understood. Learning efficiency will naturally be greatly improved. Therefore, if you want to learn operations and maintenance knowledge well, the first thing you need to do is to establish an effective knowledge system.

You may ask, but these operations and maintenance knowledge seem to be independent tools, right? I was once puzzled by this too. Although I already had a good understanding of each tool, I still lacked an effective knowledge system to connect them together until I attended a class.

Thanks to Chai Feng from Odd-e (https://chaifeng.com/about/), one time he gave me a class on DevOps (https://chaifeng.com/devops-tech-stack/). His explanation of operations and maintenance knowledge enlightened me, and since then, my operations and maintenance knowledge has a system.

To be precise, his class is about operations and maintenance for programmers. Today, I will reorganize this system according to my understanding and share it with you. This can also be considered as completing a knowledge output (http://time.geekbang.org/column/article/84663).

Alright, let’s get started!

Java Knowledge System #

As I mentioned earlier, the best way to learn something new is to learn incrementally. If we can find the connection between it and the existing knowledge system, we can apply the understanding of the existing knowledge to it.

As programmers, we already have a comprehensive knowledge system, which is our understanding of programming design. Understanding the knowledge system of DevOps can be borrowed from this system. How do we understand this?

Let’s take Java development, the most common scenario, as an example. What should I know to become a qualified Java programmer?

First, it is definitely the Java language. I need to understand the various syntax features of Java. However, understanding syntax alone won’t allow you to write decent programs. We also need to master the core libraries.

For Java, it is the various classes in the JDK, such as the most common ones like String, List, Map, and so on.

In theory, mastering the basic syntax and core libraries allows you to develop any program. But in practice, in order to avoid reinventing the wheel and reduce unnecessary workload, we also use a large number of third-party libraries, such as Google Guava, SLF4J, and so on.

In addition to implementing functionality, there are some structural codes that repeatedly appear. For example, in common REST services, we need to map database tables and objects together, convert results into JSON, and assemble various components of the system together.

To reduce structural code duplication, development frameworks have emerged. The most common development framework in Java is Spring.

At this point, you can complete basic code writing, but that’s not enough.

In Java, you won’t do everything from the ground up. For example, although you write REST services, you rarely touch the lowest-level HTTP implementations because these tasks are handled by the runtime environment.

What we need to do is deploy the packaged software to these runtime environments. In the world of Java, this responsibility is taken by containers like Tomcat and Jetty.

If you have just joined this field and start writing code using frameworks like Spring Boot, you may not have encountered such deployment processes because these frameworks have encapsulated the container and simplified the deployment process.

Tomcat and Jetty are often deployed on a single machine. However, in real-world scenarios, a single machine is usually not enough. What we might need is a cluster.

You might think of using Nginx for load balancing, but if you want to use the native Java solution, it’s time for enterprise-level application servers to come into play, such as IBM WebSphere, Oracle WebLogic Server, JBoss Enterprise Application Platform, and so on.

At this point, a complete Java application solution has been deployed. But what does this have to do with our knowledge of DevOps? We can use the same system to understand DevOps knowledge.

Operations Knowledge System #

First, let’s understand the language of operations. What is the language of operations? It is Shell, and the most common one is probably Bash. We interact with the operating system and computers, but we cannot directly use the operating system kernel. Shell provides us with an interface to access the services provided by the operating system kernel.

You may think that I am using a metaphor here by comparing Shell to a language, but that is not the case. Shell itself is a programming language. Most people know that Shell can be programmed, but almost no one learns Shell as a programming language. Basically, when needed, they search for it and copy and paste the code without truly understanding it.

The result is that once you write a script, you spend a lot of time struggling with its syntax just to make it run.

Once you have the language, the next step is the core library. What is the core library of operations? It is the various Unix/Linux core commands provided by Shell, such as: ls, cd, ps, grep, kill, cut, sort, uniq, etc. They are almost bound together with the operating system and released with the operating system.

After understanding the core part, you also need to know some third-party libraries. The third-party libraries in operations knowledge are commands that are not part of the core operating system commands, such as: rsync, curl, etc.

Are there frameworks available for operations like Java? Think about it, Java frameworks provide some general capabilities, and in operational work, there are also some general capabilities, such as: checking if a package is installed before installing it; checking if a service is started before starting it, and so on. So, the tools that can help us do these tasks well are our operations frameworks.

By now, you should have understood that when I refer to operations frameworks, I am actually talking about configuration management tools like Chef, Puppet, Ansible, etc. They help us automate these tedious tasks according to our definition.

Once you have the basic configuration of the software environment, the next step is to find a runtime environment to run the software. At this point, we need to understand technologies like virtual machines and Docker, which help us with deployment on a single machine.

In general, once we understand these contents, we can build a development or test environment. Unless there are very few users, we can consider deploying on a single machine in the production environment. Otherwise, all the technologies we have discussed so far are in the development stage.

If we need a cluster or a high availability environment, we need to further understand other technologies. This is when more complex technologies come into play, such as cloud technologies like Amazon AWS, OpenStack, and domestic cloud providers like Alibaba Cloud. If you are using basic technologies like Docker, you will need technologies like Kubernetes, Docker Swarm, etc.

By now, a relatively complete operations knowledge system has been established. Now you have a knowledge map and can navigate the operations field without getting lost easily. I hope you can take it and continue to expand your territory.

Summary #

Our focus today is on deploying the artifacts generated during the development process. The deployment process relies on operations knowledge, and every programmer should learn operations knowledge to ensure a clearer understanding of the software’s operation. Moreover, deployment work is very suitable for automation.

However, learning operations tools can be very difficult because many tools are scattered and lack a systematic approach.

Here, I have introduced you to an operations knowledge system, which is inspired by Java’s knowledge system and includes various aspects such as programming languages, core libraries, third-party libraries, development frameworks, standalone deployment, and cluster deployment. I have organized the various technologies mentioned today into a table below for your reference to better understand operations knowledge.

If there is only one thing you can remember from today’s content, please remember: Learn operations knowledge systematically.

Finally, I would like you to share your thoughts on what other operations knowledge can be included in this knowledge map. Please write your ideas in the comments section.

Thank you for reading. If you find this article helpful, please feel free to share it with your friends.