42 Software Deployment Practice Iam System Security Reinforcement and Horizontal Scaling Practices

42 Software Deployment Practice IAM System Security Reinforcement and Horizontal Scaling Practices #

Hello, I am Kong Lingfei.

In this session and the previous two sessions, we have been discussing the deployment of IAM (Identity and Access Management) based on physical or virtual machines. In the previous sessions, we learned how to deploy a highly available IAM application. Today, let’s take a look at the security enhancement and scalability of IAM applications. In this session, I will guide you through the process of securing IAM applications and provide specific steps for scaling up and down.

Next, let’s first see how to enhance the security of IAM applications.

Strengthening the Security of IAM Applications #

Services like iam-apiserver, iam-authz-server, MariaDB, Redis, and MongoDB provide the ability to bind to specific network interfaces. We can bind these services to internal network interfaces so that they only accept requests from within the internal network. This helps to strengthen our system.

We can also use iptables to achieve similar functionality. By consolidating security concerns into iptables rules, we can more easily maintain security settings.

This course focuses on strengthening the system using iptables to make it more secure. First, let me introduce the iptables tool.

Introduction to iptables #

iptables is the best firewall tool for Linux and is a user-level tool in the netfilter network subsystem of the Linux kernel.

netfilter provides a series of interfaces called HOOK points, which allow users to manipulate the packet flow of incoming and forwarded packets within the system. By registering packet processing functions at HOOK points, we can achieve packet forwarding, packet filtering, address translation, and other functions.

Users define various rules using iptables, which are then passed to the netfilter in the kernel. Netfilter filters network packets based on these rules. iptables software is usually installed by default on Linux systems. The firewall processes incoming network packets based on the rules in iptables.

The data structure in iptables is organized into tables, chains, and rules.

  • Tables: Tables provide specific functionality and each table contains multiple chains. There are a total of 5 tables in iptables: filter, nat, mangle, raw, and security. These tables are used for packet filtering, network address translation, packet restructuring, data tracing, and SELinux labeling, respectively.
  • Chains: Chains are the paths through which packets propagate, and each chain can have one or more rules. When a packet reaches a chain, iptables starts with the first rule in the chain and checks if the packet satisfies the conditions defined by the rule. If it does, iptables processes the packet according to the method defined by the rule. Otherwise, it continues to the next rule. If the packet doesn’t match any rule in the chain, iptables processes the packet based on the default policy defined for that chain.
  • Rules: Rules are stored in the kernel’s packet filtering table and describe how packets should be handled if they meet certain conditions. If a packet doesn’t meet the conditions, the next rule is evaluated.

In iptables, the types and functions of the tables and chains are as follows:

In the table above, the processing order of the five tables matters. When a packet reaches a chain, it is processed in the order of RAW, MANGLE, NAT, FILTER, and SECURITY.

So far, I have provided some basic knowledge about iptables, but it is not enough. To use iptables to strengthen your system, you also need to understand how to use the iptables tool. Next, let me introduce how iptables processes network packets.

Network Packet Processing Flow #

The network packet processing flow is as shown in the following diagram:

It can be divided into two steps.

In the first step, when a packet enters the network interface, it first enters the PREROUTING chain and determines whether to forward based on the destination IP address.

The second step is divided into two cases: if the packet is destined for the local machine, it reaches the INPUT chain. Upon arrival, any process on the machine will receive the packet. Programs on the local machine can send packets, which will go through the OUTPUT chain and then the POSTROUTING chain; if the packet is to be forwarded and the kernel allows forwarding, it will go through the FORWARD chain and finally the POSTROUTING chain.

Introduction to iptables Tool Usage #

iptables is powerful, so there are many ways to use it. Here, I will introduce the usage of the iptables tool and provide some examples.

  1. Command Format

The syntax of iptables is as follows:

iptables [-t table_name] command_option [chain_name] [rule_match] [-j target_action_or_jump]

Here is an example of using iptables:

iptables -t nat -I PREROUTING -p tcp --dport 8080 -j DNAT --to 10.0.4.88

Let me explain some of the parameters mentioned above.

  • table_name/chain_name: Specifies the table/chain that the iptables command operates on.
  • command_option: Specifies how iptables rules are processed, such as inserting, adding, deleting, or viewing rules.
  • rule_match: Specifies the conditions that the matching packets must meet for processing.
  • target_action_or_jump: Specifies how the firewall processes packets.

The iptables command options are divided into management control options and general options.

The management control options are as follows:

The general options are as follows:

There are several ways to process packets (target actions or jumps), as shown in the following table:

I have introduced the usage of the iptables tool above. Since there is quite a lot of content, you may still not know how to use the iptables tool. It’s okay, next you can look at some examples I provided to help you understand.

  1. Command Examples

The following command examples assume the use of the FILTER table, which means the rules are stored in the FILTER table. Hence, each iptables command is equivalent to adding the -t filter parameter.

  1. Reject all ICMP protocol packets entering the firewall:
$ iptables -I INPUT -p icmp -j REJECT
  1. Allow all packets to be forwarded by the firewall except the ICMP protocol:
$ iptables -A FORWARD -p ! icmp -j ACCEPT
  1. Reject data forwarded from the host with IP address 192.168.1.10 and allow data forwarded from the network segment 192.168.0.0/24:
$ iptables -A FORWARD -s 192.168.1.11 -j REJECT
$ iptables -A FORWARD -s 192.168.0.0/24 -j ACCEPT
  1. Discard packets with a source address in the private network from entering the firewall machine via the external interface (eth1):

access: firewall: drop: - eth1: 192.168.0.0/16 - eth1: 172.16.0.0/12 - eth1: 10.0.0.0/8 allow: - eth1: 202.13.0.0/16: tcp: 22 - eth1: 127.0.0.1: tcp: 20-1024 - eth1: 192.168.0.0/24: udp: 53 - eth1: 0.0.0.0/0: icmp: Echo-Request - eth1: 00:0c:29:27:55:3F - eth1: 0.0.0.0/0: tcp: 20,21,25,110,1250-1280 - eth1: ! 192.168.1.20-192.168.1.99 - eth1: tcp ! –syn - eth1: 0.0.0.0/0: tcp: 20,21,80,20450-20480 - eth1: 0.0.0.0/0: tcp: 20450-20480 - eth1: 0.0.0.0/0: tcp: ESTABLISHED,RELATED

  - 3306
  - 6379
  - 27017


  1. 生成新的iptables初始化脚本：


```plaintext
$ go run tools/geniptables/main.go -c access.yaml -t app -a -o firewall.sh
$ ls firewall.sh
firewall.sh

你可以打开firewall.sh文件，查看该脚本设置的规则。

  1. 将firewall.sh脚本拷贝到10.0.4.20和10.0.4.21节点执行：
$ scp firewall.sh [email protected]:/tmp/
$ scp firewall.sh [email protected]:/tmp/

登陆10.0.4.20和10.0.4.21机器，执行/tmp/firewall.sh。

  1. 在10.0.4.20（数据库节点）节点上，设置iptables规则，以允许各节点访问：

因为数据库节点也位于10.0.4.20节点，所以只需要添加新的rule，并将iptables -A INPUT -j DROP规则放到最后执行即可。

$ go run tools/geniptables/main.go -c access.yaml -t db -o addrules.sh

然后，将addrules.sh脚本拷贝到10.0.4.20节点执行。

注意，因为iptables是按顺序进行规则过滤的，所以需要将iptables -A INPUT -j DROP规则放在新设置规则的后面，否则执行不到新设置的规则。你可以在设置完iptables规则之后，执行下面的命令来将DROP放到最后：

iptables -A INPUT -j LOG --log-level 7 --log-prefix "Default Deny"
iptables -A INPUT -j DROP

生成的addrules.sh脚本加入以上设置。

第二步，设置重启自动加载iptables规则。

前面我们在各个节点设置了iptables规则，但是这些规则在系统重启后会丢失。为了使系统重启后自动重新设置这些规则，我们需要将当前的iptables规则保存起来，让系统重启时自动加载。需要进行下面两个步骤。

  1. 保存现有的规则：
$ sudo iptables-save > /etc/sysconfig/iptables
  1. 添加下面的命令行到/etc/rc.d/rc.local文件中：
$ iptables-restore < /etc/sysconfig/iptables

第三步，自动化。

在上面的步骤中，我们自动生成了iptables规则，并手动登陆到节点进行设置。你肯定也发现了，整个流程手动操作过多，容易出错，效率还低。你可以参考设置过程，将这些设置工作自动化，比如编写脚本，一键刷新所有节点的iptables规则。

另外，我们再来看下在新增节点和删除节点两种场景下，如何设置iptables规则。

场景1：新增节点

如果我们要扩容一个节点，也需要在新节点设置防火墙规则，并在数据库节点设置防火墙规则允许来自新节点的访问。

假如我们新增一个10.0.4.22节点，这里要设置防火墙规则，需要下面的4个步骤。

  1. 编辑access.yaml，在hosts列表下新增10.0.4.22节点IP。编辑后内容如下：
# 允许登录SSH节点的来源IP，可以是固定IP(例如10.0.4.2)，也可以是个网段，0.0.0.0/0代表不限制来源IP
ssh-source: 10.0.4.0/24

# IAM应用节点列表（来源IP）
hosts:
  - 10.0.4.20
  - 10.0.4.21
  - 10.0.4.22

# 来源IP可以访问的应用端口列表（iam-apiserver, iam-authz-server, iam-pump对外暴露的的端口）
ports:
  - 8080
  - 8443
  - 9090
  - 9443
  - 7070

# 来源IP可以访问的数据库端口列表（Redis, MariaDB, MongoDB）
dbports:
  - 3306
  - 6379
  - 27017

- 3306
- 6379
- 27017
  1. Set iptables rules on the 10.0.4.22 node:
$ go run tools/geniptables/main.go -c access.yaml -t app -a -o firewall.sh

Copy the firewall.sh script to the 10.0.4.22 node and execute it.

  1. Add rules on existing nodes to allow access to the Nginx service from 10.0.4.22:
$ go run tools/geniptables/main.go -c access.yaml -t app 10.0.4.22 -o addrules.sh

Copy the addrules.sh script to the existing nodes and execute it.

  1. Add iptables rules on the database node to allow access from the new node:
$ go run tools/geniptables/main.go -c access.yaml -t db 10.0.4.22 -o addrules.sh

Copy the addrules.sh script to the 10.0.4.20 node and execute it.

Scenario 2: Removing a Node

If we want to remove a node, we need to remove its access permissions from both the remaining nodes and the database node. For example, if we want to remove the 10.0.4.22 node, we need to perform the following 3 steps to set up the firewall rules.

  1. Remove access permission to 10.0.4.22 node from the remaining nodes:
$ go run tools/geniptables/main.go -c access.yaml -t app --delete 10.0.4.22 -o delete.sh

Copy the delete.sh script to the remaining nodes (10.0.4.20, 10.0.4.21) and execute it.

  1. Remove access permission to 10.0.4.22 node from the database node:
$ go run tools/geniptables/main.go -c access.yaml -t db --delete 10.0.4.22 -o delete.sh

Copy the delete.sh script to the 10.0.4.20 node and execute it.

  1. Remove the offline node from the hosts section of the access.yaml file.

IAM Security Hardening (Internal Network Security) #

Now let’s consider the second scenario: suppose our system is deployed in a secure internal network environment. In this case, securing the system becomes exceptionally simple, as we only need to allow clients with source IPs in the internal network to access various ports provided by us. After setting up the iptables rules, we no longer need to make any changes when adding or removing nodes.

This can be divided into 5 steps.

Step 1: Go to the root directory of the IAM project source code.

Step 2: Configure accesss.yaml (The tool automatically generates iptables setup scripts based on this configuration). The content of the file is as follows (configs/access.yaml):

# Allow SSH login from source IP, can be a fixed IP (e.g. 10.0.4.2) or a subnet. 0.0.0.0/0 means no restriction on the source IP.
ssh-source: 10.0.4.0/24

# Application port list accessible from source IPs (ports exposed by iam-apiserver, iam-authz-server, iam-pump)
ports:
  - 8080
  - 8443
  - 9090
  - 9443
  - 7070

# Database port list accessible from source IPs (Redis, MariaDB, MongoDB)
dbports:
  - 3306
  - 6379
  - 27017

In the above configuration, we specify only the IAM service ports and database ports.

Step 3: Generate iptables initialization script:

$ go run tools/geniptables/main.go -c access.yaml -t app --cidr=10.0.4.0/24 -a -o firewall.sh
$ ls firewall.sh
firewall.sh

Step 4: Copy the firewall.sh script to the 10.0.4.20 and 10.0.4.21 nodes:

$ scp firewall.sh [email protected]:/tmp/
$ scp firewall.sh [email protected]:/tmp/

Log in to the 10.0.4.20 and 10.0.4.21 machines and execute /tmp/firewall.sh.

Step 5: Set iptables rules on the 10.0.4.20 (database node) to allow access from all nodes.

Since the database node is also located on the 10.0.4.20 node, we only need to add a new rule and place the iptables -A INPUT -j DROP rule at the end.

$ go run tools/geniptables/main.go -c access.yaml -t db --cidr=10.0.4.0/24 -o addrules.sh

Then, copy the addrules.sh script to the 10.0.4.20 node and execute it.

To add a new node, you only need to execute Step 3 again to generate the firewall.sh script, and then copy the firewall.sh script to the new node and execute it. No action is required to delete a node.

Next, let’s take a look at how to perform elastic scaling operations on the IAM application.

Elastic Scaling #

Elastic scaling includes scaling up and scaling down. Scaling up means easily adding computing nodes to distribute workload and expand computing capacity as the business grows. Scaling down means easily reducing computing nodes to decrease costs as the business volume decreases.

During the initial launch of a system, the business volume is usually not large, but as the product iterates and the user base grows, the system will experience an increasing number of requests and greater pressure. At this point, our system architecture needs to have the ability to horizontally scale to meet business needs and avoid system breakdown due to high system load.

Some e-commerce systems, before major promotional events like Double 11, will proactively scale up their computing nodes to handle the upcoming peak traffic. However, after the event, the traffic gradually decreases, and at this time, our system needs to have the ability to scale down, reducing computing nodes and saving costs.

A scalable system architecture is something we must ensure when designing a system. If a system lacks scalability, when we need to scale up or scale down later, we will need to make significant changes to the code, which not only increases additional workload but also slows down product iteration speed. And think about it, even after making changes, it requires testing, and after deployment, there may still be bugs introduced due to code changes. In short, a system architecture without scalability can be said to be a source of endless trouble.

IAM system has considered scalability from the beginning of its design. We can easily scale up or scale down the system. Next, I will explain how to scale up and scale down the system respectively.

System Scaling Up #

The steps to scale up the system are simple, you just need to follow these 5 steps:

  1. Apply for computing nodes based on your needs. Unless specified, the configuration and operating system of the computing nodes should be consistent with the existing nodes.
  2. Deploy iam-apiserver, iam-authz-server, and iam-pump on the new nodes, following the same deployment process as the other nodes.
  3. Deploy Nginx on the new nodes and add the IP address of the new nodes to the Nginx upstream configuration of all existing nodes. Restart Nginx.
  4. Deploy Keepalived on the new nodes and add the IP address of the new nodes to the unicast_peer configuration of all existing nodes. Restart Keepalived.
  5. Modify the iptables rules and refresh the iptables on all machines.

System Scaling Down #

System scaling down is the reverse operation of scaling up and also consists of 5 steps:

  1. Determine the nodes to be deleted based on your needs.
  2. Shut down the iam-apiserver, iam-authz-server, and iam-pump services on the nodes to be deleted.
  3. Remove the IP address of the nodes to be deleted from the Nginx upstream configuration of all remaining nodes. Restart Nginx.
  4. Remove the IP address of the nodes to be deleted from the unicast_peer configuration of all remaining nodes. Restart Keepalived.
  5. Modify the iptables rules and refresh the iptables on all remaining machines.

Summary #

Security is crucial for application software. When deploying applications, it is important to assess their security and take measures to ensure it.

The simplest and most effective way to enhance application security during software deployment is to use iptables rules. The implementation approach is straightforward: use iptables rules to only allow specific IP addresses to access specific ports.

After the business is officially launched, there might be periods of high or low traffic. During peak periods, additional machines may be required to increase system throughput. This can be achieved by installing the necessary service components on new machines and configuring Nginx and Keepalived. Then, the new server can be added to the Nginx upstream group. During low traffic periods, servers can be removed from the Nginx upstream list and the IAM application services can be shut down.

Homework Exercises #

  1. Please expand one more machine based on the content learned in this lecture.
  2. Think about what other good application security reinforcement methods you have when deploying applications. You are welcome to share them in the comments section.

Feel free to discuss and exchange thoughts with me in the comments section. See you in the next lecture.