k8s系列03-kubeadm部署calico网络的k8s集群

本文主要在centos7系统上基于docker和calico组件部署v1.23.6版本的k8s原生集群，由于集群主要用于自己平时学习和测试使用，加上资源有限，暂不涉及高可用部署。

此前写的一些关于k8s基础知识和集群搭建的一些方案，有需要的同学可以看一下。

1、准备工作

1.1 calico-集群节点信息

机器均为8C8G的虚拟机，硬盘为100G。

IP	Hostname
10.31.88.1	tiny-calico-master-88-1.k8s.tcinternal
10.31.88.11	tiny-calico-worker-88-11.k8s.tcinternal
10.31.88.12	tiny-calico-worker-88-12.k8s.tcinternal
10.88.64.0/18	podSubnet
10.88.0.0/18	serviceSubnet

1.2 检查mac和product_uuid

同一个k8s集群内的所有节点需要确保mac地址和product_uuid均唯一，开始集群初始化之前需要检查相关信息

# 检查mac地址
ip link 
ifconfig -a

# 检查product_uuid
sudo cat /sys/class/dmi/id/product_uuid

1.3 配置ssh免密登录（可选）

如果k8s集群的节点有多个网卡，确保每个节点能通过正确的网卡互联访问

# 在root用户下面生成一个公用的key，并配置可以使用该key免密登录
su root
ssh-keygen
cd /root/.ssh/
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys


cat >> ~/.ssh/config <

1.4 修改hosts文件

cat >> /etc/hosts <

1.5 关闭swap内存

# 使用命令直接关闭swap内存
swapoff -a
# 修改fstab文件禁止开机自动挂载swap分区
sed -i '/swap / s/^\(.*\)$/#\1/g' /etc/fstab

1.6 配置时间同步

这里可以根据自己的习惯选择ntp或者是chrony同步均可，同步的时间源服务器可以选择阿里云的ntp1.aliyun.com或者是国家时间中心的ntp.ntsc.ac.cn。

使用ntp同步

# 使用yum安装ntpdate工具
yum install ntpdate -y

# 使用国家时间中心的源同步时间
ntpdate ntp.ntsc.ac.cn

# 最后查看一下时间
hwclock

使用chrony同步

# 使用yum安装chrony
yum install chrony -y

# 设置开机启动并开启chony并查看运行状态
systemctl enable chronyd.service
systemctl start chronyd.service
systemctl status chronyd.service

# 当然也可以自定义时间服务器
vim /etc/chrony.conf

# 修改前
$ grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

# 修改后
$ grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server ntp.ntsc.ac.cn iburst

# 重启服务使配置文件生效
systemctl restart chronyd.service

# 查看chrony的ntp服务器状态
chronyc sourcestats -v
chronyc sources -v

1.7 关闭selinux

# 使用命令直接关闭
setenforce 0

# 也可以直接修改/etc/selinux/config文件
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

1.8 配置防火墙

k8s集群之间通信和服务暴露需要使用较多端口，为了方便，直接禁用防火墙

# centos7使用systemctl禁用默认的firewalld服务
systemctl disable firewalld.service

1.9 配置netfilter参数

这里主要是需要配置内核加载br_netfilter和iptables放行ipv6和ipv4的流量，确保集群内的容器能够正常通信。

cat <

1.10 关闭IPV6（可选）

虽然新版本的k8s已经支持双栈网络，但是本次的集群部署过程并不涉及IPv6网络的通信，因此关闭IPv6网络支持

# 直接在内核中添加ipv6禁用参数
grubby --update-kernel=ALL --args=ipv6.disable=1

1.11 配置IPVS（可选）

IPVS是专门设计用来应对负载均衡场景的组件，kube-proxy 中的 IPVS 实现通过减少对 iptables 的使用来增加可扩展性。在 iptables 输入链中不使用 PREROUTING，而是创建一个假的接口，叫做 kube-ipvs0，当k8s集群中的负载均衡配置变多的时候，IPVS能实现比iptables更高效的转发性能。

注意在4.19之后的内核版本中使用nf_conntrack模块来替换了原有的nf_conntrack_ipv4模块

(Notes: use nf_conntrack instead of nf_conntrack_ipv4 for Linux kernel 4.19 and later)

# 在使用ipvs模式之前确保安装了ipset和ipvsadm
sudo yum install ipset ipvsadm -y

# 手动加载ipvs相关模块
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4

# 配置开机自动加载ipvs相关模块
cat <

2、安装container runtime

2.1 安装docker

详细的官方文档可以参考这里，由于在刚发布的1.24版本中移除了docker-shim，因此安装的版本≥1.24的时候需要注意容器运行时的选择。这里我们安装的版本低于1.24，因此我们继续使用docker。

docker的具体安装可以参考我之前写的这篇文章，这里不做赘述。

# 安装必要的依赖组件并且导入docker官方提供的yum源
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo

# 我们直接安装最新版本的docker
yum install docker-ce docker-ce-cli containerd.io

2.2 配置cgroup drivers

CentOS7使用的是systemd来初始化系统并管理进程，初始化进程会生成并使用一个 root 控制组 (cgroup), 并充当 cgroup 管理器。 Systemd 与 cgroup 集成紧密，并将为每个 systemd 单元分配一个 cgroup。 我们也可以配置容器运行时和 kubelet 使用 cgroupfs。 连同 systemd 一起使用 cgroupfs 意味着将有两个不同的 cgroup 管理器。而当一个系统中同时存在cgroupfs和systemd两者时，容易变得不稳定，因此最好更改设置，令容器运行时和 kubelet 使用 systemd 作为 cgroup 驱动，以此使系统更为稳定。 对于 Docker, 需要设置 native.cgroupdriver=systemd 参数。

参考官方的说明文档：

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers

参考配置说明文档

https://kubernetes.io/zh/docs/setup/production-environment/container-runtimes/#docker

sudo mkdir /etc/docker
cat <

2.3 关于kubelet的cgroup driver

k8s官方有详细的文档介绍了如何设置kubelet的cgroup driver，需要特别注意的是，在1.22版本开始，如果没有手动设置kubelet的cgroup driver，那么默认会设置为systemd

Note: In v1.22, if the user is not setting the cgroupDriver field under KubeletConfiguration, kubeadm will default it to systemd.

一个比较简单的指定kubelet的cgroup driver的方法就是在kubeadm-config.yaml加入cgroupDriver字段

# kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.21.0
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd

我们可以直接查看configmaps来查看初始化之后集群的kubeadm-config配置。

$ kubectl describe configmaps kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       
Annotations:  

Data
====
ClusterConfiguration:
----
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.6
networking:
  dnsDomain: cali-cluster.tclocal
  serviceSubnet: 10.88.0.0/18
scheduler: {}


BinaryData
====

Events:  

当然因为我们需要安装的版本高于1.22.0并且使用的就是systemd，因此可以不用再重复配置。

3、安装kube三件套

对应的官方文档可以参考这里

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

kube三件套就是kubeadm、kubelet 和 kubectl，三者的具体功能和作用如下：

• kubeadm：用来初始化集群的指令。
• kubelet：在集群中的每个节点上用来启动 Pod 和容器等。
• kubectl：用来与集群通信的命令行工具。

需要注意的是：

• kubeadm不会帮助我们管理kubelet和kubectl，其他两者也是一样的，也就是说这三者是相互独立的，并不存在谁管理谁的情况；
• kubelet的版本必须小于等于API-server的版本，否则容易出现兼容性的问题；
• kubectl并不是集群中的每个节点都需要安装，也并不是一定要安装在集群中的节点，可以单独安装在自己本地的机器环境上面，然后配合kubeconfig文件即可使用kubectl命令来远程管理对应的k8s集群；

CentOS7的安装比较简单，我们直接使用官方提供的yum源即可。需要注意的是这里需要设置selinux的状态，但是前面我们已经关闭了selinux，因此这里略过这步。

# 直接导入谷歌官方的yum源
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


# 接下来直接安装三件套即可
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# 如果网络环境不好出现gpgcheck验证失败导致无法正常读取yum源，可以考虑关闭该yum源的repo_gpgcheck
sed -i 's/repo_gpgcheck=1/repo_gpgcheck=0/g' /etc/yum.repos.d/kubernetes.repo
# 或者在安装的时候禁用gpgcheck
sudo yum install -y kubelet kubeadm kubectl --nogpgcheck --disableexcludes=kubernetes



# 如果想要安装特定版本，可以使用这个命令查看相关版本的信息
sudo yum list --nogpgcheck kubelet kubeadm kubectl --showduplicates --disableexcludes=kubernetes
# 这里我们为了保留使用docker-shim，因此我们按照1.24.0版本的前一个版本1.23.6
sudo yum install -y kubelet-1.23.6-0 kubeadm-1.23.6-0 kubectl-1.23.6-0 --nogpgcheck --disableexcludes=kubernetes

# 安装完成后配置开机自启kubelet
sudo systemctl enable --now kubelet

4、初始化集群

4.1 编写配置文件

在集群中所有节点都执行完上面的三点操作之后，我们就可以开始创建k8s集群了。因为我们这次不涉及高可用部署，因此初始化的时候直接在我们的目标master节点上面操作即可。

# 我们先使用kubeadm命令查看一下主要的几个镜像版本
# 因为我们此前指定安装了旧的1.23.6版本，这里的apiserver镜像版本也会随之回滚
$ kubeadm config images list
I0506 11:24:17.061315   16055 version.go:255] remote version is much newer: v1.24.0; falling back to: stable-1.23
k8s.gcr.io/kube-apiserver:v1.23.6
k8s.gcr.io/kube-controller-manager:v1.23.6
k8s.gcr.io/kube-scheduler:v1.23.6
k8s.gcr.io/kube-proxy:v1.23.6
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

# 为了方便编辑和管理，我们还是把初始化参数导出成配置文件
$ kubeadm config print init-defaults > kubeadm-calico.conf

• 考虑到大多数情况下国内的网络无法使用谷歌的k8s.gcr.io镜像源，我们可以直接在配置文件中修改imageRepository参数为阿里的镜像源
• kubernetesVersion字段用来指定我们要安装的k8s版本
• localAPIEndpoint参数需要修改为我们的master节点的IP和端口，初始化之后的k8s集群的apiserver地址就是这个
• serviceSubnet和dnsDomain两个参数默认情况下可以不用修改，这里我按照自己的需求进行了变更
• nodeRegistration里面的name参数修改为对应master节点的hostname
• 新增配置块使用ipvs，具体可以参考官方文档

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.31.88.1
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: tiny-calico-master-88-1.k8s.tcinternal
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.23.6
networking:
  dnsDomain: cali-cluster.tclocal
  serviceSubnet: 10.88.0.0/18
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

4.2 初始化集群

此时我们再查看对应的配置文件中的镜像版本，就会发现已经变成了对应阿里云镜像源的版本

# 查看一下对应的镜像版本，确定配置文件是否生效
$ kubeadm config images list --config kubeadm-calico.conf
registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.6
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6
registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.6
registry.aliyuncs.com/google_containers/kube-proxy:v1.23.6
registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/etcd:3.5.1-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6

# 确认没问题之后我们直接拉取镜像
$ kubeadm config images pull --config kubeadm-calico.conf
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.23.6
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.6
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.1-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

# 初始化
$ kubeadm init --config kubeadm-calico.conf
[init] Using Kubernetes version: v1.23.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
...此处略去一堆输出...

当我们看到下面这个输出结果的时候，我们的集群就算是初始化成功了。

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.31.88.1:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:a4189d36d164a865be540d48fcd10ff13e2f90ed6e901201b6ea2baf96dae0ae

4.3 配置kubeconfig

刚初始化成功之后，我们还没办法马上查看k8s集群信息，需要配置kubeconfig相关参数才能正常使用kubectl连接apiserver读取集群信息。

# 对于非root用户，可以这样操作
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 如果是root用户，可以直接导入环境变量
export KUBECONFIG=/etc/kubernetes/admin.conf

# 添加kubectl的自动补全功能
echo "source <(kubectl completion bash)" >> ~/.bashrc

前面我们提到过kubectl不一定要安装在集群内，实际上只要是任何一台能连接到apiserver的机器上面都可以安装kubectl并且根据步骤配置kubeconfig，就可以使用kubectl命令行来管理对应的k8s集群。

配置完成后，我们再执行相关命令就可以查看集群的信息了。

$ kubectl cluster-info
Kubernetes control plane is running at https://10.31.88.1:6443
CoreDNS is running at https://10.31.88.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'

$ kubectl get nodes -o wide
NAME                                     STATUS     ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
tiny-calico-master-88-1.k8s.tcinternal   NotReady   control-plane,master   4m15s   v1.23.6   10.31.88.1            CentOS Linux 7 (Core)   3.10.0-1160.62.1.el7.x86_64   docker://20.10.14

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                                             READY   STATUS    RESTARTS   AGE     IP           NODE                                     NOMINATED NODE   READINESS GATES
kube-system   coredns-6d8c4cb4d-r8r9q                                          0/1     Pending   0          4m20s                                                        
kube-system   coredns-6d8c4cb4d-ztq6w                                          0/1     Pending   0          4m20s                                                        
kube-system   etcd-tiny-calico-master-88-1.k8s.tcinternal                      1/1     Running   0          4m25s   10.31.88.1   tiny-calico-master-88-1.k8s.tcinternal              
kube-system   kube-apiserver-tiny-calico-master-88-1.k8s.tcinternal            1/1     Running   0          4m26s   10.31.88.1   tiny-calico-master-88-1.k8s.tcinternal              
kube-system   kube-controller-manager-tiny-calico-master-88-1.k8s.tcinternal   1/1     Running   0          4m27s   10.31.88.1   tiny-calico-master-88-1.k8s.tcinternal              
kube-system   kube-proxy-v6cg9                                                 1/1     Running   0          4m20s   10.31.88.1   tiny-calico-master-88-1.k8s.tcinternal              
kube-system   kube-scheduler-tiny-calico-master-88-1.k8s.tcinternal            1/1     Running   0          4m25s   10.31.88.1   tiny-calico-master-88-1.k8s.tcinternal              

4.4 添加worker节点

这时候我们还需要继续添加剩下的两个节点作为worker节点运行负载，直接在剩下的节点上面运行集群初始化成功时输出的命令就可以成功加入集群：

$ kubeadm join 10.31.88.1:6443 --token abcdef.0123456789abcdef \
>         --discovery-token-ca-cert-hash sha256:a4189d36d164a865be540d48fcd10ff13e2f90ed6e901201b6ea2baf96dae0ae
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

如果不小心没保存初始化成功的输出信息也没有关系，我们可以使用kubectl工具查看或者生成token

# 查看现有的token列表
$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2022-05-07T05:19:08Z   authentication,signing                                                        system:bootstrappers:kubeadm:default-node-token

# 如果token已经失效，那就再创建一个新的token
$ kubeadm token create
e31cv1.lbtrzwp6mzon78ue
$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   23h         2022-05-07T05:19:08Z   authentication,signing                                                        system:bootstrappers:kubeadm:default-node-token
e31cv1.lbtrzwp6mzon78ue   23h         2022-05-07T05:51:40Z   authentication,signing                                                        system:bootstrappers:kubeadm:default-node-token

# 如果找不到--discovery-token-ca-cert-hash参数，则可以在master节点上使用openssl工具来获取
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null |    openssl dgst -sha256 -hex | sed 's/^.* //'
a4189d36d164a865be540d48fcd10ff13e2f90ed6e901201b6ea2baf96dae0ae

添加完成之后我们再查看集群的节点可以发现这时候已经多了两个node，但是此时节点的状态还是NotReady，接下来就需要部署CNI了。

$ kubectl get nodes
NAME                                      STATUS     ROLES                  AGE    VERSION
tiny-calico-master-88-1.k8s.tcinternal    NotReady   control-plane,master   20m    v1.23.6
tiny-calico-worker-88-11.k8s.tcinternal   NotReady                    105s   v1.23.6
tiny-calico-worker-88-12.k8s.tcinternal   NotReady                    35s    v1.23.6

5、安装CNI

5.1 编写manifest文件

calico的安装也比较简单，官方提供了多种安装方式，我们这里使用yaml（自定义manifests）进行安装，并且使用etcd作为datastore。

# 我们先把官方的yaml模板下载下来，然后对关键字段逐个修改
curl https://projectcalico.docs.tigera.io/manifests/calico-etcd.yaml -O

针对calico-etcd.yaml文件，我们需要修改一些参数以适配我们的集群：

• CALICO_IPV4POOL_CIDR参数，配置的是pod的网段，这里我们使用此前计划好的10.88.64.0/18；CALICO_IPV4POOL_BLOCK_SIZE参数，配置的是分配子网的大小，默认是26

              # The default IPv4 pool to create on startup if none exists. Pod IPs will be
              # chosen from this range. Changing this value after installation will have
              # no effect. This should fall within `--cluster-cidr`.
              - name: CALICO_IPV4POOL_CIDR
                value: "10.88.64.0/18"
              - name: CALICO_IPV4POOL_BLOCK_SIZE
                value: "26"
  

• CALICO_IPV4POOL_IPIP参数，控制是否启用ip-ip模式，默认情况下是Always，由于我们的节点都在同一个二层网络，这里修改为Never或者是CrossSubnet都可以。

  其中Never表示不启用ip-ip模式，而CrossSubnet则表示仅当跨子网的时候才启用ip-ip模式

              # Enable IPIP
              - name: CALICO_IPV4POOL_IPIP
                value: "Never"
  

• ConfigMap里面的etcd_endpoints变量配置etcd的连接端口和地址，为了安全我们这里开启TLS认证，当然如果不想配置证书的，也可以不使用TLS，然后这三个字段直接留空不做修改

  kind: ConfigMap
  apiVersion: v1
  metadata:
    name: calico-config
    namespace: kube-system
  data:
    # Configure this with the location of your etcd cluster.
    # etcd_endpoints: "http://:"
    # If you're using TLS enabled etcd uncomment the following.
    # You must also populate the Secret below with these files.
    # etcd_ca: ""   # "/calico-secrets/etcd-ca"
    # etcd_cert: "" # "/calico-secrets/etcd-cert"
    # etcd_key: ""  # "/calico-secrets/etcd-key"
    etcd_endpoints: "https://10.31.88.1:2379"
    etcd_ca: "/etc/kubernetes/pki/etcd/ca.crt"
    etcd_cert: "/etc/kubernetes/pki/etcd/server.crt"
    etcd_key: "/etc/kubernetes/pki/etcd/server.key"
  

• Secret里面的 name: calico-etcd-secrets下面的data字段，需要把上面的三个证书内容使用该命令cat | base64 -w 0转成base64编码格式

  ---
  # Source: calico/templates/calico-etcd-secrets.yaml
  # The following contains k8s Secrets for use with a TLS enabled etcd cluster.
  # For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
  apiVersion: v1
  kind: Secret
  type: Opaque
  metadata:
    name: calico-etcd-secrets
    namespace: kube-system
  data:
    # Populate the following with etcd TLS configuration if desired, but leave blank if
    # not using TLS for etcd.
    # The keys below should be uncommented and the values populated with the base64
    # encoded contents of each file that would be associated with the TLS data.
    # Example command for encoding a file contents: cat  | base64 -w 0
    etcd-key: LS0tLS1CRUdJTi......tLS0tCg==
    etcd-cert: LS0tLS1CRUdJT......tLS0tLQo=
    etcd-ca: LS0tLS1CRUdJTiB......FLS0tLS0K
  

5.2 部署calico

修改完成之后我们直接部署即可

$ kubectl apply -f calico-etcd.yaml
secret/calico-etcd-secrets created
configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created

# 查看pod是否正常运行
$ kubectl get pods -A
NAMESPACE     NAME                                                             READY   STATUS    RESTARTS        AGE
kube-system   calico-kube-controllers-5c4bd49f9b-6b2gr                         1/1     Running   5 (3m18s ago)   6m18s
kube-system   calico-node-bgsfs                                                1/1     Running   5 (2m55s ago)   6m18s
kube-system   calico-node-tr88g                                                1/1     Running   5 (3m19s ago)   6m18s
kube-system   calico-node-w59pc                                                1/1     Running   5 (2m36s ago)   6m18s
kube-system   coredns-6d8c4cb4d-r8r9q                                          1/1     Running   0               3h8m
kube-system   coredns-6d8c4cb4d-ztq6w                                          1/1     Running   0               3h8m
kube-system   etcd-tiny-calico-master-88-1.k8s.tcinternal                      1/1     Running   0               3h8m
kube-system   kube-apiserver-tiny-calico-master-88-1.k8s.tcinternal            1/1     Running   0               3h8m
kube-system   kube-controller-manager-tiny-calico-master-88-1.k8s.tcinternal   1/1     Running   0               3h8m
kube-system   kube-proxy-n65sb                                                 1/1     Running   0               169m
kube-system   kube-proxy-qmxhp                                                 1/1     Running   0               168m
kube-system   kube-proxy-v6cg9                                                 1/1     Running   0               3h8m
kube-system   kube-scheduler-tiny-calico-master-88-1.k8s.tcinternal            1/1     Running   0               3h8m

# 查看calico-kube-controllers的pod日志是否有报错
$ kubectl logs -f calico-kube-controllers-5c4bd49f9b-6b2gr -n kube-system

5.3 pod安装calicoctl

calicoctl是用来查看管理calico的命令行工具，定位上有点类似于calico版本的kubectl，因为我们前面使用了etcd作为calico的datastore，这里直接选择在k8s集群中以pod的形式部署calicoctl的方式更加简单。

• calicoctl的版本最好和部署的calico一致，这里均为v3.22.2
• calicoctl的etcd配置最好和部署的calico一致，因为前面部署calico的时候etcd开启了TLS，因此这里我们也要修改yaml文件开启TLS

# 为了方便后期管理，我们先把calicoctl.yaml下载到本地再进行部署
$ wget https://projectcalico.docs.tigera.io/manifests/calicoctl-etcd.yaml

$ cat calicoctl-etcd.yaml
# Calico Version v3.22.2
# https://projectcalico.docs.tigera.io/releases#v3.22.2
# This manifest includes the following component versions:
#   calico/ctl:v3.22.2

apiVersion: v1
kind: Pod
metadata:
  name: calicoctl
  namespace: kube-system
spec:
  nodeSelector:
    kubernetes.io/os: linux
  hostNetwork: true
  containers:
  - name: calicoctl
    image: calico/ctl:v3.22.2
    command:
      - /calicoctl
    args:
      - version
      - --poll=1m
    env:
    - name: ETCD_ENDPOINTS
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_endpoints
    # If you're using TLS enabled etcd uncomment the following.
    # Location of the CA certificate for etcd.
    - name: ETCD_CA_CERT_FILE
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_ca
    # Location of the client key for etcd.
    - name: ETCD_KEY_FILE
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_key
    # Location of the client certificate for etcd.
    - name: ETCD_CERT_FILE
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_cert
    volumeMounts:
    - mountPath: /calico-secrets
      name: etcd-certs
  volumes:
    # If you're using TLS enabled etcd uncomment the following.
    - name: etcd-certs
      secret:
        secretName: calico-etcd-secrets

修改完成之后我们直接部署即可使用

$ kubectl apply -f calicoctl-etcd.yaml
pod/calicoctl created

# 创建完成后我们查看calicoctl的运行状态
$ kubectl get pods -A | grep calicoctl
kube-system   calicoctl                                                        1/1     Running   0             9s

# 检验一下是否能够正常工作
$ kubectl exec -ti -n kube-system calicoctl -- /calicoctl get nodes
NAME
tiny-calico-master-88-1.k8s.tcinternal
tiny-calico-worker-88-11.k8s.tcinternal
tiny-calico-worker-88-12.k8s.tcinternal

$ kubectl exec -ti -n kube-system calicoctl -- /calicoctl get profiles -o wide
NAME                                                 LABELS
projectcalico-default-allow
kns.default                                          pcns.kubernetes.io/metadata.name=default,pcns.projectcalico.org/name=default
kns.kube-node-lease                                  pcns.kubernetes.io/metadata.name=kube-node-lease,pcns.projectcalico.org/name=kube-node-lease
kns.kube-public                                      pcns.kubernetes.io/metadata.name=kube-public,pcns.projectcalico.org/name=kube-public
kns.kube-system                                      pcns.kubernetes.io/metadata.name=kube-system,pcns.projectcalico.org/name=kube-system
...此处略去一堆输出...

# 查看ipam的分配情况
$ calicoctl ipam show
+----------+---------------+-----------+------------+--------------+
| GROUPING |     CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+---------------+-----------+------------+--------------+
| IP Pool  | 10.88.64.0/18 |     16384 | 2 (0%)     | 16382 (100%) |
+----------+---------------+-----------+------------+--------------+


# 为了方便可以在bashrc中设置alias
cat >> ~/.bashrc <

完整版本calicoctl命令可以参考官方文档。

5.4 binary安装calicoctl

使用pod方式部署calicoctl虽然简单，但是有个问题就是无法使用calicoctl node命令，这个命令需要访问部分宿主机的文件系统。因此这里我们再二进制部署一个calicoctl。

Note that if you run calicoctl in a container, calicoctl node ... commands will not work (they need access to parts of the host filesystem).

# 直接下线二进制文件即可使用
$ cd /usr/local/bin/
$ curl -L https://github.com/projectcalico/calico/releases/download/v3.22.2/calicoctl-linux-amd64 -o calicoctl
$ chmod +x ./calicoctl

二进制的calicoctl会优先读取配置文件，当找不到配置文件的时候才会去读取环境变量，这里我们直接配置/etc/calico/calicoctl.cfg，注意etcd的证书直接和前面部署calico时使用的证书文件一致即可。

# 配置calicoctl的配置文件
$ mkdir /etc/calico
$ cat /etc/calico/calicoctl.cfg
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: etcdv3
  etcdEndpoints: "https://10.31.88.1:2379"
  etcdCACert: |
      -----BEGIN CERTIFICATE-----
      MIIC9TCCAd2gAwIBAgIBADANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwdldGNk
      LWNhMB4XDTIyMDUwNjA1MTg1OVoXDTMyMDUwMzA1MTg1OVowEjEQMA4GA1UEAxMH
      ZXRjZC1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANFFqq4Mk3DE
      6UW581xnZPFrHqQWlGr/KptEywKH56Bp24OAnDIAkSz7KAMrJzL+OiVsj9YJV59F
      9qH/YzU+bppctDnfk1yCuavkcXgLSd9O6EBhM2LkGtF9AdWMnFw9ui2jNhFC/QXj
      zCvq0I1c9o9gulbFmSHwIw2GLQd7ogO+PpfLsubRscJdKkCUWVFV0mb8opccmXoF
      vXynRX0VW3wpN+v66bD+HTdMSNK1JljfBngh9LAkibjUx7bMrHvu/GOalNCSWrtG
      lss/hhWkzwV7Y7AIXgvxxcmDdfswe5lUYLvW2CP4e+tXfB3i2wg10fErc8z63lix
      v9BWkIIalScCAwEAAaNWMFQwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMB
      Af8wHQYDVR0OBBYEFH49PpnJYxze8aq0PVwgpY4Fo6djMBIGA1UdEQQLMAmCB2V0
      Y2QtY2EwDQYJKoZIhvcNAQELBQADggEBAAGL6KwN80YEK6gZcL+7RI9bkMKk7UWW
      V48154CgN8w9GKvNTm4l0tZKvsWCnR61hiJtLQcG0S8HYHAvL1DBjOXw11bNilLy
      vaVM+wqOOIxPsXLU//F46z3V9z1uV0v/yLLlg320c0wtG+OLZZIn8O+yUhtOHM09
      K0JSAF2/KhtNxhrc0owCTOzS+DKsb0w1SzQmS0t/tflyLfc3oJZ/2V4Tqd72j7iI
      cDBa36lGqtUBf8MXu+Xza0cdhy/f19AqkeM2fe+/DrbzR4zDVmZ7l4dqYGLbKHYo
      XaLn8bSToYQq4dlA/oAlyyH0ekB5v0DyYiHwlqgZgiu4qcR3Gw8azVk=
      -----END CERTIFICATE-----
  etcdCert: |
      -----BEGIN CERTIFICATE-----
      MIIDgzCCAmugAwIBAgIIePiBSOdMGwcwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UE
      AxMHZXRjZC1jYTAeFw0yMjA1MDYwNTE4NTlaFw0yMzA1MDYwNTE4NTlaMDExLzAt
      BgNVBAMTJnRpbnktY2FsaWNvLW1hc3Rlci04OC0xLms4cy50Y2ludGVybmFsMIIB
      IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqZM/jBrdXLR3ctee7LVJhGSA
      4usg/JQXGyOAd52OkkOLYwn3fvwqeo0Z0cX0q4mqaF0cnrPYc4eExX/3fJpF3Fxy
      D6vdpEZ/FrnzCAkibEYtK/UVhTKuV7n/VdbjFPGl8CpppuGVs6o+4NFZxffW7em0
      8m/FK/7SDkV2qXCyG94kOaUCeDEgdBKE3cPCZQ4maFuwXi08bYs2CiTfbfa4dsT5
      3yzaoQVX9BaBqE9IGmsHDFuxp1X8gkJXs+7wwHQX39o1oXmci6T4IVxVHA5GRbTv
      pCDG5Wye7QqKgnxO1KRF42FKs1Nif7UJ0iR35Ydpa7cat7Fr0M7l+rZLCDTJgwID
      AQABo4G9MIG6MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
      KwYBBQUHAwIwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBR+PT6ZyWMc3vGqtD1c
      IKWOBaOnYzBaBgNVHREEUzBRgglsb2NhbGhvc3SCJnRpbnktY2FsaWNvLW1hc3Rl
      ci04OC0xLms4cy50Y2ludGVybmFshwQKH1gBhwR/AAABhxAAAAAAAAAAAAAAAAAA
      AAABMA0GCSqGSIb3DQEBCwUAA4IBAQC+pyH14/+US5Svz04Vi8QIduY/DVx1HOQq
      hfrIZKOZCH2iKU7fZ4o9QpQZh7D9B8hgpXM6dNuFpd98c0MVPr+LesShu4BHVjHl
      gPvUWEVB2XD5x51HqnMV2OkhMKooyAUIzI0P0YKN29SFEyJGD1XDu4UtqvBADqf7
      COvAuqj4VbRgF/iQwNstjqZ47rSzvyp6rIwqFoHRP+Zi+8KL1qmozGjI3+H+TZFM
      Gv3b5DRx2pmfY+kGVLO5bjl3zxylRPjCDHaRlQUWiOYSWS8OHYRCBZuSLvW4tht0
      JjWjUAh4hF8+3lyNrfx8moz7tfm5SG2q01pO1vjkhrhxhINAwaac
      -----END CERTIFICATE-----
  etcdKey: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAqZM/jBrdXLR3ctee7LVJhGSA4usg/JQXGyOAd52OkkOLYwn3
      fvwqeo0Z0cX0q4mqaF0cnrPYc4eExX/3fJpF3FxyD6vdpEZ/FrnzCAkibEYtK/UV
      hTKuV7n/VdbjFPGl8CpppuGVs6o+4NFZxffW7em08m/FK/7SDkV2qXCyG94kOaUC
      eDEgdBKE3cPCZQ4maFuwXi08bYs2CiTfbfa4dsT53yzaoQVX9BaBqE9IGmsHDFux
      p1X8gkJXs+7wwHQX39o1oXmci6T4IVxVHA5GRbTvpCDG5Wye7QqKgnxO1KRF42FK
      s1Nif7UJ0iR35Ydpa7cat7Fr0M7l+rZLCDTJgwIDAQABAoIBAE1gMw7q8zbp4dc1
      K/82eWU/ts/UGikmKaTofiYWboeu6ls2oQgAaCGjYLSnbw0Ws/sLAZQo3AtbOuoj
      ifoBKv9x71nXQjtDL5pfHtX71QkyvEniev9cMNE2vZudgeB8owsDT1ImfPiOJkLP
      Q/dhL2E/0qEM/xskGxUH/S0zjxHHfPZZsYODhkVPWc6Z+XEDll48fRCFn4/48FTN
      9GbRvo7dv34EHmNYA20K4DMHbZUdrPqSZpKWzAPJXnDlgZbpvUeAYOJxqZHQtCm1
      zbSOyM1Ql6K0Ayro0L5GAzap+0yGuk79OWiPnEsdPneVsATKG7dT7RZIL/INrOqQ
      0wjUmQECgYEA02OHdT1K5Au6wtiTqKD99WweltnvFd4C/Z3dobEj8M8qN6uiKCca
      PievWahnxAlJEah3RiOgtarwA+0E/Jgsw99Qutp5BR/XdD3llTNczkPkg/RkWpve
      2f/4DlZQrxuIem7UNLl+5BacfmF691DQQoX2RoIkvQxYJGTUNXvrSUkCgYEAzVyz
      mvN+dvSwzAlm0gkfVP5Ez3DFESUrWd0FR2v1HR6qHQy/dkgkkic6zRGCJtGeT5V7
      N0kbVSHsz+wi6aQkFy0Sp0TbgZzjPhSwNtk+2JsBRvMp0CYczgrfyvWuAQ3gbXGc
      N8IkcZSSOv8TuigCnnYf2Xaz8LM50AivScnb6GsCgYEAyq4ScgnLpa3NawbnRPbf
      qRH6nl7lC01sBqn3mBHVSQ4JB4msF92uHsxEJ639mAvjIGgrvHdqnuT/7nOypVJv
      EXsr14ykHpKyLQUv/Idbw3V7RD3ufqYW3WS8/VorUEoQ6HsdQlRc4ur/L3ndwgWd
      OTtir6YW/aA5XuPCSGnBZekCgYB6VtlgW+Jg91BDnO41/d0+guN3ONUNa7kxpau5
      aqTxHg11lNySmFPBBcHP3LhOa94FxyVKQDEaPEWZcDE0QuaFMALGxwyFYHM3zpdT
      dYQtAdp26/Fi4PGUBYJgpI9ubVffmyjXRr7zMvESWFbmNWOqBvDeWgrEP+EW/7V9
      HdX11QKBgE1czchlibgQ/bhAl8BatKRr1X/UHvblWhmyApudOfFeGOILR6u/lWvY
      SS+Rg0y8nnZ4hTRSXbd/sSEsUJcSmoBc1TivWzl32eVuqe9CcrUZY0JSLtoj1KiP
      adRcCZtVDETXbW326Hvgz+MnqrIgzx+Zgy4tNtoAAbTv0q83j45I
      -----END RSA PRIVATE KEY-----

配置完成之后我们检查一下效果

$ calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 10.31.88.11  | node-to-node mesh | up    | 08:26:30 | Established |
| 10.31.88.12  | node-to-node mesh | up    | 08:26:30 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

$ calicoctl get nodes
NAME
tiny-calico-master-88-1.k8s.tcinternal
tiny-calico-worker-88-11.k8s.tcinternal
tiny-calico-worker-88-12.k8s.tcinternal

$ calicoctl ipam show
+----------+---------------+-----------+------------+--------------+
| GROUPING |     CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+---------------+-----------+------------+--------------+
| IP Pool  | 10.88.64.0/18 |     16384 | 2 (0%)     | 16382 (100%) |
+----------+---------------+-----------+------------+--------------+

6、部署测试用例

集群部署完成之后我们在k8s集群中部署一个nginx测试一下是否能够正常工作。首先我们创建一个名为nginx-quic的命名空间（namespace），然后在这个命名空间内创建一个名为nginx-quic-deployment的deployment用来部署pod，最后再创建一个service用来暴露服务，这里我们先使用nodeport的方式暴露端口方便测试。

$ cat nginx-quic.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-quic

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-quic-deployment
  namespace: nginx-quic
spec:
  selector:
    matchLabels:
      app: nginx-quic
  replicas: 4
  template:
    metadata:
      labels:
        app: nginx-quic
    spec:
      containers:
      - name: nginx-quic
        image: tinychen777/nginx-quic:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-quic-service
  namespace: nginx-quic
spec:
  selector:
    app: nginx-quic
  ports:
  - protocol: TCP
    port: 8080 # match for service access port
    targetPort: 80 # match for pod access port
    nodePort: 30088 # match for external access port
  type: NodePort

部署完成后我们直接查看状态

# 直接部署
$ kubectl apply -f nginx-quic.yaml
namespace/nginx-quic created
deployment.apps/nginx-quic-deployment created
service/nginx-quic-service created

# 查看deployment的运行状态
$ kubectl get deployment -o wide -n nginx-quic
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                          SELECTOR
nginx-quic-deployment   4/4     4            4           55s   nginx-quic   tinychen777/nginx-quic:latest   app=nginx-quic

# 查看service的运行状态
$ kubectl get service -o wide -n nginx-quic
NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE   SELECTOR
nginx-quic-service   NodePort   10.88.52.168           8080:30088/TCP   66s   app=nginx-quic

# 查看pod的运行状态
$ kubectl get pods -o wide -n nginx-quic
NAME                                     READY   STATUS    RESTARTS   AGE   IP             NODE                                      NOMINATED NODE   READINESS GATES
nginx-quic-deployment-7457f4d579-24q9z   1/1     Running   0          75s   10.88.120.72   tiny-calico-worker-88-12.k8s.tcinternal              
nginx-quic-deployment-7457f4d579-4svv9   1/1     Running   0          75s   10.88.84.68    tiny-calico-worker-88-11.k8s.tcinternal              
nginx-quic-deployment-7457f4d579-btrjj   1/1     Running   0          75s   10.88.120.71   tiny-calico-worker-88-12.k8s.tcinternal              
nginx-quic-deployment-7457f4d579-lvh6x   1/1     Running   0          75s   10.88.84.69    tiny-calico-worker-88-11.k8s.tcinternal              


# 查看IPVS规则
$ ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.17.0.1:30088 rr
  -> 10.88.84.68:80               Masq    1      0          0
  -> 10.88.84.69:80               Masq    1      0          0
  -> 10.88.120.71:80              Masq    1      0          0
  -> 10.88.120.72:80              Masq    1      0          0
TCP  10.31.88.1:30088 rr
  -> 10.88.84.68:80               Masq    1      0          0
  -> 10.88.84.69:80               Masq    1      0          0
  -> 10.88.120.71:80              Masq    1      0          0
  -> 10.88.120.72:80              Masq    1      0          0
TCP  10.88.52.168:8080 rr
  -> 10.88.84.68:80               Masq    1      0          0
  -> 10.88.84.69:80               Masq    1      0          0
  -> 10.88.120.71:80              Masq    1      0          0
  -> 10.88.120.72:80              Masq    1      0          0

最后我们进行测试，这个nginx-quic的镜像默认情况下会返回在nginx容器中获得的用户请求的IP和端口

# 首先我们在集群内进行测试
# 直接访问pod
$ curl 10.88.84.68:80
10.31.88.1:34612
# 直接访问service的ClusterIP，这时请求会被转发到pod中
$ curl 10.88.52.168:8080
10.31.88.1:58978
# 直接访问nodeport，这时请求会被转发到pod中，不会经过ClusterIP
$ curl 10.31.88.1:30088
10.31.88.1:56595

# 接着我们在集群外进行测试
# 直接访问三个节点的nodeport，这时请求会被转发到pod中，不会经过ClusterIP
# 由于externalTrafficPolicy默认为Cluster，因此nginx拿到的IP就是我们访问的节点的IP，而非客户端IP
$ curl 10.31.88.1:30088
10.31.88.1:27851
$ curl 10.31.88.11:30088
10.31.88.11:16540
$ curl 10.31.88.12:30088
10.31.88.12:5767