Aliyun ECS 自建 K8S 集群(生产环境)

建议使用阿里云自身的容器服务吧，毕竟自建k8s集群太折腾

1、资源架构

前期使用 2 master + 3 worker配置，后期再扩展worker节点

资源、主机名	IP	配置	服务
SLB1-ALB	172.18.20.55	内网基础版	master VIP
SLB2		外网IP，应用服务	ingress
k8s-master1	172.18.20.44	ecs 4VCPU,16G + 80G	ETCD,API server, Controller Manager, Scheduler
k8s-master2	172.18.20.45	ecs 4VCPU,16G + 80G	ETCD,API server, Controller Manager, Scheduler
k8s-master3	172.18.20.46	ecs 4VCPU,16G + 80G	ETCD,API server, Controller Manager, Scheduler
k8s-worker1	172.18.20.47	ecs 8vCPU,32G + 80G	kubelet, kube-proxy, docker, fluentd
k8s-worker2	172.18.20.48	ecs 8vCPU,32G + 80G	kubelet, kube-proxy, docker, fluentd
k8s-worker3	172.18.20.49	ecs 8vCPU,32G + 80G	kubelet, kube-proxy, docker, fluentd

架构图

堆叠（Stacked） etcd 拓扑

堆叠的 etcd 拓扑

外部 etcd 拓扑

外部 etcd 拓扑

2、基础环境

2.1、软件版本

由于kubernetes 1.20 版本开始已弃用docker，推荐使用containerd

软件	版本
os	CentOS 7.9
containerd	1.4.3
Kubernetes	1.21
etcd	3.4

2.2、系统配置

没有特别说明的，所有主机需执行

• 修改hostname

hostname k8s-master1

• 关闭防火墙

# systemctl stop firewalld
# systemctl disable firewalld

• 关闭selinux

# sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
# setenforce 0  # 临时

• 关闭 swap

# swapoff -a  # 临时
# sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

• 添加 hosts

# cat >> /etc/hosts << EOF
172.18.20.44 k8s-master1
172.18.20.45 k8s-master2
172.18.20.46 k8s-master3
...
EOF

• 配置节点主机免密访问(master)

# ssh-keygen 
# ssh-copy-id root@k8s-*

• 将桥接的IPv4流量传递到iptables的链

# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# sysctl --system 

• 调整系统内核参数

  # cat > /etc/sysctl.d/kubernetes.conf <

• 错误提示

  /proc/sys/net/bridge/bridge-nf-call-iptables: 没有那个文件或目录
  

  记得运行 modprobe br_netfilter

• 加载ipvs 模块

  cat > /etc/sysconfig/modules/ipvs.modules <

3、etcd 集群部署

k8s集群使用堆叠 ETCD 时，可不单独创建，本人是为了延长 etcd 证书才单独部署集群，实际架构也是堆叠式

3.1 使用 etcdadm 工具

下载

# wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.3/etcdadm-linux-amd64
# mv etcdadm-linux-amd64 /usr/local/bin/etcdadm
# chmod +x /usr/local/bin/etcdadm

选择一个节点初始化etcd

etcdadm init --install-dir="/opt/etcd/" --name=etcd-1

参数解析

• --install-dir 安装目录
• --name 节点名称
• server-cert-extra-sans 负载均衡地址（单独部署使用）

拷贝证书到其他节点

scp /etc/etcd/pki/ca.* root@master2:/etc/etcd/pki/
scp /etc/etcd/pki/ca.* root@master3:/etc/etcd/pki/

加入集群

 etcdadm join https://172.18.20.44:2379 --install-dir="/opt/etcd/" --name=etcd-2

安装完成后查看集群

修改环境变量

source /etc/etcd/etcdctl.env

查看

# /opt/bin/etcdctl member list 
18ed9897779ce358, started, k8s-master1, https://172.18.20.50:2380, https://172.18.20.50:2379, false
3e04e89693dd5c26, started, k8s-master2, https://172.18.20.51:2380, https://172.18.20.51:2379, false
68562c67a29d6f06, started, k8s-master3, https://172.18.20.49:2380, https://172.18.20.49:2379, false

建议直接使用 etcdctl.sh

注意：etcdadm 创建的证书期限为一年。需要一年更换一次，这对生产环境来说很不友好。目前有两个解决方案

• 1、二进制部署 etcd，使用openssl 生成证书

• 2、修改etcdadm源码

  下载源码

  git clone https://github.com/kubernetes-sigs/etcdadm.git
  

修改 certs/pkiutil/pki_helpers.go 文件

certificateValidity =time.Hour * 24 * 365 * 10 # 10年期限

const (
        // PrivateKeyBlockType is a possible value for pem.Block.Type.
        PrivateKeyBlockType = "PRIVATE KEY"
        // PublicKeyBlockType is a possible value for pem.Block.Type.
        PublicKeyBlockType = "PUBLIC KEY"
        // CertificateBlockType is a possible value for pem.Block.Type.
        CertificateBlockType = "CERTIFICATE"
        // RSAPrivateKeyBlockType is a possible value for pem.Block.Type.
        RSAPrivateKeyBlockType = "RSA PRIVATE KEY"
        rsaKeySize             = 2048
        certificateValidity    = time.Hour * 24 * 365 * 10
)

编译（需要golang 1.15 以上版本）

make

  编译完成后使用 etcdadm 重新生成集群

3.2、二进制部署

太麻烦了，网上找资料吧。ectdadm非常nice

4、安装 docker 和 containerd (所有节点)

1.20 版本模式使用 containerd ，可不用安装docker

添加 yum 源

# yum install -y yum-utils
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装

yum install -y containerd.io

生成默认配置

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

添加私有镜像仓库

containerd 修改 config.toml 配置

[plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    # 阿里云私有镜像仓库
        [plugins."io.containerd.grpc.v1.cri".cri.registry.mirrors."registry-vpc.cn-shenzhen.aliyuncs.com"]
          endpoint = ["http://registry-vpc.cn-shenzhen.aliyuncs.com"]

      [plugins."io.containerd.grpc.v1.cri".registry.auths]
        [plugins."io.containerd.grpc.v1.cri".registry.auths."registry-vpc.cn-shenzhen.aliyuncs.com"]
          username = "username"
          password = "password

启动

systemctl restart containerd
systemctl enable containerd

5、使用kubeadm部署高可用集群

5.1、安装kubeadm

yum源

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装

yum install -y kubeadm kubelet kubectl

无法使用kubeadm启动kubelet时，需要修改 kubelet 使用 container

# vim /usr/lib/systemd/system/kubelet.services.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

5.2、初始化集群

kubeadm默认会创建一个堆叠式的etcd集群，并不需要另外部署

kubeadm init --control-plane-endpoint "k8s-master-slb:6443" --upload-certs --node-name "k8s-master1"

参数解析

• --control-plane-endpoint apiserver集群地址
• --upload-certs 证书

由于国内无法访问google的镜像地址，这里需要使用国内的地址来下载镜像，下载完成后需要更新 tag 为 k8s.gcr.io

5.3、使用外部 etcd 初始化集群

创建 kubeadm 初始化配置文件

# 生成 kubeadm 默认初始化模板
kubeadm config print init-defaults > kubeadm-config.yaml

kubeadm-config 使用说明 https://blog.51cto.com/foxhound/2517491?source=dra

修改

cat > kubeadm-config.yaml <

下载镜像

kubeadm config images pull --config kubeadm-config.yaml

下载镜像脚本

#!/bin/bash

images=(
kube-apiserver:v1.20.4
kube-controller-manager:v1.20.4
kube-scheduler:v1.20.4
kube-proxy:v1.20.4
pause:3.2
etcd:3.4.13-0
coredns:1.7.0)

for image in ${images[@]}; do
        # docker
        #docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
        #docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
        #docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/${image}
        # cri
        ctr -n k8s.io i tag registry.cn-hangzhou.aliyuncs.com/google_containers/${image} k8s.gcr.io/${image}
done

初始化

kubeadm init --config kubeadm-config.yaml  --upload-certs

初始化成功会出现以下信息

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
    --control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 

然后集群二和集群三 运行命令加入

kubeadm join k8s-master-slb:7443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:138388af441152652765f8b1959e39db63c97049c3408a61c1a60cac5c8d8256 \
    --control-plane --certificate-key bacab8cd43592812f0e3a186aaa615463c87e9280c0e2ae951b54b138325537d

查看集群

# kubectl get nodes
NAME          STATUS     ROLES                  AGE     VERSION
k8s-master1   NotReady   control-plane,master   11m     v1.20.5
k8s-master2   NotReady   control-plane,master   8m54s   v1.20.5
k8s-master3   NotReady   control-plane,master   8m31s   v1.20.5

由于还未部署网络插件（cni）,状态还是 NotReady

• 注意：使用阿里云 SLB 时，由于apiserver 还未启动，所以此时 6443 端口并未监听，使用 SLB地址时会无法安装成功。修改成当前服务器地址安装成功后再切换 HOST 地址为 SLB地址

  本机地址 172.18.20.44
  负载均衡地址： 172.18.20.55
  172.18.20.44 k8s-master-slb
  

• 注意：token有效期为24小时，失效后请在主节点使用以下命令重新生成

kubeadm token create --print-join-command

5.4、部署工作节点

安装 containerd ,kubeadm,kubelet,kubectl 等

工作节点运行 join

kubeadm join k8s-master-slb:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:12b4fe0053bafd6b1b0e05482912b44ddcf88d1d1429e3c611d109ad5bf93ac0

master 查看节点

# kubectl get nodes
NAME          STATUS     ROLES                  AGE     VERSION
k8s-master1   Ready      control-plane,master   6m28s   v1.20.5
k8s-master2   NotReady   control-plane,master   3m25s   v1.20.5
k8s-master3   NotReady   control-plane,master   3m11s   v1.20.5
k8s-worker1   NotReady                    2m28s   v1.20.5
k8s-worker2   NotReady                    4s      v1.20.5

由于没有部署 CNI 网络插件，状态还是 NotReady

5.5、部署 CNI 网络插件

master 节点运行

需确保kube-flannel.yml文件里的 "Network": "10.10.0.0/16"IP内容与 kube-controller-manager.conf 配置的 --cluster-cidr 一致

即 kubeadm-config 里的 podSubnet

# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# kubectl apply -f kube-flannel.yml
# kubectl get pods -n kube-system

再次查看节点

# kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
k8s-master1   Ready    control-plane,master   18h   v1.20.5
k8s-master2   Ready    control-plane,master   18h   v1.20.5
k8s-master3   Ready    control-plane,master   18h   v1.20.5
k8s-worker1   Ready                     18h   v1.20.5
k8s-worker2   Ready                     18h   v1.20.5

5.6、Kubelet驱逐策略优化

修改工作节点kubelet启动参数，更改Pod驱逐策略

vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 
Environment="EVICTION_HARD=--eviction-hard=memory.available<2Gi,nodefs.available<5Gi,imagefs.available<100Gi"
Environment="EVICTION_RECLAIM=--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=1Gi,imagefs.available=2Gi"

重启kubelet容器，并查看kubelet进程启动参数

systemctl daemon-reload && systemctl restart kubelet

6、单独部署coredns

不依赖kubeadm的方式，适用于不是使用kubeadm创建的k8s集群，或者kubeadm初始化集群之后，删除了dns相关部署。

# 在calico网络中也配置一个coredns # 10.96.0.10 为k8s官方指定的kube-dns地址

mkdir coredns && cd coredns
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
chmod +x deploy.sh
./deploy.sh -i 10.96.0.10 > coredns.yml

kubectl apply -f coredns.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

7、Kubernetes 云管理控制器

自建 k8s 想要使用阿里云的负载均衡、存储等服务时，需要部署阿里云提供的组件 cloud-controller-manager

github：https://github.com/kubernetes/cloud-provider-alibaba-cloud

帮助文档：https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/docs/getting-started.md

7.1、安装Alibaba CloudProvider 组件

修改 kubelet 启动服务

修改kubelet 启动参数，添加 --cloud-provider=external，并且在kubelet中添加

--hostname-override=${REGION_ID}.${INSTANCE_ID} --provider-id=${REGION_ID}.${INSTANCE_ID}

获取 region-id 和 instance-id

echo `curl -s http://100.100.100.200/latest/meta-data/region-id`.`curl -s http://100.100.100.200/latest/meta-data/instance-id`

# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=external --hostname-override=cn-shenzhen.xxxxxxx --provider-id=cn-shenzhen.xxxxxxx

配置阿里云 AccessKeyID，AccessKeySecret

AccessKey & AccessKeySecret 必须以 base64 方式

# base64 AccessKey & AccessKeySecret
$ echo -n "$AccessKeyID" |base64
$ echo -n "$AcceessKeySecret"|base64

$ cat <cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  cloud-config.conf: |-
    {
        "Global": {
            "accessKeyID": "$your-AccessKeyID-base64",
            "accessKeySecret": "$your-AccessKeySecret-base64"
        }
    }
EOF

$ kubectl create -f cloud-config.yaml

添加 kubeconfig 配置文件(所有master节点)

vim /etc/kubernetes/cloud-controller-manager.conf

kind: Config
contexts:
- context:
    cluster: alw-cluster
    user: system:cloud-controller-manager
  name: system:cloud-controller-manager@alw-cluster
current-context: system:cloud-controller-manager@alw-cluster
users:
- name: system:cloud-controller-manager
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: $CA_DATA
    server: https://172.18.20.44:6443
  name: alw-cluster

$CA_DATA 可以使用命令 cat /etc/kubernetes/pki/ca.crt|base64 -w 0 获取。

server： 修改为本机IP

编写 cloud-controller-manager.yaml 文件

也可以使用官方推荐的更详细配置 cloud-controller-manager.yml

# vim cloud-controller-manager.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: cloud-controller-manager
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: cloud-controller-manager
    tier: control-plane
  name: cloud-controller-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cloud-controller-manager
      tier: control-plane
  template:
    metadata:
      labels:
        app: cloud-controller-manager
        tier: control-plane
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: cloud-controller-manager
      tolerations:
        - effect: NoSchedule
          operator: Exists
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          operator: Exists
          key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
        node-role.kubernetes.io/master: ""
      containers:
        - command:
          -  /cloud-controller-manager
          - --kubeconfig=/etc/kubernetes/cloud-controller-manager.conf
          - --address=127.0.0.1
          - --allow-untagged-cloud=true
          - --leader-elect=true
          - --cloud-provider=alicloud     # Add your own cloud provider here!
          - --use-service-account-credentials=true
          - --cloud-config=/etc/kubernetes/config/cloud-config.conf
          - --configure-cloud-routes=true
          - --allocate-node-cidrs=true
          - --route-reconciliation-period=3m
          # replace ${cluster-cidr} with your own cluster cidr
          - --cluster-cidr=10.10.0.0/16
          image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.339-g9830b58-aliyun
          livenessProbe:
            failureThreshold: 8
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10258
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 15
          name: cloud-controller-manager
          resources:
            requests:
              cpu: 200m
          volumeMounts:
            - mountPath: /etc/kubernetes/
              name: k8s
            - mountPath: /etc/ssl/certs
              name: certs
            - mountPath: /etc/pki
              name: pki
            - mountPath: /etc/kubernetes/config
              name: cloud-config
      hostNetwork: true
      volumes:
        - hostPath:
            path: /etc/kubernetes
          name: k8s
        - hostPath:
            path: /etc/ssl/certs
          name: certs
        - hostPath:
            path: /etc/pki
          name: pki
        - configMap:
            defaultMode: 420
            items:
              - key: cloud-config.conf
                path: cloud-config.conf
            name: cloud-config
          name: cloud-config

配置解析

• --cloud-provider=alicloud 云服务商
• --cluster-cidr 集群 pod 地址

运行

# kubectl apply -f cloud-controller-manager.yaml
# kubectl get pods -n kube-system 
NAME                                  READY   STATUS    RESTARTS   AGE
cloud-controller-manager-7jbzc        1/1     Running   0          109m
cloud-controller-manager-dfpkv        1/1     Running   0          109m
cloud-controller-manager-lqvtz        1/1     Running   4          109m

完成组件的部署后，接下来就可以使用阿里云的负载均衡了

8、部署 Ingress-nginx

安装 ingress-nginx 控制器

下载：

wget  https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/cloud/deploy.yaml

国内无法访问 google 的镜像仓库，需自己推一个到阿里云镜像仓库并设置为公有

# docker pull k8s.gcr.io/ingress-nginx/controller:v0.45.0
# docker tag k8s.gcr.io/ingress-nginx/controller:v0.45.0 registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
# docker login --username=**** --password=**** registry.cn-shenzhen.aliyuncs.com
# docker push registry.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0

修改配置

...
apiVersion: apps/v1
kind: Deployment
    ……
        # 国内无法访问 google 的镜像仓库，需自己推一个到阿里云镜像仓库并设置为公有
        image: registry-vpc.cn-shenzhen.aliyuncs.com/anlewo/ingress-nginx-controller:v0.45.0
    ……

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-3.27.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.45.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
  annotations:
    # 指明SLB实例地址类型为私网类型
    # service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
    # 修改为您的私网SLB实例ID
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: *********-cn-shenzhen-st3-a01
    # 是否自动创建SLB端口监听（会覆写已有端口监听），也可手动创建端口监听
    #service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: 'true'
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

运行

# kubectl apply -f deploy.yaml

9、部署Aliyun存储插件 csi

9.1、官方ack csi插件

阿里云官方文档：https://help.aliyun.com/document_detail/134722.html?spm=a2c4g.11186623.6.822.7c525ccfsHWlPe

github：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver

RBAC 安装：

下载RBAC配置文件到操作机，并部署：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/rbac.yaml

执行：

$ kubectl apply -f rbac.yaml

CSI-Plugin 安装：

下载普通模版

1. 下载模板：

下载最新版本的CSI Plugin部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-plugin.yaml

将部署模板下载到您的操作机，并保存（csi-plugin.yaml）。

2. 适配模板并部署：

根据集群所在的Region修改模板中的镜像地址。例如：如果是cn-beijing的集群

则将 registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0 中的：

registry 改为 registry-vpc

cn-hangzhou 改为 cn-shenzhen

即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0

模板中的其他镜像也是如此更新；

执行部署：

$ kubectl apply -f csi-plugin.yaml

3. 检查安装情况：

$ kubectl get pod -nkube-system | grep csi-plugin

$ kubectl describe ds csi-plugin -nkube-system | grep Image

CSI-Provisioner 安装：

1. 下载模板：

下载最新版本的CSI Provisioner部署模板：https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/deploy/ack/csi-provisioner.yaml

将部署模板下载到您的操作机，并保存（csi-provisioner.yaml）。

2. 适配模板并部署：

根据集群所在的Region修改模板中的镜像地址。例如：如果是cn-beijing的集群

则将 registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun 中的：

registry 改为 registry-vpc

cn-hangzhou 改为 cn-shenzhen

即：registry-vpc.cn-shenzhen.aliyuncs.com/acs/csi-provisioner:v1.6.0-e360c7e43-aliyun

模板中的其他镜像也是如此更新；

执行部署：

$ kubectl apply -f csi-provisioner.yaml

3. 检查安装情况：

$ kubectl get pod -nkube-system | grep csi-provisioner

$ kubectl describe deploy csi-provisioner -nkube-system | grep Image

10、部署Dashboard

下载部署文件：

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.0/aio/deploy/recommended.yaml

默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：

# vim recommended.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

部署：

# kubectl apply -f recommended.yaml
# kubectl get pods,svc -n kubernetes-dashboard
NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-6b4884c9d5-mjl66   1/1     Running   0          23h
pod/kubernetes-dashboard-7bfbb48676-frmsf        1/1     Running   0          23h

NAME                                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
service/dashboard-metrics-scraper   ClusterIP   10.0.0.173           8000/TCP        23h
service/kubernetes-dashboard        NodePort    10.0.0.145           443:30001/TCP   23h

访问地址：https://NodeIP:30001

创建service account并绑定默认cluster-admin管理员集群角色：

kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

查看 token：

kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

使用输出的 token 登录 Dashboard

结语

至此，一个生成环境可用的，基于阿里云ecs的k8s集群就搭建完成。但集群的搭建只是基础，后续的维护使用才是重点，包括prometheus监控，istio等