kubeflow1.3安装记录-外网环境下
kubeflow安装在能访问外网的情况下十分简单，但是如果不能就会很困难很多，下面首先在能访问外网的情况下进行，再试着在局域网（公司网络）内进行安装。这里机器使用腾讯云服务器，选择2台香港地区的竞价实例，CentOS7.6，配置为8核16G，根目录磁盘空间300G，网络为1Mbps的按带宽计费模式，2台机器共计每小时￥1.23。安全组选择放通所有端口，登录方式为自定义密码登录
安装docker
首先在两台机器上安装docker
yum -y install yum-utils && \
yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo && \
yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.4.3-3.1.el7.x86_64.rpm && \
yum install docker-ce -y
systemctl --now enable docker
yum install -y yum-utils device-mapper-persistent-data lvm2 git如果因为网络问题无法使用yum安装的话，可以到https://download.docker.com/linux/static/stable/x86_64/ 下载tar.gz文件，解压后将docker/目录下全部文件复制到 /usr/bin 目录下，创建启动文件 /etc/systemd/system/docker.service ，内容如下
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd --selinux-enabled=false
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target执行 systemctl daemon-reload && systemctl restart docker 完成docker的安装
PS：默认情况下，docker会将镜像文件存放在 /var 目录下，如果该目录空间不足，docker会自动删除某些镜像，可以在启动中指定 --graph=/root/images 来自定义镜像文件存放的位置
安装Harbor
在kubeflow的使用过程中，我们需要一个私有的镜像仓库来存放执行过程中生成的镜像文件，这里我们选择部署一套harbor作为镜像仓库，也可以选择docker registry，选择一台机器操作即可
安装docker-compose：
wget https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
mv docker-compose-Linux-x86_64 docker-compose
chmod +x docker-compose
mv docker-compose /usr/bin/可执行 docker-compose version 验证是否安装成功
wget https://github.com/goharbor/harbor/releases/download/v1.9.2/harbor-offline-installer-v1.9.2.tgz
tar -zxvf harbor-offline-installer-v1.9.2.tgz
cd harbor
vim harbor.yaml
./install.sh修改harbor.yaml中的hostname和port
访问 http://43.128.14.116:86/，即可打开harbor页面，43.128.14.116为服务器外网地址、172.19.0.14为其内网地址，默认账号admin，密码为 Harbor12345。登录之后创建一个名为kubeflow的公开项目作为我们的私有仓库。
修改docker配置文件，/etc/docker/daemon.json，加入如下内容，之后重启docker
{ "insecure-registries": ["172.19.0.14:86"] }
docker login -u admin -p Harbor12345 172.19.0.14:86
harbor搭建完成 ，docker-compose up -d 重启
安装Kubernetes
使用kubeadm来安装k8s，kubeflow1.3要求k8s的版本最低为1.15，推荐1.17+，这里选择1.19版本进行安装
sudo setenforce 0 #关闭selinux
sudo swapoff -a #关闭内存交换
#开启ipvs
modprobe br_netfilter
cat > /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs
#更新yum源，创建 /etc/yum.repos.d/kubernetes.repo（所有节点）
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum makecache fast -y
#调整iptables
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
#修改环境变量/etc/profile,加入
export KUBECONFIG=/etc/kubernetes/admin.conf
export GODEBUG=x509ignoreCN=0
source /etc/profile
yum install -y kubelet-1.19.1 kubeadm-1.19.1 kubectl-1.19.1
systemctl enable kubelet && systemctl restart kubelet选择一台机器作为k8s集群master，进行初始化设置
kubeadm config print init-defaults > kubeadm-config.yaml
编辑 kubeadm-config.yaml ，修改advertiseAddress为本机内网ip，在 networking 模块下加入
podSubnet: 10.244.0.0/16
kubeadm init --config=kubeadm-config.yaml --upload-certs |tee kubeadm-init.log
在另一台机器上执行 kubeadm join 加入集群，之后进行k8s集群的配置工作
#配置flannel
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml
#重启pod
kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'
#使master也能当做node使用
kubectl taint node vm-0-14-centos node-role.kubernetes.io/master-修改k8s配置文件，进入 /etc/kubernetes/manifests/
修改 kube-apiserver.yaml，加入如下内容
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-account-issuer=kubernetes.default.svc
修改 kube-controller-manager.yaml 、kube-scheduler.yaml，将 - --port=0 注释掉
重启kubelet：systemctl restart kubelet
设置NFS、创建PV
在安装kubeflow之前，首先设置nfs并创建pv，安装kubeflow时mysq、katib等组件时需要使用pvc，如果不提前创建，会导致这些节点状态为pending
yum install -y nfs-utils rpcbind 选择一台机器作为nfs服务器
#创建共享文件夹
mkdir /root/nfs-kubeflow
cd /root/nfs-kubeflow
mkdir v1
mkdir v2
mkdir v3
mkdir v4
mkdir v5
#配置nfs
vim /etc/exports
#加入如下内容:/root/nfs-kubeflow *(insecure,rw,no_root_squash,no_all_squash,sync)
#使配置生效
exportfs -r
#启动服务rpcbind、nfs服务
service rpcbind start
service nfs start 
配置pv文件 pv.yaml，加入如下内容，注意替换path与server
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv001
labels:
name: pv001
spec:
nfs:
path: /root/nfs-kubeflow/v1
server: 172.19.0.14
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 15Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv002
labels:
name: pv002
spec:
nfs:
path: /root/nfs-kubeflow/v2
server: 172.19.0.14
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 25Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv003
labels:
name: pv003
spec:
nfs:
path: /root/nfs-kubeflow/v3
server: 172.19.0.14
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 25Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv004
labels:
name: pv004
spec:
nfs:
path: /root/nfs-kubeflow/v4
server: 172.19.0.14
accessModes: ["ReadWriteMany","ReadWriteOnce"]
capacity:
storage: 25Gi执行 kubectl apply -f pv.yaml 创建 pv
安装kubeflow
wget https://github.com/kubeflow/manifests/archive/refs/tags/v1.3.0.zip
unzip v1.3.0.zip
wget https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
mv kustomize_3.2.0_linux_amd64 kustomize
chmod +x kustomize
mv kustomize /usr/bin/cd manifests-1.3.0/
#安装istio
kustomize build common/istio-1-9-0/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-9-0/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-9-0/istio-install/base | kubectl apply -f -
#安装cert-manager
kustomize build common/cert-manager/cert-manager-kube-system-resources/base | kubectl apply -f -
kustomize build common/cert-manager/cert-manager-crds/base | kubectl apply -f -
kustomize build common/cert-manager/cert-manager/overlays/self-signed | kubectl apply -f -
#安装dex
kustomize build common/dex/overlays/istio | kubectl apply -f -
#安装oidc
kustomize build common/oidc-authservice/base | kubectl apply -f -
#安装knative-serving
kustomize build common/knative/knative-serving-crds/base | kubectl apply -f -
kustomize build common/knative/knative-serving-install/base | kubectl apply -f -
kustomize build common/istio-1-9-0/cluster-local-gateway/base | kubectl apply -f -
#安装knative-eventing
kustomize build common/knative/knative-eventing-crds/base | kubectl apply -f -
kustomize build common/knative/knative-eventing-install/base | kubectl apply -f -
#创建kubeflow namespace
kustomize build common/kubeflow-namespace/base | kubectl apply -f -
#创建kubeflow-roles
kustomize build common/kubeflow-roles/base | kubectl apply -f -
#创建istio-resource
kustomize build common/istio-1-9-0/kubeflow-istio-resources/base | kubectl apply -f -
#安装pipeline
kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user | kubectl apply -f -
#安装kfserving
kustomize build apps/kfserving/upstream/overlays/kubeflow | kubectl apply -f -
#安装katib
kustomize build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -
#安装kubeflow dashboard
kustomize build apps/centraldashboard/upstream/overlays/istio | kubectl apply -f -
#安装admission-webhook
kustomize build apps/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -
#安装notebook
kustomize build apps/jupyter/notebook-controller/upstream/overlays/kubeflow | kubectl apply -f -
#安装jupyter
kustomize build apps/jupyter/jupyter-web-app/upstream/overlays/istio | kubectl apply -f -
#安装kfam
kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -
#安装volume
kustomize build apps/volumes-web-app/upstream/overlays/istio | kubectl apply -f -
#安装tensorboard
kustomize build apps/tensorboard/tensorboards-web-app/upstream/overlays/istio | kubectl apply -f -
kustomize build apps/tensorboard/tensorboard-controller/upstream/overlays/kubeflow | kubectl apply -f -
#安装各种operator
kustomize build apps/tf-training/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/pytorch-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/mpi-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/mxnet-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/xgboost-job/upstream/overlays/kubeflow | kubectl apply -f -
#创建namespace
kustomize build common/user-namespace/base | kubectl apply -f -
等待所有pod均为running状态
暴露指定端口30000，编辑kubeflow-ui-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: istio-ingressgateway
install.operator.istio.io/owning-resource: unknown
istio: ingressgateway
istio.io/rev: default
operator.istio.io/component: IngressGateways
release: istio
name: istio-ingressgateway
namespace: istio-system
spec:
ports:
- name: status-port
port: 15021
protocol: TCP
targetPort: 15021
- name: http2
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30000
- name: https
port: 443
protocol: TCP
targetPort: 8443
- name: tcp
port: 31400
protocol: TCP
targetPort: 31400
- name: tls
port: 15443
protocol: TCP
targetPort: 15443
selector:
app: istio-ingressgateway
istio: ingressgateway
type: NodePort执行 kubectl apply -f kubeflow-nodeport.yaml
到这里kubeflow1.3的部署就完成了，可通过 http://ip:30000 来访问，默认账号为 [email protected]，密码为12341234
设置 storageClass
默认情况下，创建notebook server会报如下错误，原因是没有配置默认的StorageClass
编辑storage-nfs.yaml，注意替换NFS_SERVER、NFS_PATH内容
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-client-provisioner
containers:
- name: nfs-provisioner
image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: kubeflow/nfs
- name: NFS_SERVER
value: 172.19.0.14
- name: NFS_PATH
value: /root/nfs-kubeflow/v5
volumes:
- name: nfs-client-root
nfs:
server: 172.19.0.14
path: /root/nfs-kubeflow/v5编辑storage-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: kubeflow-user-example-com
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]编辑storage-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: kubeflow-nfs-storage
provisioner: kubeflow/nfs依次执行如下命令
kubectl apply -f storage-nfs.yaml
kubectl apply -f storage-rbac.yaml
kubectl apply -f storage-class.yaml
kubectl patch storageclass kubeflow-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'刷新页面之后，不再报错
设置Https访问
如果要通过 NodePort / LoadBalancer / Ingress 暴露服务到非 localhost 网络，那么必须使用 https。否则能打开主页面但是 notebook、ssh等均无法连接。创建生成 ssl 自签名证书脚本 create_self-signed-cert.sh
#!/bin/bash -e
help ()
{
echo ' ================================================================ '
echo ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
echo ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'
echo ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
echo ' --ssl-size: ssl加密位数，默认2048；'
echo ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
echo ' 使用示例:'
echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
echo ' ================================================================'
}
case "$1" in
-h|--help) help; exit;;
esac
if [[ $1 == '' ]];then
help;
exit;
fi
CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
case "$key" in
--ssl-domain) SSL_DOMAIN=$value ;;
--ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
--ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
--ssl-size) SSL_SIZE=$value ;;
--ssl-date) SSL_DATE=$value ;;
--ca-date) CA_DATE=$value ;;
--ssl-cn) CN=$value ;;
esac
done
# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca
# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}
## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}
SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt
echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m | 生成 SSL Cert | \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"
if [[ -e ./${CA_KEY} ]]; then
echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
mv ${CA_KEY} "${CA_KEY}"-bak
openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi
if [[ -e ./${CA_CERT} ]]; then
echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
mv ${CA_CERT} "${CA_CERT}"-bak
openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi
echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <> ${SSL_CONFIG} <> ${SSL_CONFIG}
done
if [[ -n ${SSL_TRUSTED_IP} ]]; then
ip=(${SSL_TRUSTED_IP})
for i in "${!ip[@]}"; do
echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
done
fi
fi
echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}
echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}
echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
-CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
-days ${SSL_DATE} -extensions v3_req \
-extfile ${SSL_CONFIG}
echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/ /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/ /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/ /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/ /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/ /'
echo
echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/ /'
echo
echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt chmod +x create_self-signed-cert.sh
./create_self-signed-cert.sh --ssl-domain=kubeflow.cn
kubectl create --namespace istio-system secret tls kf-tls-cert --key /root/ssl/kubeflow.cn.key --cert /root/ssl/kubeflow.cn.crt
kubectl edit cm config-domain --namespace knative-serving
#在 data 下面添加：kubeflow.cn: ""
编辑kubeflow-https.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: kubeflow-gateway
namespace: kubeflow
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
- hosts:
- '*'
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: kf-tls-cert执行 kubectl apply -f kubeflow-https.yaml
执行 kubectl -n istio-system get service istio-ingressgateway 获取https端口为32143
可通过 https://ip:32143 来进行 https 访问
Multi Users 隔离设置
kubeflow安装结束后，默认情况下只能通过 [email protected] 一个账号登录操作，下面进行多账户号设置，设置完成后，每个账户拥有独立的 namespace与profile
首先创建用户的 profile，下面的yaml文件创建了2个profile，namespace分别为test1与test2
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
name: test1
spec:
owner:
kind: User
name: [email protected]
---
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
name: test2
spec:
owner:
kind: User
name: [email protected]执行 kubectl get configmap dex -n auth -o jsonpath='{.data.config\.yaml}' > dex-yaml.yaml 将dex的用户信息输出到dex-yaml.yaml文件，其中[email protected] 部署是按照时创建的，编辑该文件，添加test1与test2部分
执行 kubectl create configmap dex --from-file=config.yaml=dex-yaml.yaml -n auth --dry-run=client -o yaml | kubectl apply -f - 使新用户生效
执行 kubectl rollout restart deployment dex -n auth 重启dex，这样就完成了多用户的设置