Tungsten Fabric(6):部署更高版本的TF
之前文章中描述了部署R1902版本的踩坑过程。后续准备升级一下，目标定在了R2003，主要是https://github.com/Juniper/contrail-controller和镜像都有这个版本。很多细节和之前一样，下面就列一些需要关注的点。
部署环境
- 系统信息
[root@master01 ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@master01 ~]#
[root@master01 ~]# uname -a
Linux master01 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@master01 ~]#
- 节点信息
192.168.122.79 部署机，本地registry
192.168.122.80 k8s master ，contrail controller
192.168.122.81 k8s node01
192.168.122.82 k8s node02
部署用的contrail-ansible-deploye版本
其实这个关系应该不大的，稳妥起见还是使用对应的版本
[root@deploy R2003]# git clone https://github.com/Juniper/contrail-ansible-deployer.git -b R2003
修改RedHat.yml
[root@deploy contrail-ansible-deployer]# vim playbooks/roles/k8s/tasks/RedHat.yml
# using this to avoid issue https://github.com/ansible/ansible/issues/20711
- name: make cache to import gpg keys
#command: "yum -q makecache -y --disablerepo='*' --enablerepo='Kubernetes'"
command: "yum -q makecache -y --disablerepo='*' --enablerepo='epel'"
command: "rm -rf /etc/yum.repos.d/Kubernetes.repo"
when: k8s_package_version is defined
instance.yaml
注意CONTRAIL_VERSION: 2003-latest即可
[root@deploy contrail-ansible-deployer]# cat config/instances.yaml
provider_config:
bms:
ssh_pwd: ""
ssh_user: root
ssh_public_key: /root/.ssh/id_rsa.pub
ssh_private_key: /root/.ssh/id_rsa
domainsuffix: local
instances:
bms1:
provider: bms
roles:
config_database:
config:
control:
analytics_database:
analytics:
webui:
k8s_master:
kubemanager:
ip: 192.168.122.80
bms2:
provider: bms
roles:
vrouter:
k8s_node:
ip: 192.168.122.81
bms3:
provider: bms
roles:
vrouter:
k8s_node:
ip: 192.168.122.82
global_configuration:
CONTAINER_REGISTRY: 192.168.122.79
contrail_configuration:
CONTRAIL_VERSION: 2003-latest
CONTRAIL_CONTAINER_TAG: 2003-latest
KUBERNETES_CLUSTER_PROJECT: {}
deploy主机将公钥拷贝到这三台设备上，可以免密ssh登录。
拉取版本
- contrail的版本，可以从https://hub.docker.com/u/opencontrailnightly拉取，kube相关的镜像，可以从https://hub.docker.com/u/mirrorgooglecontainers拉取
- kube镜像需要拉取v1.12.9,这个在R2003 ReleaseNote上已经注明了
192.168.122.79/kube-proxy v1.12.9 295526df163c 12 months ago 95.7MB
192.168.122.79/kube-apiserver v1.12.9 8ea704c2d4a7 12 months ago 194MB
192.168.122.79/kube-controller-manager v1.12.9 f473e8452c8e 12 months ago 164MB
192.168.122.79/kube-scheduler v1.12.9 c79506ccc1bc 12 months ago 58.4MB
- contrail的组件，R2003多了一个contrail-provisioner，所以需要在拉取镜像的脚本中加一下
安装kube
首先每个部署节点添加阿里的k8s源
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
后续可以等待ansible脚本安装，或者在三个节点自己先手动安装好
# yum install -y kubelet-1.12.9 kubeadm-1.12.9 kubectl-1.12.9
安装其它必备软件
# yum -y install python-devel -y
# pip install wheel setuptools docker_compose
ansible的版本
不要直接安装最新版的ansible！否则在后续部署的时候，会有本不影响部署的步骤无法pass，导致部署失败(血泪教训)。
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|success` use `result is success`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
我的部署机上用的版本是
[root@deploy contrail-ansible-deployer]# pip list | grep ansible
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support it
ansible 2.7.15
docker-ce的版本
之前部署R1912的时候，docker-ce的版本安装的是docker-ce-18.03.1.ce-1.el7.centos，结果在install_k8s的时候，会有报错
[root@master01 ~]# kubeadm init --token-ttl 0 --kubernetes-version v1.12.9 --apiserver-advertise-address 192.168.122.80 --pod-network-cidr 10.32.0.0/12 &&\n mkdir -p $HOME/.kube &&\n cp -u /etc
[init] using Kubernetes version: v1.12.9
[preflight] running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.03.1-ce. Latest validated version: 18.06
所以需要装18.06版本
yum install -y docker-ce-18.06.3.ce-3.el7
systemctl enable docker
systemctl start docker
本地仓库的端口
之前没有在意，跟着文档说明一步一步做，将registry的5000端口映射到了deploy宿主机的80端口。最新部署的时候，没有映射到80端口，结果导致install_k8s无法通过。
[root@master01 ~]# kubeadm config images pull
I0616 05:04:31.387655 26257 version.go:93] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt:)
I0616 05:04:31.388398 26257 version.go:94] falling back to the local client version: v1.12.9
failed to pull image "k8s.gcr.io/kube-apiserver:v1.12.9": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
所以需要
(1)registry必须将registry docker的5000端口映射为宿主机的80
(2)修改各个节点的hosts文件，将k8s.gcr.io指向本地registry宿主机IP
(3)修改各个节点的/etc/docker/daemon.json ,将本地registry的IP(假装是k8s.gcr.io)写进"insecure-registries"列表
一点提示
- 在执行完install_k8s.yml 后，各个节点的kubelet服务应该是up，但是因为还没有安装contrail-cni，所以
[root@node01 yum.repos.d]# systemctl status kubelet
* kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
`-10-kubeadm.conf
Active: active (running) since Wed 2020-06-17 23:03:23 EDT; 3min 35s ago
Docs: https://kubernetes.io/docs/
Main PID: 25610 (kubelet)
Tasks: 22
Memory: 43.1M
CGroup: /system.slice/kubelet.service
`-25610 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni
Jun 17 23:06:36 master01 kubelet[25610]: W0617 23:06:36.164147 25610 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 17 23:06:36 master01 kubelet[25610]: E0617 23:06:36.166776 25610 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:...fig uninitialized
Jun 17 23:06:41 master01 kubelet[25610]: W0617 23:06:41.183873 25610 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 17 23:06:41 master01 kubelet[25610]: E0617 23:06:41.185117 25610 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:...fig uninitialized
Jun 17 23:06:46 master01 kubelet[25610]: W0617 23:06:46.190000 25610 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 17 23:06:46 master01 kubelet[25610]: E0617 23:06:46.194590 25610 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:...fig uninitialized
Jun 17 23:06:51 master01 kubelet[25610]: W0617 23:06:51.207761 25610 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 17 23:06:51 master01 kubelet[25610]: E0617 23:06:51.208174 25610 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:...fig uninitialized
Jun 17 23:06:56 master01 kubelet[25610]: W0617 23:06:56.215738 25610 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 17 23:06:56 master01 kubelet[25610]: E0617 23:06:56.227728 25610 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:...fig uninitialized
Hint: Some lines were ellipsized, use -l to show in full.
[root@master01 yum.repos.d]#
部署之后的检查
- 查看contrail组件的状态
[root@master01 ~]# contrail-status
Pod Service Original Name Original Version State Id Status
redis contrail-external-redis master-105 running 7136ed933b5c Up 2 hours
analytics api contrail-analytics-api master-105 running 0a111a7edfd0 Up 34 minutes
analytics collector contrail-analytics-collector master-105 running 105507305da8 Up 34 minutes
analytics nodemgr contrail-nodemgr master-105 running f6a1d9c3145d Up 34 minutes
analytics provisioner contrail-provisioner master-105 running 78273234d4ba Up 34 minutes
config api contrail-controller-config-api master-105 running 94912927b988 Up 41 minutes
config device-manager contrail-controller-config-devicemgr master-105 running 5804934ee4ea Up 41 minutes
config dnsmasq contrail-controller-config-dnsmasq master-105 running f0c1799b990e Up 41 minutes
config nodemgr contrail-nodemgr master-105 running a015edc033b5 Up 41 minutes
config provisioner contrail-provisioner master-105 running 77f056eb01da Up 41 minutes
config schema contrail-controller-config-schema master-105 running 2cdd6c62ca7a Up 41 minutes
config stats contrail-controller-config-stats master-105 running 2e399f2cd91d Up 40 minutes
config svc-monitor contrail-controller-config-svcmonitor master-105 running 2e11f2b1c20a Up 40 minutes
config-database cassandra contrail-external-cassandra master-105 running ec6ea9393eb3 Up About an hour
config-database nodemgr contrail-nodemgr master-105 running 4a152016f50e Up About an hour
config-database provisioner contrail-provisioner master-105 running 1ed205426bbc Up 41 minutes
config-database rabbitmq contrail-external-rabbitmq master-105 running a82c31f5dd57 Up About an hour
config-database zookeeper contrail-external-zookeeper master-105 running 4d2e32b0de17 Up About an hour
control control contrail-controller-control-control master-105 running 4f83acc754ce Up 36 minutes
control dns contrail-controller-control-dns master-105 running c8572afe3b1a Up 36 minutes
control named contrail-controller-control-named master-105 running 07771fca3ba5 Up 36 minutes
control nodemgr contrail-nodemgr master-105 running d33bce33655c Up 36 minutes
control provisioner contrail-provisioner master-105 running d7baa52ebc23 Up 36 minutes
database cassandra contrail-external-cassandra master-105 running f0218c363df5 Up 35 minutes
database nodemgr contrail-nodemgr master-105 running 8a18c1903e05 Up 35 minutes
database provisioner contrail-provisioner master-105 running f49298d6dbcb Up 35 minutes
database query-engine contrail-analytics-query-engine master-105 running 60406338c8a0 Up 35 minutes
kubernetes kube-manager contrail-kubernetes-kube-manager master-105 running 3b1905176974 Up 32 minutes
webui job contrail-controller-webui-job master-105 running 4fa0cc25d81e Up 38 minutes
webui web contrail-controller-webui-web master-105 running 2f8e79cb9e19 Up 38 minutes
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail config-database ==
nodemgr: initializing (Disk for DB is too low. )
zookeeper: active
rabbitmq: active
cassandra: active
== Contrail kubernetes ==
kube-manager: active
== Contrail database ==
nodemgr: initializing (Disk for DB is too low. )
query-engine: active
cassandra: active
== Contrail analytics ==
nodemgr: active
api: active
collector: active
== Contrail webui ==
web: active
job: active
== Contrail config ==
svc-monitor: active
nodemgr: active
device-manager: active
api: active
schema: active
[root@master01 ~]#
Disk for DB is too low.的提示在实验环境中可以忽略。
- 查看pod状态,需要全是running
[root@master01 ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system coredns-85c98899b4-8lk8h 1/1 Running 0 135m 10.47.255.252 node01
kube-system coredns-85c98899b4-h9bjb 1/1 Running 0 135m 10.47.255.251 node01
kube-system etcd-master01 1/1 Running 2 35m 192.168.122.80 master01
kube-system kube-apiserver-master01 1/1 Running 2 36m 192.168.122.80 master01
kube-system kube-controller-manager-master01 1/1 Running 3 36m 192.168.122.80 master01
kube-system kube-proxy-ffxls 1/1 Running 1 134m 192.168.122.81 node01
kube-system kube-proxy-jqbxp 1/1 Running 1 134m 192.168.122.82 node02
kube-system kube-proxy-kc4fh 1/1 Running 2 135m 192.168.122.80 master01
kube-system kube-scheduler-master01 1/1 Running 2 36m 192.168.122.80 master01
kube-system kubernetes-dashboard-76456c6d4b-vhkgp 1/1 Running 0 134m 192.168.122.81 node01
[root@master01 ~]#
- 查看node状态
[root@master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 NotReady master 166m v1.12.9
node01 Ready 165m v1.12.9
node02 Ready 165m v1.12.9
[root@master01 ~]#
[root@master01 ~]# mkdir -p /etc/cni/net.d/
[root@master01 ~]# scp root@node02:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf
[root@master01 ~]# systemctl restart kubelet
#等待一分钟
[root@master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master01 Ready master 166m v1.12.9
node01 Ready 165m v1.12.9
node02 Ready 165m v1.12.9
[root@master01 ~]#