Kubernetes etcd calico self-hosted deployment note

序言

网络太慢了，所以这篇博客我希望记录完全离线安装kubernetes的细节。

1. 版本固定
2. 稳定可靠的环境安装和清除
3. 形成自动化脚本

env	version
centos	7
docker	19.03
kubernetes	v18.0
calico	–

calico

Calico 遵从Kubernetes网络模型，以CNI(Container Network Interface)插件的形式安装在Kubernetes上。
参考：install calico on kubernetes

Calico CNI的配置

IPIP模式和BGP模式。

IPIP模式就是主机IP报文包装Pod IP报文。BGP则直接路由转发，每个主机作为路由器。

要在Kubernetes上安装CNI插件，下载对应插件的yaml定义。Calico插件的YAML定义称为manifest,在calico networking manifest里面可以配置网络模式（默认是IPIP网络）：

 # Enable IPIP
 - name: CALICO_IPV4POOL_IPIP
   value: "Always" # 这里改成off,使用BGP网络模式

关于Calico网络模式：我看到有个博客讲的挺清楚的：k8s网络之Calico网络

附录：离线安装

git

version: 1.8.3

# git
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/git-1.8.3.1-21.el7_7.x86_64.rpm
# rsync
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/rsync-3.1.2-10.el7.x86_64.rpm
# perl(Error)
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/perl-Error-0.17020-2.el7.noarch.rpm
# perl-Git
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/perl-Git-1.8.3.1-21.el7_7.noarch.rpm
# perl(Term::ReadKey)
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/perl-TermReadKey-2.30-20.el7.x86_64.rpm

安装顺序：

rpm -ivh rsync-3.1.2-10.el7.x86_64.rpm
rpm -ivh perl-Error-0.17020-2.el7.noarch.rpm
rpm -ivh perl-TermReadKey-2.30-20.el7.x86_64.rpm
rpm -ivh perl-Git-1.8.3.1-21.el7_7.noarch.rpm git-1.8.3.1-21.el7_7.x86_64.rpm

gcc

rpm离线下载地址

# gcc-c++
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/gcc-c++-4.8.5-39.el7.x86_64.rpm
# gcc 包及依赖
# gcc
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/gcc-4.8.5-39.el7.x86_64.rpm
# cpp
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/cpp-4.8.5-39.el7.x86_64.rpm
# glibc-devel >= 2.2.90-12
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/glibc-devel-2.17-307.el7.1.x86_64.rpm
  # glibc = 2.17-307.el7.1
  wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/glibc-2.17-307.el7.1.x86_64.rpm
       # glibc-common = 2.17-307.el7.1
       wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/glibc-common-2.17-307.el7.1.x86_64.rpm    
  # glibc-headers
  wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/glibc-headers-2.17-307.el7.1.x86_64.rpm
      # kernel-headers
      wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/kernel-headers-3.10.0-1127.el7.x86_64.rpm
  # glibc-headers = 2.17-307.el7.1
  wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/glibc-headers-2.17-307.el7.1.x86_64.rpm
  
# libgcc >= 4.8.5-39.el7
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libgcc-4.8.5-39.el7.x86_64.rpm
# libgomp = 4.8.5-39.el7
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libgomp-4.8.5-39.el7.x86_64.rpm

# 可独立可装包
# libmpc.so.3()(64bit)
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libmpc-1.0.1-3.el7.x86_64.rpm
# libmpfr.so.4()(64bit）
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/mpfr-3.1.1-4.el7.x86_64.rpm
# libstdc++ = 4.8.5-39.el7
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libstdc++-4.8.5-39.el7.x86_64.rpm
# libstdc++-devel = 4.8.5-39.el7
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libstdc++-devel-4.8.5-39.el7.x86_64.rpm

安装顺序：

# libmpfr.so.4()(64bit）
rpm -ivh mpfr-3.1.1-4.el7.x86_64.rpm
# libmpc.so.3()(64bit)
rpm -ivh libmpc-1.0.1-3.el7.x86_64.rpm
# libstdc++ = 4.8.5-39.el7
rpm -ivhU libstdc++-4.8.5-39.el7.x86_64.rpm
# libstdc++-devel = 4.8.5-39.el7
rpm -ivhU libstdc++-devel-4.8.5-39.el7.x86_64.rpm

# cpp
rpm -ivh cpp-4.8.5-39.el7.x86_64.rpm

######### glibc-devel
rpm -iUvh glibc-2.17-307.el7.1.x86_64.rpm glibc-common-2.17-307.el7.1.x86_64.rpm
# kernel-headers
rpm -ivh kernel-headers-3.10.0-1127.el7.x86_64.rpm
# glibc-headers
rpm -ivh glibc-headers-2.17-307.el7.1.x86_64.rpm
# glibc-devel
rpm -ivh glibc-devel-2.17-307.el7.1.x86_64.rpm

########## libgcc >= 4.8.5-39.el7
rpm -iUvh libgcc-4.8.5-39.el7.x86_64.rpm

########## libgomp = 4.8.5-39.el7
rpm -iUvh libgomp-4.8.5-39.el7.x86_64.rpm
########## gcc
rpm -iUvh gcc-4.8.5-39.el7.x86_64.rpm
rpm -iUvh gcc-c++-4.8.5-39.el7.x86_64.rpm

docker

# docker
wget -c https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-19.03.9-3.el7.x86_64.rpm
# docker cli
wget -c https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-cli-19.03.9-3.el7.x86_64.rpm
# containerd.io
wget -c  https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.4-3.1.el7.x86_64.rpm
# libcgroup
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/libcgroup-0.41-21.el7.x86_64.rpm
# container-selinux
wget -c http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-3.el7.noarch.rpm
# policycoreutils-python
wget -c http://mirror.centos.org/centos/7/os/x86_64/Packages/policycoreutils-python-2.5-34.el7.x86_64.rpm
yum install --downloadonly --downloaddir=/root/rpm policycoreutils-python-2.5-34.el7.x86_64

安装顺序：

rpm -ivh libcgroup-0.41-21.el7.x86_64.rpm
rpm -iUvh audit-libs-2.8.5-4.el7.x86_64.rpm audit-2.8.5-4.el7.x86_64.rpm
rpm -iUvh policycoreutils-2.5-34.el7.x86_64.rpm
rpm -iUvh python-IPy-0.75-6.el7.noarch.rpm
rpm -iUvh audit-libs-python-2.8.5-4.el7.x86_64.rpm checkpolicy-2.5-8.el7.x86_64.rpm libsemanage-python-2.5-14.el7.x86_64.rpm policycoreutils-python-2.5-34.el7.x86_64.rpm setools-libs-3.3.8-4.el7.x86_64.rpm
rpm -ivh policycoreutils-2.5-34.el7.x86_64.rpm

rpm -ivh docker-ce-cli-19.03.9-3.el7.x86_64.rpm
rpm -iUvh containerd.io-1.2.4-3.1.el7.x86_64.rpm
rpm -iUvh container-selinux-2.107-3.el7.noarch.rpm
rpm -iUvh docker-ce-19.03.9-3.el7.x86_64.rpm

kubenetes yum源

这个部分是从阿里云上拉取kubernetes的镜像。

将下面内容保存至kuber.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

复制到yum repo库：

cp -b conf/kuber.repo /etc/yum.repos.d/kubernetes.repo
yum install -y --downloadonly --downloaddir=.  kubelet kubeadm kubectl --disableexcludes=kubernetes

安装顺序：

rpm -ivh socat-1.7.3.2-2.el7.x86_64.rpm
rpm -ivh libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm
rpm -ivh conntrack-tools-1.4.4-7.el7.x86_64.rpm
rpm -ivh 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm
rpm -iUvh 029bc0d7b2112098bdfa3f4621f2ce325d7a2c336f98fa80395a3a112ab2a713-kubernetes-cni-0.8.6-0.x86_64.rpm  de422b616a367cafae90aef704625fc34b0b222353f4fb59235bb3cf2f9d0988-kubelet-1.18.8-0.x86_64.rpm efd73a4178ebf9939f86b4200dba0247a57ead65f2403d8576b241faf478ac42-kubectl-1.18.8-0.x86_64.rpm f1795288ba09abd5e6738fbf82e237e3792ee2dcda4512fa5d35388dae622cef-kubeadm-1.18.8-0.x86_64.rpm

Kubernetes setup

参考：Creating a cluster with kubeadm 使用kubeadm搭建kubernetes环境。

获取Kubernetes镜像

# 拉取镜像
kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.0

输出：

W0815 07:33:13.305739   20742 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.18.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.18.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.18.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.18.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.2
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.4.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.6.7

这些镜像是要注意离线保存的。

获取Calico Addon镜像

参考：
https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises

选择etcd方式保存路由信息：
https://docs.projectcalico.org/manifests/calico-etcd.yaml

docker pull calico/pod2daemon-flexvol:v3.15.1
docker pull calico/node:v3.15.1
docker pull calico/kube-controllers:v3.15.1
docker pull calico/cni:v3.15.1