KubeEdge部署 ~ Kubernetes 1.17.4 部署

本文介绍了如何在 ubuntu 18.04 64 bit 双核 CPU 虚拟机上使用 kubeadm 部署 Kubernetes 1.17.4 集群的过程，网络插件为 flannel v0.11.0，镜像源为阿里云

一、 安装docker

apt-get install docker.io
执行如下命令新建 /etc/docker/daemon.json 文件：

cat > /etc/docker/daemon.json <<-EOF
{
  "registry-mirrors": [
    "https://a8qh6yqv.mirror.aliyuncs.com",
    "http://hub-mirror.c.163.com"
  ],
  "exec-opts": ["native.cgroupdriver=systemd"]
}

EOF

注：
registry-mirrors 为镜像加速器地址。
native.cgroupdriver=systemd 表示使用的 cgroup 驱动为 systemd（k8s 使用此方式），默认为 cgroupfs。修改原因是 kubeadm.conf 中修改k8s的驱动方式不成功。

重启docker，查看 cgroup：

# systemctl restart docker 
# docker info | grep -i cgroup
Cgroup Driver: systemd
出现 systemd 表示修改成功。

二、部署 k8s master 主机

k8s的部署分 master 主机和 node 节点。本节为 master 主机。

2.1 关闭swap

编辑 /etc/fstab 文件，注释掉swap分区挂载的行，示例：

# swap was on /dev/sda5 during installation
UUID=aaa38da3-6e60-4e9d-bfc6-7128fd05f1c7 none swapsw  0  0

再执行：   # swapoff -a

2.2 添加国内k8s源     此处选择阿里云的：

# cat < /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
添加key： # cat https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

如不成功，则先通过一些方法下载：https://packages.cloud.google.com/apt/doc/apt-key.gpg ，放到工程目录。再执行：

# cat apt-key.gpg | sudo apt-key add -

也可以添加Key:   sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg |sudo  apt-key add -

3.3 更新源

# sudo apt-get update
安装 kubeadm、kubectl、kubelet、kubernetes-cni 等工具。

# sudo apt-get install -y kubeadm kubectl kubelet kubernetes-cni

注1：安装 kubeadm 会自动安装 kubectl、kubelet 和 kubernetes-cni，故只指定 kubeadm 亦可。
注2：本文安装时，得到的版本为1.17.4，kubernetes-cni 为0.7.5。下载文件位于 /var/cache/apt/archives/ 目录中。

3.4 获取部署所需的镜像版本

# sudo kubeadm config images list
输出如下：

W1214 08:46:14.303772    8461 version.go:102] falling back to the local client version: v1.17.4
W1214 08:46:14.304223    8461 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1214 08:46:14.304609    8461 validation.go:28] Cannot validate kubelet config - no validator is available
k8s.gcr.io/kube-apiserver:v1.17.4
k8s.gcr.io/kube-controller-manager:v1.17.4
k8s.gcr.io/kube-scheduler:v1.17.4
k8s.gcr.io/kube-proxy:v1.17.4
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.5

前面提示的警告信息可不理会。此处是确认本版本 kubeadm 匹配的镜像的版本，因为各组件版本不同可能出现兼容性问题。

3.5 拉取镜像文件。
一般地，国内无法直接下载 k8s.gcr.io 的镜像。方式有二：
1、在初始化k8s时，使用阿里云镜像地址，此地址可以顺利下载，见下初始化。
2、下载好前述镜像。使用如下脚本pullk8s.sh（注意脚本必须添加x属性）：

#!/bin/bash
# 下面的镜像应该去除"k8s.gcr.io/"的前缀，版本换成kubeadm config images list命令获取到的版本
images=(
    kube-apiserver:v1.17.4
    kube-controller-manager:v1.17.4
    kube-scheduler:v1.17.4
    kube-proxy:v1.17.4
    pause:3.1
    etcd:3.4.3-0
    coredns:1.6.5
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
    docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

拉取：

chmod +x pullk8s.sh
bash pullk8s.sh  (或 ./pullk8s.sh)

3.6 网络
设置网络配置：

mkdir -p /etc/cni/net.d

cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.244.0.0/16",
        "routes": [
            {"dst": "0.0.0.0/0"}
        ]
    }
}
EOF

cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
    "cniVersion": "0.3.0",
    "type": "loopback"
}
EOF

经实践，此步骤不做亦可。

3.7 下载flannel镜像
docker pull quay.io/coreos/flannel:v0.11.0-amd64
注：如果无法下载，需要使用其它方法。
flannel 镜像信息：

# docker images | grep flannel
quay.io/coreos/flannel   v0.11.0-amd64 ff281650a721 11 months ago 52.6MB

3.8 初始化
版本一：

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
  --image-repository registry.aliyuncs.com/google_containers

注：
–pod-network-cidr 指定了网络段，后续网络插件会使用到（本文使用 flannel）。
–image-repository 指定了镜像地址，默认为 k8s.gcr.io，此处指定为阿里云镜像地址 registry.aliyuncs.com/google_containers。
注意，其它参数默认。

上述命令等同如下命令：

sudo kubeadm init \
  --apiserver-advertise-address=192.168.50.128 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.17.0 \
  --service-cidr=10.1.0.0/16\
  --pod-network-cidr=10.244.0.0/16

输出：

W0318 14:36:29.662899    9495 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0318 14:36:29.662966    9495 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.4
[preflight] Running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0318 14:36:31.481612    9495 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0318 14:36:31.517795    9495 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.044383 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ubuntu as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ubuntu as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ma01mg.38n1fcelg0wk35s3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.50.128:6443 --token ma01mg.38n1fcelg0wk35s3 \
    --discovery-token-ca-cert-hash sha256:48a197ee4268416f8a6930418d7b9eb1f31ca3ce5b21240857e253c85161b6b1 

版本二，根据前文脚本自行拉取版本：

kubeadm init --pod-network-cidr=10.244.0.0/16

注：在初始化过成功可能遇到的问题：

1、 [ERROR Port-10251]: Port 10250 is in use

解决办法：

重启一下kubeadm：     [root@k8s-master ~]#   kubeadm reset

如果重启后端口仍然被占用,  查看占用端口的进程：netstat  -tunlp  | grep 10250

强制杀死占用端口的进程： sudo fuser -k -n tcp 10250

 

部署成功后 ,根据提示，根据拷贝 admin.conf 文件到当前用户相应目录下。admin.conf 文件后续会使用到（需要拷贝到 node 节点）。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

初始化时，如不存在则自动下载镜像，初始化后镜像如下：

# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy                v1.17.4             7d54289267dc        1 days ago         116MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.17.4             0cae8d5cc64c        1 days ago         171MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.17.4            5eb3b7486872        1 days ago         161MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.17.4             78c190f736b1        1 days ago         94.4MB
registry.aliyuncs.com/google_containers/coredns                   1.6.5               70f311871ae1        6 weeks ago         41.6MB
registry.aliyuncs.com/google_containers/etcd                      3.4.3-0             303ce5db0e90        8 weeks ago         288MB
registry.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

此时 pod 状态如下：

# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-67qtv          0/1     Pending   0          3h26m
coredns-9d85f5447-cg87c          0/1     Pending   0          3h26m
etcd-ubuntu                      1/1     Running   0          3h27m
kube-apiserver-ubuntu            1/1     Running   0          3h27m
kube-controller-manager-ubuntu   1/1     Running   0          3h27m
kube-proxy-chqbq                 1/1     Running   0          3h26m
kube-scheduler-ubuntu            1/1     Running   0          3h27m

除 coredns 状态为 Pending外，其它 pod 均运行。这是因为没有部署网络插件导致的。本文选用 flannel 。

执行如下命令部署 flannel：

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
注：
使用 flannel 仓库的 kube-flannel.yml 文件部署。详细可参考该文件。
如果无法访问，则可手动下载 https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml 文件到当前目录，再执行 kubectl apply -f kube-flannel.yml 命令。

# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

三、node 节点

k8s的部署分 master 主机和 node 节点。本节为 node 节点。

3.1 前置条件
在 node 节点上操作。
1、安装kubeadm，见前述。
2、下载flannel镜像，见前述（如果不预先下载，在加入集群时会自动下载）。
3、将主机的 /etc/kubernetes/admin.conf 文件拷贝到 node 节点的 /etc/kubernetes/ 目录。（注：在　master 节点使用 scp 命令即可，kubernetes 不存在自行创建）

3.2 加入集群
此时，k8s服务还没有启动。执行如下命令以加入节点：

kubeadm join 192.168.50.128:6443 --token 1rpp8b.axfud1xrsvx4q8nw \
    --discovery-token-ca-cert-hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3c 

提示信息如下：

[preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...


This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

加入群集过程中会下载必须的k8s镜像，注意，master 主机已经指定为阿里源的源，所以 node 节点上亦是该源。

REPOSITORY                                           TAG                 IMAGE ID            CREATED             SIZE
registry.aliyuncs.com/google_containers/kube-proxy   v1.17.0             7d54289267dc        2 weeks ago         116MB
registry.aliyuncs.com/google_containers/coredns      1.6.5               70f311871ae1        7 weeks ago         41.6MB
quay.io/coreos/flannel                               v0.11.0-amd64       ff281650a721        11 months ago       52.6MB
registry.aliyuncs.com/google_containers/pause        3.1                 da86e6ba6ca1        2 years ago         742kB

成功加入后，本节点有如下相关服务在运行：

# ps aux | grep kube
root       3269  1.6  4.2 754668 86784 ?        Ssl  Dec20  18:34 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1
root       3632  0.1  1.1 140104 22412 ?        Ssl  Dec20   2:14 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=node
root       4385  0.0  1.6 407356 33704 ?        Ssl  Dec20   0:51 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
root     121292  0.0  0.0  14228  1032 pts/0    S+   00:33   0:00 grep --color=auto kube

主要有 kubelet、kube-proxy、flanneld，等。

docker 容器列表如下：

# docker ps
CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS              PORTS               NAMES
2fde9bb78fd7        ff281650a721                                         "/opt/bin/flanneld -…"   7 minutes ago       Up 7 minutes                            k8s_kube-flannel_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_1
aa7ca3d5825e        registry.aliyuncs.com/google_containers/kube-proxy   "/usr/local/bin/kube…"   8 minutes ago       Up 8 minutes                            k8s_kube-proxy_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0
ac61ed8d7295        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 8 minutes ago       Up 8 minutes                            k8s_POD_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_0
423f9e42c082        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 8 minutes ago       Up 8 minutes                            k8s_POD_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0

查看 flannel 网络信息：

# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

查看本机 IP 信息：

# ifconfig 

五、验证

在 master 节点执行：

# kubectl get nodes       
NAME     STATUS   ROLES    AGE     VERSION
node     Ready       17m     v1.17.0
ubuntu   Ready    master   5h11m   v1.17.0

可以看到两台机器已为 Ready 状态。node 机器由 NotReady 变为 Ready，耗时大约 10 余秒。

使用 busybox 镜像简单测试 pod。在 master 节点执行：

# kubectl run -i --tty busybox --image=latelee/busybox --restart=Never -- sh

稍等片刻，即可进入 busybox 命令行：

 # uname -a
Linux busybox 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 GNU/Linux

另起命令行，查看 pod 运行状态：

# kubectl get pod -o wide 
NAME      READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
busybox   1/1     Running   0          74s   10.244.1.4   node              

可以看到 pod 为 Running 状态，运行在 node 上。
在 node 节点上查看：

# docker ps | grep busybox
ba5d1a480294        latelee/busybox                                      "sh"                     2 minutes ago       Up 2 minutes                            k8s_busybox_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0
8c643171ac09        registry.aliyuncs.com/google_containers/pause:3.1    "/pause"                 2 minutes ago       Up 2 minutes                            k8s_POD_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0

此时在 master 节点退出 busybox， pod 依旧存在，但不是 READY 状态，node 主机也没有 busybox 容器运行。

验证通过，k8s部署成功。
 

 

参考文档：https://blog.csdn.net/subfate/article/details/103774072