使用二进制安装包部署kubernetes v1.9.8集群的参考手册

本次部署使用3个节点：

• Master节点，10.0.2.15
• Node1节点，10.0.2.4      docker0网桥默认地址手动配置为：172.16.10.1/24
• Node2节点，10.0.2.5      docker0网桥默认地址手动配置为：172.16.20.1/24

使用二进制文件方式安装k8s集群的方法。部署的是Kubernetes v1.9.8。

https://storage.googleapis.com/kubernetes-release/release/v1.9.8/kubernetes-server-linux-amd64.tar.gz

我们使用CentOS7.5，注意在每个节点上处理一下防火墙策略：

firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" source address="10.0.2.0/24" accept"

firewall-cmd --reload

一、使用二进制方式部署k8s的Master节点

1、etcd服务

从 https://github.com/coreos/etcd/releases下载etcd二进制文件，将etcd和etcdctl文件复制到/usr/bin目录。

配置systemd服务文件：/usr/lib/systemd/system/etcd.service

[root@bogon system]# more etcd.service

[Unit]

Description=Etcd Server

After=network.target

[Service]

Type=simple

EnvironmentFile=-/etc/etcd/etcd.conf

WorkingDirectory=/var/lib/etcd/

ExecStart=/usr/bin/etcd

Restart=on-failure

[Install]

WantedBy=multi-user.target

配置和启动etcd服务：

[root@k8s-master ~]# ls

anaconda-ks.cfg  etcd-v3.3.5-linux-amd64  etcd-v3.3.5-linux-amd64.tar.gz

[root@k8s-master ~]# cd etcd-v3.3.5-linux-amd64

[root@k8s-master etcd-v3.3.5-linux-amd64]# ls

Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

[root@k8s-master etcd-v3.3.5-linux-amd64]# mv etcd etcdctl /usr/bin

[root@k8s-master etcd-v3.3.5-linux-amd64]# systemctl daemon-reload

[root@k8s-master etcd-v3.3.5-linux-amd64]# systemctl enable etcd.service

Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

[root@k8s-master etcd-v3.3.5-linux-amd64]# mkdir -p /var/lib/etcd/

[root@k8s-master etcd-v3.3.5-linux-amd64]# systemctl start etcd.service

[root@k8s-master etcd-v3.3.5-linux-amd64]# etcdctl cluster-health

member 8e9e05c52164694d is healthy: got healthy result from http://localhost:2379

cluster is healthy

[root@k8s-master etcd-v3.3.5-linux-amd64]#

2、kube-apiserver服务

将解压后kube-apiserver、kube-controller-manager、kube-scheduler以及管理要使用的kubectl二进制命令文件放到/usr/bin目录，即完成这几个服务的安装。

接下来对api服务进行必要的配置。

配置systemd服务文件：/usr/lib/systemd/system/kube-apiserver.service

# more /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation= https://github.com/kubernetes/kubernetes

After=etcd.service

Wants=etcd.service

[Service]

EnvironmentFile=/etc/kubernetes/apiserver

ExecStart=/usr/bin/kube-apiserver $KUBE_API_ARGS

Restart=on-failure

Type=notify

[Install]

WantedBy=multi-user.target

配置文件：

# mkdir -p /etc/kubernetes

# more /etc/kubernetes/apiserver

KUBE_API_ARGS="--storage-backend=etcd3 --etcd-servers=http://127.0.0.1:2379  --insecure-bind-address=0.0.0.0  --insecure-port=8080 --service-cluster-ip-range=10.10.10.0/24 --service-node-port-range=1-65535 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,DefaultStorageClass,ResourceQuota --logtostderr=true  --log-dir=/var/log/kubernetes --v=2"

3、kube-controller-manager服务

kube-controller-manager服务依赖于kube-apiserver服务：

配置systemd服务文件：/usr/lib/systemd/system/kube-controller-manager.service

# more /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation= https://github.com/GoogleCloudPlatform/kubernetes

After=kube-apiserver.service

Requires=kube-apiserver.service

[Service]

EnvironmentFile=-/etc/kubernetes/controller-manager

ExecStart=/usr/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

[root@k8s-master system]# more /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation= https://github.com/GoogleCloudPlatform/kubernetes

After=kube-apiserver.service

Requires=kube-apiserver.service

[Service]

EnvironmentFile=-/etc/kubernetes/controller-manager

ExecStart=/usr/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置文件：

# more /etc/kubernetes/controller-manager

KUBE_CONTROLLER_MANAGER_ARGS="--master=http://10.0.2.15:8080 --logtostderr=true --log-dir=/var/log/kubernetes --v=2"

4、kube-scheduler服务

kube-scheduler服务也依赖于kube-apiserver服务。

配置systemd服务文件：/usr/lib/systemd/system/kube-scheduler.service

# more /usr/lib/systemd/system/kube-scheduler.service

[Unit]

Description=Kubernetes Scheduler

Documentation= https://github.com/GoogleCloudPlatform/kubernetes

After=kube-apiserver.service

Requires=kube-apiserver.service

[Service]

EnvironmentFile=-/etc/kubernetes/scheduler

ExecStart=/usr/bin/kube-scheduler $KUBE_SCHEDULER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

配置文件：

# cat /etc/kubernetes/scheduler

KUBE_SCHEDULER_ARGS="--master=http://10.0.2.15:8080 --logtostderr=true --log-dir=/var/log/kubernetes --v=2"

5、完成以上配置后，按顺序启动这些服务

systemctl daemon-reload

systemctl enable kube-apiserver.service

systemctl start kube-apiserver.service

systemctl enable kube-controller-manager.service

systemctl start kube-controller-manager.service

systemctl enable kube-scheduler.service

systemctl start kube-scheduler.service

检查每个服务的健康状态：

systemctl status kube-apiserver.service

systemctl status kube-controller-manager.service

systemctl status kube-scheduler.service

如果有服务出现异常，可执行journalctl -xe -u kube-scheduler查看一个指定的服务的事件信息日志。

二、使用二进制方式部署k8s的Node节点

在Node节点上，以同样的方式把从压缩包中解压出的二进制文件kubelet  kube-proxy放到/usr/bin目录中。

1、安装docker ce

先安装依赖工具包：

yum install -y yum-utils device-mapper-persistent-data lvm2

安装docker yum源：

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

安装docker引擎：

yum install  docker-ce.x86_64

执行docker info查看安装结果。当前安装的是docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm。

将docker0网桥的默认地址配置为我们指定的：172.16.10.1

# pwd

/usr/lib/systemd/system

[root@k8s-node1 system]# more docker.service

[Unit]

Description=Docker Application Container Engine

Documentation= https://docs.docker.com

After=network-online.target firewalld.service

Wants=network-online.target

[Service]

Type=notify

# the default is not to use systemd for cgroups because the delegate issues still

# exists and systemd currently does not support the cgroup feature set required

# for containers run by docker

ExecStart=/usr/bin/dockerd --bip=172.16.10.1/24

ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.

# Only systemd 226 and above support this version.

#TasksMax=infinity

TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

# kill only the docker process, not all processes in the cgroup

KillMode=process

# restart the docker process if it exits prematurely

Restart=on-failure

StartLimitBurst=3

StartLimitInterval=60s

[Install]

WantedBy=multi-user.target

注：--bip=172.16.10.1/24，将docker0网桥默认使用的网络地址变更为我们指定的地址。

启动docker服务：

systemctl daemon-reload

systemctl start docker

查看node1的网络地址和路由信息：

# ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: enp0s3: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:9f:89:14 brd ff:ff:ff:ff:ff:ff

    inet 10.0.2.4/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3

       valid_lft 774sec preferred_lft 774sec

    inet6 fe80::a00:27ff:fe9f:8914/64 scope link

       valid_lft forever preferred_lft forever

3: docker0: mtu 1500 qdisc noqueue state DOWN group default

    link/ether 02:42:22:bc:7a:ea brd ff:ff:ff:ff:ff:ff

    inet 172.16.10.1/24 brd 172.16.10.255 scope global docker0

       valid_lft forever preferred_lft forever

[root@k8s-node1 system]# ip r

default via 10.0.2.1 dev enp0s3 proto dhcp metric 100

10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.4 metric 100

172.16.10.0/24 dev docker0 proto kernel scope link src 172.16.10.1

2、kubelet服务

配置systemd服务文件：/usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet Server

Documentation= https://github.com/GoogleCloudPlatform/kubernetes

After=docker.service

Requires=docker.service

[Service]

WorkingDirectory=/var/lib/kubelet

EnvironmentFile=-/etc/kubernetes/kubelet

ExecStart=/usr/bin/kubelet $KUBELET_ARGS

Restart=on-failure

KillMode=process

[Install]

WantedBy=multi-user.target

配置文件：

# cd  /etc/kubernetes

# more kubelet

KUBELET_ARGS="--address=10.0.2.4 --port=10250 --hostname-override=10.0.2.4 --allow-privileged=false --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cluster-dns

=10.10.10.2 --cluster-domain=cluster.local --fail-swap-on=false --logtostderr=true --log-dir=/var/log/kubernetes --v=4"

用于kubelet连接Master Apiserver的配置文件：

# more kubelet.kubeconfig apiVersion: v1

kind: Config

clusters:

  - cluster:

      server: http://10.0.2.15:8080

    name: local

contexts:

  - context:

      cluster: local

    name: local

current-context: local

启动kubelet服务：

mkdir -p /var/lib/kubelet

systemctl daemon-reload

systemctl enable kubelet

systemctl start kubelet

systemctl status kubelet

3、kube-proxy服务

kube-proxy服务依赖于network服务。

配置systemd服务文件：/usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Kube-proxy Server

Documentation= https://github.com/GoogleCloudPlatform/kubernetes

After=network.service

Requires=network.service

[Service]

EnvironmentFile=/etc/kubernetes/proxy

ExecStart=/usr/bin/kube-proxy $KUBE_PROXY_ARGS

Restart=on-failure

LimitNOFILE=65536

KillMode=process

[Install]

WantedBy=multi-user.target

配置文件：

# cat /etc/kubernetes/proxy

KUBE_PROXY_ARGS="--master=http://10.0.2.15:8080 --hostname-override=10.0.2.4 --logtostderr=true --log-dir=/var/log/kubernetes --v=4"

启动kube-proxy服务：

systemctl daemon-reload

systemctl enable kube-proxy

systemctl start kube-proxy

systemctl status kube-proxy

4、按照前面相同的步骤配置node2节点

唯一的不同之处是在指定docker0网桥的默认地址时使用下面的网段和地址：

ExecStart=/usr/bin/dockerd --bip=172.16.20.1/24

注：请在配置过程中将使用节点ip地址的地方也替换为node2的实际ip地址10.0.2.5

三、对搭建完成的k8s集群进行一些验证测试

在Master上查看k8s各项服务状态

[root@k8s-master bin]# kubectl get nodes

NAME       STATUS    ROLES     AGE       VERSION

10.0.2.4   Ready         28m       v1.9.8

10.0.2.5   Ready         1m        v1.9.8

[root@k8s-master bin]# kubectl get svc

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE

kubernetes   ClusterIP   10.10.10.1           443/TCP   3h

[root@k8s-master bin]# kubectl get cs

NAME                 STATUS    MESSAGE             ERROR

scheduler            Healthy   ok                  

controller-manager   Healthy   ok                  

etcd-0               Healthy   {"health":"true"}