记一次详细安装 kubernetes、istio 步骤

写在前面

先说一下我的机器配置，热乎乎的裸机，一共三台配置如下

• 10.20.1.103 4C 8G 磁盘 50G node4 master centos7
• 10.20.1.104 4C 8G 磁盘 50G node5 node centos7
• 10.20.1.105 4C 8G 磁盘 50G node6 node centos7

不要忘记修改你的主机名以及 hosts 文件映射，不然你的主机名就会全是 localhost…而且如果全是 localhost 的话使用 kubeadm 加入集群时也会出现 bug。

我的安装方法是基于 k8s 官方的推荐方法 kubeadm，istio 同样也是，如果想看原文访问以下连接。

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
https://istio.io/docs/setup/getting-started/

我在这里默认你已经安装好了 docker
基本就这些了，下面就开始了！

kubernetes

step1 安装常用应用

yum install -y vim wget

step2 验证机器

验证你的每个节点是否唯一，一般新的机器都没有任何问题。

ip link
sudo cat /sys/class/dmi/id/product_uuid

step3 开放端口

每台机器开放端口不用多说

// master 执行
[root@localhost ~]# firewall-cmd --zone=public --add-port=6443/tcp --permanent
success
[root@localhost ~]# firewall-cmd --zone=public --add-port=2379/tcp --permanent
success
[root@localhost ~]# firewall-cmd --zone=public --add-port=2380/tcp --permanent
success
[root@localhost ~]# firewall-cmd --zone=public --add-port=10250/tcp --permanent
success
[root@localhost ~]# firewall-cmd --zone=public --add-port=10251/tcp --permanent
success
[root@localhost ~]# firewall-cmd --zone=public --add-port=10252/tcp --permanent
success
[root@localhost ~]# firewall-cmd --reload
success

// node 执行
[root@localhost ~]# firewall-cmd --zone=public --add-port=10250/tcp --permanent
success
[root@localhost ~]# firewall-cmd --reload
success

step4 安装组件

安装 kubeadm, kubelet 和 kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

setenforce 0

sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

截止到上一步，如果你不修改你上网的姿势你是不会安装成功的，失败如下图。这和正常。

[root@localhost ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * extras: mirrors.huaweicloud.com
 * updates: mirrors.huaweicloud.com
https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2404:6800:4012::200e: Network is unreachable"
Trying other mirror.
https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2404:6800:4012::200e: Network is unreachable"
Trying other mirror.
^Chttps://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno 14] curl#56 - "Callback aborted"
Trying other mirror.

因为我们访问不到 google，所以我们修改一下源，去其他路径下载，修改之后如下面代码块所示：实际只是注释了两行代码而已。

vim /etc/yum.repos.d/kubernetes.repo

name=Kubernetes
# baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
# gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kube*

验证安装，可以看到我们已经安装成功了

[root@localhost ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[root@localhost ~]# kubelet --version
Kubernetes v1.17.1
[root@localhost ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-14T21:02:14Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

启动 kubelet

systemctl enable --now kubelet

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

systemctl daemon-reload
systemctl restart kubelet

截止到这里已经成功一半了。下一步是启动集群。

step5 启动集群

// 先启动 master
[root@localhost ~]# kubeadm init --kubernetes-version=v1.17.0
W0117 17:16:23.316968   11538 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0117 17:16:23.317084   11538 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or highe

我们执行命令之后，可以看到一些警告以及错误，那么我们执行以下命令，关闭防火墙以及交换分区

systemctl stop firewalld
swapoff -a

再次执行可以看到只有一个警告，这个警告我们修改 docker 的驱动即可。如下图所示，修改之后重启 docker

[root@node1 ~]# vim /etc/docker/daemon.json 

{
 "exec-opts":["native.cgroupdriver=systemd"]
}

重启之后我们再次执行，此时肯定会再次失败。很简单可安装时的失败是一致的。我们访问不到 google。也就是我们 pull 不下来镜像。

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.17.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.3-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.6.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1

所以我们现在的任务就是解决下面这些依赖的镜像即可。

[root@node1 ~]# docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.17.0             7d54289267dc        5 weeks ago         116MB
k8s.gcr.io/kube-scheduler            v1.17.0             78c190f736b1        5 weeks ago         94.4MB
k8s.gcr.io/kube-apiserver            v1.17.0             0cae8d5cc64c        5 weeks ago         171MB
k8s.gcr.io/kube-controller-manager   v1.17.0             5eb3b7486872        5 weeks ago         161MB
k8s.gcr.io/coredns                   1.6.5               70f311871ae1        2 months ago        41.6MB
k8s.gcr.io/etcd                      3.4.3-0             303ce5db0e90        2 months ago        288MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        2 years ago         742kB

这里我认为应该是最麻烦的一步，方法有很多，简单说几条：

• 使用阿里云的镜像然后修改名称
• 去 dockerhub 捡垃圾
• 给博主点赞留言
• 开动自己的小脑筋…

接着执行启动集群命令，以下只是一段日志，可以看到已经启动成功了。

kubeadm init --kubernetes-version=v1.17.0
......
......
......
[bootstrap-token] Using token: zlkmry.z97kut9e1ezjgetm
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.20.1.103:6443 --token zlkmry.z97kut9e1ezjgetm \
    --discovery-token-ca-cert-hash sha256:ebe4e4f09e46e4b1fff8e6d81fdc9854fb23b64d05258d52e4afbd84136bb4c7

启动节点这一步，如果你修改了主机名却没有配置 hosts 映射就会抛出以下异常，这是正常的，你需要修改映射。

	[WARNING Hostname]: hostname "node4" could not be reached
	[WARNING Hostname]: hostname "node4": lookup node6 on 8.8.8.8:53: no such host

master 节点执行

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

剩余节点加入集群

kubeadm join 10.20.1.103:6443 --token zlkmry.z97kut9e1ezjgetm \
    --discovery-token-ca-cert-hash sha256:ebe4e4f09e46e4b1fff8e6d81fdc9854fb23b64d05258d52e4afbd84136bb4c7

从输出命令我们可以看到，1：我们集群的网络没有初始化，所以每个节点是 NotReady 状态，2：我们两个子节点的镜像没有 pull 下来。集群网络我们使用 calico 在下一步解决，集群镜像我们直接把先前 master 的镜像使用 docker save 命令压缩以下然后直接发送到另外两个子节点使用 docker loca 命令直接解压，这样解决了两个子节点镜像的问题！

[root@localhost ~]# kubectl get nodes
NAME    STATUS     ROLES    AGE    VERSION
node4   NotReady   master   7m6s   v1.17.1
node5   NotReady   <none>   79s    v1.17.1
node6   NotReady   <none>   69s    v1.17.1
[root@localhost ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                            READY   STATUS              RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-6955765f44-qft45        0/1     Pending             0          6m56s   <none>        <none>   <none>           <none>
kube-system   coredns-6955765f44-wtzp2        0/1     Pending             0          6m56s   <none>        <none>   <none>           <none>
kube-system   etcd-node4                      1/1     Running             0          6m43s   10.20.1.103   node4    <none>           <none>
kube-system   kube-apiserver-node4            1/1     Running             0          6m43s   10.20.1.103   node4    <none>           <none>
kube-system   kube-controller-manager-node4   1/1     Running             0          6m43s   10.20.1.103   node4    <none>           <none>
kube-system   kube-proxy-4cx77                0/1     ContainerCreating   0          71s     10.20.1.105   node6    <none>           <none>
kube-system   kube-proxy-lw9xf                1/1     Running             0          6m56s   10.20.1.103   node4    <none>           <none>
kube-system   kube-proxy-zfzqj                0/1     ContainerCreating   0          80s     10.20.1.104   node5    <none>           <none>
kube-system   kube-scheduler-node4            1/1     Running             0          6m43s   10.20.1.103   node4    <none>           <none>

创建集群网络，这里使用的是 calico 网络

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml

到了这一步，你的 k8s 集群应该是已经正常的了，如下所示，下一步就是安装 istio！

[root@localhost ~]# kubectl get nodes
NAME    STATUS   ROLES    AGE   VERSION
node4   Ready    master   23m   v1.17.1
node5   Ready    <none>   17m   v1.17.1
node6   Ready    <none>   17m   v1.17.1
[root@localhost ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP              NODE    NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-5c45f5bd9f-wx2g5   1/1     Running   0          5m16s   192.168.139.2   node6   <none>           <none>
kube-system   calico-node-7wthb                          1/1     Running   0          5m16s   10.20.1.103     node4   <none>           <none>
kube-system   calico-node-fzdcj                          1/1     Running   0          5m16s   10.20.1.105     node6   <none>           <none>
kube-system   calico-node-glf8p                          1/1     Running   0          5m16s   10.20.1.104     node5   <none>           <none>
kube-system   coredns-6955765f44-qft45                   1/1     Running   0          23m     192.168.139.1   node6   <none>           <none>
kube-system   coredns-6955765f44-wtzp2                   1/1     Running   0          23m     192.168.139.3   node6   <none>           <none>
kube-system   etcd-node4                                 1/1     Running   0          23m     10.20.1.103     node4   <none>           <none>
kube-system   kube-apiserver-node4                       1/1     Running   0          23m     10.20.1.103     node4   <none>           <none>
kube-system   kube-controller-manager-node4              1/1     Running   0          23m     10.20.1.103     node4   <none>           <none>
kube-system   kube-proxy-4cx77                           1/1     Running   0          17m     10.20.1.105     node6   <none>           <none>
kube-system   kube-proxy-lw9xf                           1/1     Running   0          23m     10.20.1.103     node4   <none>           <none>
kube-system   kube-proxy-zfzqj                           1/1     Running   0          17m     10.20.1.104     node5   <none>           <none>
kube-system   kube-scheduler-node4                       1/1     Running   0          23m     10.20.1.103     node4   <none>           <none>

istio

step1 下载 istio

下载 istio 稳定版，截止到我安装版本为 1.4.3

curl -L https://istio.io/downloadIstio | sh -

step2 启动 istio

启动 istio，模式为：demo

./istioctl manifest apply --set profile=demo

以下为输出：

- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Tracing...
- Applying manifest for component IngressGateway...
- Applying manifest for component EgressGateway...
- Applying manifest for component Policy...
- Applying manifest for component Galley...
- Applying manifest for component Citadel...
- Applying manifest for component Telemetry...
- Applying manifest for component Injector...
- Applying manifest for component Prometheus...
- Applying manifest for component Pilot...
- Applying manifest for component Kiali...
- Applying manifest for component Grafana...
✔ Finished applying manifest for component Injector.
✔ Finished applying manifest for component Prometheus.
✔ Finished applying manifest for component Galley.
✔ Finished applying manifest for component Tracing.
✔ Finished applying manifest for component Pilot.
✔ Finished applying manifest for component IngressGateway.
✔ Finished applying manifest for component Citadel.
✔ Finished applying manifest for component Policy.
✔ Finished applying manifest for component Kiali.
✔ Finished applying manifest for component EgressGateway.
✔ Finished applying manifest for component Grafana.
✔ Finished applying manifest for component Telemetry.


✔ Installation complete

查看 pod，istio 所需镜像可以直接 pull 下来，不用拐弯抹角的麻烦。这里网速慢得估计需要等个三五分钟 pod 才会启动成功！

[root@localhost ~]# kubectl get pods -n istio-system -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
grafana-6b65874977-dr9gc                  1/1     Running   0          9m36s   192.168.139.8    node6   <none>           <none>
istio-citadel-f78ff689-dvghc              1/1     Running   0          9m38s   192.168.139.5    node6   <none>           <none>
istio-egressgateway-7b6b69ddcd-gzp4n      0/1     Running   0          9m44s   192.168.33.131   node5   <none>           <none>
istio-galley-69674cb559-w7lhq             1/1     Running   0          9m40s   192.168.33.135   node5   <none>           <none>
istio-ingressgateway-649f9646d4-86snz     0/1     Running   0          9m42s   192.168.33.132   node5   <none>           <none>
istio-pilot-7989874664-nspww              0/1     Running   0          9m40s   192.168.139.4    node6   <none>           <none>
istio-policy-5cdbc47674-5xx24             1/1     Running   6          9m38s   192.168.139.7    node6   <none>           <none>
istio-sidecar-injector-7dd87d7989-9984l   1/1     Running   0          9m41s   192.168.33.133   node5   <none>           <none>
istio-telemetry-6dccd56cf4-4h2pd          1/1     Running   7          9m40s   192.168.33.130   node5   <none>           <none>
istio-tracing-c66d67cd9-c8z89             1/1     Running   0          9m42s   192.168.33.129   node5   <none>           <none>
kiali-8559969566-bpnn4                    1/1     Running   0          9m37s   192.168.139.6    node6   <none>           <none>
prometheus-66c5887c86-zg26g               1/1     Running   0          9m41s   192.168.33.134   node5   <none>           <none>

step3 自动注入

为 default 命名空间开启自动注入：

kubectl label namespace default istio-injection=enabled

step4 部署示例

启动 bookinfo 示例，bookinfo 的 yaml 文件我们已经跟随 istio 下载下来了

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

查看结果，启动比较慢需要一两分钟

[root@localhost istio-1.4.3]# kubectl get pod -n default
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-78d78fbddf-zhz6h       2/2     Running   0          5m31s
productpage-v1-596598f447-x4tnb   2/2     Running   0          5m31s
ratings-v1-6c9dbf6b45-9mv64       2/2     Running   0          5m32s
reviews-v1-7bb8ffd9b6-7dtbg       2/2     Running   0          5m32s
reviews-v2-d7d75fff8-bmlkv        2/2     Running   0          5m32s
reviews-v3-68964bc4c8-5sxpr       2/2     Running   0          5m32s

创建入口网关：

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

返回结果：

[root@localhost istio-1.4.3]# kubectl get gateway -n default
NAME               AGE
bookinfo-gateway   6m52s
[root@localhost istio-1.4.3]# kubectl get vs -n default
NAME       GATEWAYS             HOSTS   AGE
bookinfo   [bookinfo-gateway]   [*]     6m57s

创建成功之后，尝试从网页访问：

截止到这里，详细的部署流程已经全部贴出，又遇到什么问题欢迎留言，如果对大家有用的话不要忘记点赞哦！