企业级容器技术 k8s 资源监控的部署和ui界面的部署

metrics-server部署

软件地址https://github.com/kubernetes-sigs/metrics-server
下载比较慢可以通过国内的加速进行下载 码云https://gitee.com/
如何使用这个网站，登陆后找到新建仓库选项，添加仓库名称后，拉到最下面，点击导入已有仓库。切换到github复制Clone or download选项中的网址到码云上面。点击创建，这个网站就复制过来可以进行加速下载了。
再进行克隆的时侯就需要码云上的url

git clone https://gitee.com/yang_ke_xiang/metrics-server.git

克隆时侯需要认证用户名密码是登陆码云上面的用户名密码。
下载完成后进入目录metrics-server/deploy目录下面有两个，针对docker和kubernetes

[kubeadm@server1 deploy]$ ls
docker  kubernetes

进入metrics-server-deployment.yaml文件中修改镜像，提前在私有仓库添加镜像metrics-server-amd64:v0.3.6,其他的不用修改。直接进行部署。

[kubeadm@server1 kubernetes]$ kubectl apply -f .
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

部署完成之后可以先使用命令查看部署的是否可以使用。
下来就开始填坑
部署完成后显示不可用

[kubeadm@server1 kubernetes]$ kubectl top node 
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

1.解析

首先查看api

[kubeadm@server1 kubernetes]$ kubectl get apiservices.apiregistration.k8s.io 
NAME                                   SERVICE                      AVAILABLE                      AGE
v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   5m16s

这里可以看到启动是失败的。所以我们先看看pod日志。

kubectl logs  -n kube-system metrics-server-64475bbf5d-rcw82
lookup server1 on 10.96.0.10:53: no such host]##53端口dns解析的问题
[kubeadm@server1 kubernetes]$ kubectl get svc -n kube-system 
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
kube-dns         ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   40h

应该如何把解析加进去呢？首先查看cm中的coredns读取文件顺序

[kubeadm@server1 kubernetes]$ kubectl get cm -n kube-system 
NAME                                 DATA   AGE
coredns                              1      40h
extension-apiserver-authentication   6      40h
kube-flannel-cfg                     2      40h
kube-proxy                           2      40h
kubeadm-config                       2      40h
kubelet-config-1.17                  1      40h
[kubeadm@server1 kubernetes]$ kubectl edit cm -n kube-system coredns 
forward . /etc/resolv.conf  ##本身读取的dns是转发到这里

在ready下一行进行添加如下解析，这些解析指向真实的ip地址。

hosts {
   192.168.122.2 server1
   192.168.122.4 server3
   192.168.122.5 server4
   fallthrough   ##如果提供不了解析会继续往下读取文件
}

修改完成后再使用top命令进行查看。

[kubeadm@server1 kubernetes]$ kubectl top node 
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

这时候依然报错。

再次查看日志

[kubeadm@server1 kubernetes]$ kubectl logs  -n kube-system metrics-server-64475bbf5d-rcw82
x509: certificate signed by unknown authority]##证书有问题

2.证书

可以在配置文件metrics-server-deployment.yaml中添加一个参数–kubelet-insecure-tls跳过这个检查，但是不推荐，因为不安全。
或者在每个节点的/var/lib/kubelet/config.yaml文件中的最后一行添加。

serverTLSBootstrap: true ##添加自签名证书
[root@server4 ~]# vim /var/lib/kubelet/config.yaml
[root@server4 ~]# systemctl daemon-reload
[root@server4 ~]# systemctl restart kubelet.service

这个时侯请求就过来了，之后我们去要进行签发证书。

[kubeadm@server1 ~]$ kubectl get csr
NAME        AGE     REQUESTOR             CONDITION
csr-5tzf2   5m33s   system:node:server3   Pending
csr-dzv8n   7m8s    system:node:server1   Pending
csr-kntm2   5m1s    system:node:server4   Pending
[kubeadm@server1 ~]$ kubectl certificate approve csr-dzv8n
certificatesigningrequest.certificates.k8s.io/csr-dzv8n approved
[kubeadm@server1 ~]$ kubectl certificate approve csr-5tzf2
certificatesigningrequest.certificates.k8s.io/csr-5tzf2 approved
[kubeadm@server1 ~]$ kubectl certificate approve csr-kntm2
certificatesigningrequest.certificates.k8s.io/csr-kntm2 approved

签发完成再次查看

[kubeadm@server1 ~]$ kubectl get csr
NAME        AGE     REQUESTOR             CONDITION
csr-5tzf2   8m11s   system:node:server3   Approved,Issued
csr-dzv8n   9m46s   system:node:server1   Approved,Issued
csr-kntm2   7m39s   system:node:server4   Approved,Issued        ##签发成功

证书也签发成功再次查看命令能否使用

[kubeadm@server1 ~]$ kubectl top node 
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

依然有问题，继续查看日志

[kubeadm@server1 ~]$ kubectl logs -n kube-system metrics-server-64475bbf5d-nms65
I0304 05:53:24.480910       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0304 05:53:25.546441       1 secure_serving.go:116] Serving securely on [::]:4443

这个是侯pod已经没有问题，下来就应该看看服务的问题。

[kubeadm@server1 ~]$ kubectl describe svc metrics-server -n kube-system 
Name:              metrics-server
Endpoints:         10.244.2.28:4443
[kubeadm@server1 ~]$ kubectl get pod -n kube-system -o wide
metrics-server-64475bbf5d-mfkzt   1/1     Running   0          5m44s   10.244.2.28     server3   <none>           <none>

两个ip地址相同说明已经找到，这么看服务没有问题。
再回来查看我们的api

[kubeadm@server1 ~]$ kubectl -n kube-system get apiservice
v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   71m

服务还是失败的。

[kubeadm@server1 ~]$ kubectl describe -n kube-system apiservice v1beta1.metrics.k8s.io
    Message:               failing or missing response from https://10.108.202.229:443/apis/metrics.k8s.io/v1beta1: Get https://10.108.202.229:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

从这里看应该是网络的问题。

3.网络

从这里看应该是metrics-server-64475bbf5d-mfkzt的网络段和其他的网络段不一样所导致的。

[kubeadm@server1 ~]$ kubectl get pod -n kube-system -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
coredns-9d85f5447-rq9rj           1/1     Running   2          41h   10.244.2.26     server3   <none>           <none>
coredns-9d85f5447-spkdz           1/1     Running   2          41h   10.244.2.25     server3   <none>           <none>
etcd-server1                      1/1     Running   3          41h   192.168.122.2   server1   <none>           <none>
kube-apiserver-server1            1/1     Running   3          41h   192.168.122.2   server1   <none>           <none>
kube-controller-manager-server1   1/1     Running   4          41h   192.168.122.2   server1   <none>           <none>
kube-flannel-ds-amd64-nmhbl       1/1     Running   2          40h   192.168.122.5   server4   <none>           <none>
kube-flannel-ds-amd64-qxz4d       1/1     Running   2          40h   192.168.122.4   server3   <none>           <none>
kube-flannel-ds-amd64-zqs9b       1/1     Running   3          40h   192.168.122.2   server1   <none>           <none>
kube-proxy-4blfr                  1/1     Running   3          41h   192.168.122.2   server1   <none>           <none>
kube-proxy-4p7rg                  1/1     Running   2          41h   192.168.122.5   server4   <none>           <none>
kube-proxy-9n5gp                  1/1     Running   2          41h   192.168.122.4   server3   <none>           <none>
kube-scheduler-server1            1/1     Running   4          41h   192.168.122.2   server1   <none>           <none>
metrics-server-64475bbf5d-mfkzt   1/1     Running   0          13m   10.244.2.28     server3   <none>           <none>

解决方法修改网络
在配置文件metrics-server-deployment.yaml中添加一行hostNetwork: true

  k8s-app: metrics-server
spec:
  hostNetwork: true
  serviceAccountName: metrics-server

再次查看，就发现配置已经生效，ip已经和其他处于同网段。

[kubeadm@server1 kubernetes]$ kubectl get pod -n kube-system -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
metrics-server-7cf4565bc6-9gkpw   1/1     Running   0          2m29s   192.168.122.5   server4   <none>           <none>

这样就可以进行使用了，并且可以进行采集了。

[kubeadm@server1 kubernetes]$ kubectl top node
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
server1   152m         7%     879Mi           50%       
server3   42m          2%     324Mi           18%       
server4   36m          1%     309Mi           17%     

Dashboard部署

从github上面复制文件

https://github.com/kubernetes/dashboard/blob/v2.0.0-rc5/aio/deploy/recommended.yaml

将镜像提前下好，或者直接在网上进行拉取都可以。

[kubeadm@server1 dashboard]$ kubectl create -f deploy.yaml 
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

创建完成后如何访问，如果有图形界面直接使用kubernetes-dashboard的ip进行访问。

[kubeadm@server1 dashboard]$ kubectl get svc -n kubernetes-dashboard 
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dashboard-metrics-scraper   ClusterIP   10.109.74.40    <none>        8000/TCP   20s
kubernetes-dashboard        ClusterIP   10.103.247.85   <none>        443/TCP    21s

如果没有图形界面那么就要将端口暴露出来。将type改为nodeport。

[kubeadm@server1 dashboard]$ kubectl edit svc -n kubernetes-dashboard kubernetes-dashboard
service/kubernetes-dashboard edited
[kubeadm@server1 dashboard]$ kubectl get svc -n kubernetes-dashboard 
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.109.74.40    <none>        8000/TCP        62s
kubernetes-dashboard        NodePort    10.103.247.85   <none>        443:30319/TCP   63s

暴露完成查看kubernetes-dashboard在哪个节点上部署的，那么就用那个节点的ip进行访问。

访问的是侯记得使用https方法，使用证书加密的方式。
进入界面后我们使用token的方式进行登陆。
如何找到token，首先找到对应的sa，因为sa和token是绑定的。

[kubeadm@server1 dashboard]$ kubectl -n kubernetes-dashboard get sa
NAME                   SECRETS   AGE
default                1         10m
kubernetes-dashboard   1         10m
[kubeadm@server1 dashboard]$ kubectl -n kubernetes-dashboard describe sa kubernetes-dashboard
Name:                kubernetes-dashboard
Namespace:           kubernetes-dashboard
Labels:              k8s-app=kubernetes-dashboard
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   kubernetes-dashboard-token-nvd76
Tokens:              kubernetes-dashboard-token-nvd76 ##这里就是我们要找的token
Events:              <none>

接着再使用命令拿到token，将下面的token复制进行登陆。

[kubeadm@server1 dashboard]$ kubectl -n kubernetes-dashboard describe secrets kubernetes-dashboard-token-nvd76
Name:         kubernetes-dashboard-token-nvd76
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
              kubernetes.io/service-account.uid: 2a74d749-0136-41fb-aa01-73c23e9cdc3e

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImhfMlg3TDBmNVFHOGt0UHY4RTVpVDdwNnhwZG9zT3ZjX1otSU1MN2ltcXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1udmQ3NiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjJhNzRkNzQ5LTAxMzYtNDFmYi1hYTAxLTczYzIzZTljZGMzZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.d5qwUlS4OLscaBen8nrDnS75FGT1Kgi0x-ydA3V6TBaaAxnw7iYXfmYOAcd2y2ZU_mDN2cJHpv53LWCzqp5HrapJG9tY8jCtavzbypS_h2mFg7uevbaObtUBPULl6n1kJzkKKcV7i5dffEUsr_XtreO11iZVvac2-Zg9cYO3Tl-1_HLXdm4EpAeSKCLAt1UL8A35NOIk-SdEfVIofyeFmM2Ttfs8WxpKCeahQDquE0-InQC3PpfOe8WkszPw7yzG5C1Ll8Z8xg4vzYqA263UzdAt7rUnM3af8ZjJVC-ezQhT1IHeHmJrI_7t57SBn171lE-wfw55pf2QYidHTZMBKQ
ca.crt:     1025 bytes
namespace:  20 bytes

登陆上来没有什么权限，因为当初的绑定就没有给多少，只有node和pod的权限。所以我们从新设置rbac

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

设置完成刷新一下就可以看到数据。