主节点上k8s 1.10.6的Dashboard和监控组件Heapster的安装和配置(二)

主节点上k8s 1.10.6的Dashboard和监控组件Heapster的安装和配置(二)

一：所有节点上配置 Kubeadm 所用到的镜像

这里是重中之重，因为在国内的原因，无法访问到 Google 的镜像库，所以我们需要执行以下脚本来从 Docker Hub 仓库中获取相同的镜像，并且更改 TAG 让其变成与 Google 拉去镜像一致。
新建一个 Shell 脚本，填入以下代码之后保存。

mkdir  -p  /opt/sh  &&  cd  /opt/sh 
cat  /opt/sh/auto_deploy_k8s.sh
#!/bin/bash
images=(coredns:1.1.3 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.9 k8s-dns-kube-dns-amd64:1.14.9
k8s-dns-dnsmasq-nanny-amd64:1.14.9 heapster-influxdb-amd64:v1.3.3  heapster-amd64:v1.4.2   heapster-grafana-amd64:v4.4.3)


for imageName in ${images[@]} ; do
docker pull keveon/$imageName
docker tag keveon/$imageName k8s.gcr.io/$imageName
docker rmi keveon/$imageName
done

$chmod +x    /opt/sh/auto_deploy_k8s.sh
$/opt/sh/auto_deploy_k8s.sh

二：主节点上 Kubernetes-Dashboard（WebUI）的安装和配置

1：安装dashboard

和网络插件的用法一样，dashboard也是一个容器应用，先下载yaml：

cd     /opt/k8s/config
wget  https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

对该文件进行修改，在该文件末尾，将：

#------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

修改为：

# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  # 添加Service的type为NodePort
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      # 添加映射到虚拟机的端口,k8s只支持30000以上的端口
      nodePort: 30001
  selector:
    k8s-app: kubernetes-dashboard

将:
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
修改为：
image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

如果以前安装过yaml文件的话，需要先使用以下命令进行删除：
$kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
然后再执行安装yaml
$kubectl create -f /opt/k8s/config/kubernetes-dashboard.yaml

安装完成后，执行：

 $kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
kube-system   coredns-7997f8864c-gnvd5                   1/1       Running   0          1h
kube-system   coredns-7997f8864c-svl7v                   1/1       Running   0          1h
kube-system   kube-apiserver-k8s-m1.trjcn.com            1/1       Running   0          1h
kube-system   kube-apiserver-k8s-m2.trjcn.com            1/1       Running   0          1h
kube-system   kube-apiserver-k8s-m3.trjcn.com            1/1       Running   0          1h
kube-system   kube-controller-manager-k8s-m1.trjcn.com   1/1       Running   0          1h
kube-system   kube-controller-manager-k8s-m2.trjcn.com   1/1       Running   0          1h
kube-system   kube-controller-manager-k8s-m3.trjcn.com   1/1       Running   0          1h
kube-system   kube-flannel-ds-f8647                      1/1       Running   0          1h
kube-system   kube-flannel-ds-k6scr                      1/1       Running   0          49m
kube-system   kube-flannel-ds-l8hwz                      1/1       Running   1          1h
kube-system   kube-flannel-ds-v5ht6                      1/1       Running   0          1h
kube-system   kube-proxy-78ht6                           1/1       Running   0          1h
kube-system   kube-proxy-7wsl7                           1/1       Running   0          49m
kube-system   kube-proxy-9xlds                           1/1       Running   0          1h
kube-system   kube-proxy-p5fp8                           1/1       Running   0          1h
kube-system   kube-scheduler-k8s-m1.trjcn.com            1/1       Running   0          1h
kube-system   kube-scheduler-k8s-m2.trjcn.com            1/1       Running   0          1h
kube-system   kube-scheduler-k8s-m3.trjcn.com            1/1       Running   0          1h
kube-system   kubernetes-dashboard-7d5dcdb6d9-kvxhz      1/1       Running   0          27s

打开WebUI：
https://192.168.10.110:30001
见如下页面：

2： 配置角色权限并赋权然后登陆WebUI

我们创建一个admin用户并授予admin 角色绑定，使用下面的yaml文件创建admin用户并赋予他管理员权限，然后就可以通过token 登陆dashbaord，这种认证方式本质实际上是通过Service Account 的身份认证加上Bearer token请求 API server 的方式实现。

mkdir -p  /opt/k8s/config
cat     /opt/k8s/config/user-admin.yaml

内容如下：

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

执行以上写好的配置文件：
$kubectl create -f /opt/k8s/config/user-admin.yaml

上面的admin用户创建完成后我们就可以获取到该用户对应的token了，如下命令：

$kubectl get secret -n kube-system | grep admin
admin-token-jvwk5                                kubernetes.io/service-account-token   3         26s

$kubectl     describe     secret     admin-token-jvwk5     -n     kube-system 
输出如下：
Name:         admin-token-5bf9b
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=admin
              kubernetes.io/service-account.uid=da7fe0df-97a0-11e8-981f-000c2906f499

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi1jd2o3NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY5MDliNGI5LWIwYTktMTFlOC1iNDQ3LTAwMGMyOTA3OTg4NiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbiJ9.T8kRnhiDtrKNnXklkwuBwg0DhtC2BRLMnGVuPh6lpsNulv-_-rwIQm7h9KSvG-ZulnluMhvZVp-py7_8H7uRZRmpiPM-yDkIYhGmyJJXWH88tq44ZNxAEjbkwbcxeOAD4i1Zbu0A-_8OwRUKKfPnHw0GvdS4VexkIGdIVNblRhVuhg3qOfhHiMEBQ59N9JZYK3yvkVNoNzBTVqGNK95s7a5kevsV_rYaV8T6QslWsGu0R89xzGBR73VBzUUESHwDElnVons3aPadRVD4d_JPuhKF8BCaFi68ZSAOeEG7jyahavcOTobGo7csHmDrflOKDOVonLss83Vr2a79R987Dw

然后在https://192.168.10.110:30001界面上输入用户以上生成的token即可登陆；注意！！这里必须使用firefox火狐浏览器登陆，google浏览器和IE登陆后无法跳转。

三：监控组件Heapster+influxdb+grafana的安装配置

heapster是一个监控计算、存储、网络等集群资源的工具，以k8s内置的cAdvisor作为数据源收集集群信息，并汇总出有价值的性能数据(Metrics)：cpu、内存、network、filesystem等，然后将这些数据输出到外部存储(backend)，如InfluxDB，最后再通过相应的UI界面进行可视化展示，如grafana。 另外heapster的数据源和外部存储都是可插拔的，所以可以很灵活的组建出很多监控方案，如：Heapster+ElasticSearch+Kibana等等。
Heapster的整体架构图：

本次测试使用的各软件版本如下：

1：influxdb                v1.3.3版本                      rpm包方式安装
2：grafana                 v4.4.3版本                      rpm包方式安装
3：heapster                docker 镜像使用v1.5.0版本           yaml文件使用docker镜像方式安装

3.1：主节点上安装配置influxdb

3.1.1：主节点上安装influxdb

mkdir -p /opt/k8s/rpm && cd /opt/k8s/rpm
wget https://repos.influxdata.com/rhel/7Server/x86_64/stable/influxdb-1.3.3.x86_64.rpm
rpm -ivh influxdb-1.3.3.x86_64.rpm

3.1.2：修改InfluxDB的配置，主要配置jmeter存储的数据库与端口号，还有需要将UI端口开放：

vi /etc/influxdb/influxdb.conf 修改如下位置：

#设置绑定的端口
Bind address to use for the RPC service for backup and restore.
bind-address = "127.0.0.1:8088"

找到graphite并且修改它的库与端口

[[graphite]]
  # Determines whether the graphite endpoint is enabled.
  enabled = true
  database = "graphite"
  retention-policy = ""
  bind-address = ":2003"
  protocol = "tcp"
  consistency-level = "one"

找到http，将前面的#号去掉，开放它的UI端口

[http]
  # Determines whether HTTP endpoint is enabled.
  enabled = true

  # The bind address used by the HTTP service.
  bind-address = ":8086"

  # Determines whether HTTPS is enabled.
  https-enabled = false

3.1.3：启动influxdb

$systemctl enable influxdb && systemctl start influxdb

3.2：主节点上安装配置grafana

3.2.1：主节点上安装grafana

cd /opt/k8s/rpm
yum -y install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.4.3-1.x86_64.rpm
systemctl enable grafana-server && systemctl start grafana-server

浏览器打开, http://192.168.10.110:3000
默认admin/admin

3.3：主节点上安装配置Heapster

3.3.1：下载Heapster

docker pull daocloud.io/liukuan73/heapster-amd64:v1.5.2

3.3.2：部署使用

cd   /opt/k8s/config/
wget   https://raw.githubusercontent.com/liukuan73/kubernetes-addons/master/monitor/heapster%2Binfluxdb%2Bgrafana/heapster.yaml

修改该文件内容成如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: daocloud.io/liukuan73/heapster-amd64:v1.5.2
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://192.168.10.115:6443?inClusterConfig=false&insecure=true&kubeletHttps=true&kubeletPort=10250
        #- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
        - --sink=influxdb:http://192.168.10.110:8086
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 8082
    targetPort: 8082
  selector:
    k8s-app: heapster

然后应用这2个文件：
kubectl create -f /opt/k8s/config/heapster.yaml

然后执行以下命令进行角色绑定：

kubectl create clusterrolebinding  heapster-clusterrolebing   --clusterrole=cluster-admin --user=system:anonymous  --namespace=kube-system

重要：

如果不执行这条命令，那么kubectl logs -f heapster-54b54dd8fb-d2p27 -n kube-system命令执行后，日志中会报以下错误：

E0905 11:57:02.694251       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:anonymous" cannot list nodes at the cluster scope
E0905 11:57:02.694990       1 reflector.go:190] k8s.io/heapster/metrics/heapster.go:328: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list pods at the cluster scope
E0905 11:57:02.696373       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:anonymous" cannot list nodes at the cluster scope
E0905 11:57:02.697472       1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:anonymous" cannot list nodes at the cluster scope
E0905 11:57:02.698596       1 reflector.go:190] k8s.io/heapster/metrics/processors/namespace_based_enricher.go:89: Failed to list *v1.Namespace: namespaces is forbidden: User "system:anonymous" cannot list namespaces at the cluster scope

3.3.3：完成效果展示

3.3.3.1：查看集群信息

$kubectl cluster-info
Kubernetes master is running at https://192.168.10.115:6443
Heapster is running at https://192.168.10.115:6443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://192.168.10.115:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.

3.3.3.2：通过NodePort访问

$kubectl  get  svc   -n  kube-system
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
heapster               ClusterIP   10.102.40.213   <none>        8082/TCP        42s
kube-dns               ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP   1h
kubernetes-dashboard   NodePort    10.98.50.50     <none>        443:30001/TCP   34m

3.3.3.3：通过dashboard的容器组中可以看到容器cpu和内存的使用情况

3.3.4：graphs图形配置
通过 http://192.168.10.110:3000 可以打开grafana，然后在左上角选择data sources，将data sources页面配置成如下：

然后下载grafana所需要的图形化模板：
node节点监控模板：https://grafana.com/dashboards/3649 ；
pod节点监控模板： https://grafana.com/dashboards/3646 ；

然后在guafana的主界面左上角选择”DashBoard”—-> “import”菜单，打开import dashboard界面如下：

然后将下载好的模板json文件上传上去即可，完成后效果图如下：