使用nginx统一代理dashboard,grafana,Prometheus二级目录访问

k8s上的这些管理工具必不可少，可以统一在nginx下的二级目录下。

ingress是好，但我们不方便使用内部域名，相信么。。。：）

一，prometheus改造

在prometheus的deployment中传递一下—web.external-url参数。如下所示：

- name: prometheus
        image: xxx/3rd_part/prometheus:v2.4.3
        imagePullPolicy: IfNotPresent
        args:
          - '--storage.tsdb.path=/prometheus/data/'
          - '--storage.tsdb.retention=1d'
          - '--config.file=/etc/prometheus/prometheus.yaml'
          - '--web.enable-lifecycle'
          - '--web.external-url=/prometheus'
        ports:
        - name: webui
          containerPort: 9090
        resources:
          requests:
            cpu: 400m
            memory: 500M
        #  limits:
        #    cpu: 500m
        #    memory: 500M

二，grafana改造

在grafana的deployment中，env环境变量更新GF_SERVER_ROOT_URL的值。如下所示：

containers:
      - name: grafana
        image: xxx/3rd_part/grafana/grafana:5.3.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 3000
          name: grafana
        env:
        - name: GF_SECURITY_ADMIN_USER
          value: [usr]
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: [pwd]
        - name: GF_SERVER_ROOT_URL
          value: "%(protocol)s://%(domain)s:%(http_port)s/grafana"
        resources:
          limits:
            cpu: 100m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 256Mi

三，dashboard改造

dashboard不再使用443端口，使用最简单的方式来达到目的。其yaml文件如下所示：

# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system

---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1beta2
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
        ports:
        - containerPort: 9090
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

---
# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 9090
      nodePort: 3xxxxx
  selector:
    k8s-app: kubernetes-dashboard

四，nginx配置

1，  用htpasswd命令生成包含用户名和加密的密码文件nginx_passwd。

2，  将nginx配置和nginx_passwd作成configmap文件。

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-nginx
  namespace: kube-system
data:
  default.conf: |-
    upstream prometheus {
        server prometheus:9090;
    }
    upstream grafana {
        server monitoring-grafana:80;
    }
    upstream dashboard {
        server [master_ip]:[3xxxx];
    }
    server {
        listen       80;
        server_name  localhost;


        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        location /check {
            default_type text/plain;
            return 200 "serving is ok!";
        }

        location /status {
            stub_status on;
            access_log off;
        }
        
        location  /prometheus {
            proxy_pass http://prometheus;
proxy_set_header   Host $host;
        }
     
        location /grafana {
            proxy_pass http://grafana;
            rewrite  ^/grafana/(.*)  /$1 break;
            proxy_set_header   Host $host;
        }
   
        location /dashboard {
            auth_basic            "Password please";
            auth_basic_user_file  /etc/nginx/conf.d/nginx_passwd;
            proxy_pass  http://dashboard;
            rewrite  ^/dashboard/(.*)  /$1 break;
            proxy_set_header   Host $host;
        }
        # redirect server error pages to the static page /50x.html
        # chengang from k8s config map file
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

    }
  nginx_passwd: |-
        user:${password}

3,  制作nginx的deployment和service文件。

挂载了nginx的configmap文件。在其中加了一个[3xxxx]端口，这就是其它应用的入口。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: proxy-nginx
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: proxy-nginx
    spec:
      containers:
      - name: nginx
        image: xxx/official_hub/nginx:1.13-alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
        - name: nginx-conf
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-conf
        configMap:
          name: proxy-nginx
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  name: proxy-nginx
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: [3xxxx]
  selector:
        k8s-app: proxy-nginx

五，测试URL

 

http://[master_ip:3xxxx]/grafana
http://[master_ip:3xxxx]/prometheus
http://[master_ip:3xxxx]/dashboard