12 kubernetes 监控(三)

监控博客应用

给 kubeblog springboot 应用添加监控项

vim pom.xml

        
            org.springframework.boot
            spring-boot-starter-actuator
        
        
            io.micrometer
            micrometer-registry-prometheus
        

重新打包

mvn package

登录 jcr

docker login 192.168.190.131:8081

重新构建镜像

docker build -t 192.168.190.131:8081/docker-local/kubeblog:1.2 .

添加 latest 标签

docker tag 192.168.190.131:8081/docker-local/kubeblog:1.2 192.168.190.131:8081/docker-local/kubeblog:latest

推送到 jcr

docker push 192.168.190.131:8081/docker-local/kubeblog:1.2
docker push 192.168.190.131:8081/docker-local/kubeblog:latest

更新 helm chart

修改 value.yaml 文件中的 image 的 tag 为 latest

打包 helm chart

helm package kubeblog

部署新版本 helm chart

helm install kubeblog kubeblog-0.1.2.tgz

访问博客应用:http://192.168.190.132:30002/

博客应用监控数据:http://192.168.190.132:30002/actuator/prometheus

prometheus 监控博客应用

清理命名空间

kubectl delete ns monitoring

开始

看一下现在有什么

kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
kubeblog-7d9b64cb96-rz6cc   1/1     Running   2          15m
kubeblog-7d9b64cb96-v94kx   1/1     Running   6          15m
kubeblog-7d9b64cb96-zt5sv   1/1     Running   3          15m
mysql-0                     1/1     Running   0          15m

部署 prometheus

vim springboot-all-in-one.yml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-k8s
  namespace: monitoring
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
    component: core
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort
  ports:
    - port: 9090
      protocol: TCP
      name: webui
      nodePort: 30091
  selector:
    app: prometheus
    component: core
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubeblog-config
  namespace: monitoring
data:
  prometheus.yaml: |
    global:
      scrape_interval:     10s 
      evaluation_interval: 10s 
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
    rule_files:
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
        - targets: ['192.168.190.132:30091']
      - job_name: 'spring-actuator'
        metrics_path: '/actuator/prometheus'
        scrape_interval: 5s
        static_configs:
        - targets: ['192.168.190.132:30002']
---
apiVersion: v1
kind: Secret
data:
  admin-password: YWRtaW4=
  admin-username: YWRtaW4=
metadata:
  name: grafana
  namespace: monitoring
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-core
  namespace: monitoring
  labels:
    app: prometheus
    component: core
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: core
  template:
    metadata:
      name: prometheus-main
      labels:
        app: prometheus
        component: core
    spec:
      serviceAccountName: prometheus-k8s
      containers:
      - name: prometheus
        image: prom/prometheus:v1.7.0
        args:
          - '-storage.local.retention=12h'
          - '-storage.local.memory-chunks=500000'
          - '-config.file=/etc/prometheus/prometheus.yaml'
          - '-alertmanager.url=http://alertmanager:9093/'
        ports:
        - name: webui
          containerPort: 9090
        resources:
          requests:
            cpu: 500m
            memory: 500M
          limits:
            cpu: 500m
            memory: 500M
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus
      volumes:
      - name: config-volume
        configMap:
          name: kubeblog-config

kubectl apply -f springboot-all-in-one.yml

可以看到

namespace/monitoring created
clusterrolebinding.rbac.authorization.k8s.io/prometheus unchanged
clusterrole.rbac.authorization.k8s.io/prometheus unchanged
serviceaccount/prometheus-k8s created
service/prometheus created
configmap/kubeblog-config created
secret/grafana created
deployment.apps/prometheus-core created

部署 Grafana

vim grafana-all-in-one.yml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: grafana-import-dashboards
  namespace: monitoring
data:
  grafana-net-2-dashboard.json: |
    {
      "__inputs": [{
        "name": "DS_PROMETHEUS",
        "label": "Prometheus",
        "description": "",
        "type": "datasource",
        "pluginId": "prometheus",
        "pluginName": "Prometheus"
      }],
      "__requires": [{
        "type": "panel",
        "id": "singlestat",
        "name": "Singlestat",
        "version": ""
      }, {
        "type": "panel",
        "id": "text",
        "name": "Text",
        "version": ""
      }, {
        "type": "panel",
        "id": "graph",
        "name": "Graph",
        "version": ""
      }, {
        "type": "grafana",
        "id": "grafana",
        "name": "Grafana",
        "version": "3.1.0"
      }, {
        "type": "datasource",
        "id": "prometheus",
        "name": "Prometheus",
        "version": "1.0.0"
      }],
      "id": null,
      "title": "Prometheus Stats",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": true,
      "sharedCrosshair": false,
      "rows": [{
        "collapse": false,
        "editable": true,
        "height": 178,
        "panels": [{
          "cacheTimeout": null,
          "colorBackground": false,
          "colorValue": false,
          "colors": ["rgba(245, 54, 54, 0.9)", "rgba(237, 129, 40, 0.89)", "rgba(50, 172, 45, 0.97)"],
          "datasource": "${DS_PROMETHEUS}",
          "decimals": 1,
          "editable": true,
          "error": false,
          "format": "s",
          "id": 5,
          "interval": null,
          "links": [],
          "maxDataPoints": 100,
          "nullPointMode": "connected",
          "nullText": null,
          "postfix": "",
          "postfixFontSize": "50%",
          "prefix": "",
          "prefixFontSize": "50%",
          "span": 3,
          "sparkline": {
            "fillColor": "rgba(31, 118, 189, 0.18)",
            "full": false,
            "lineColor": "rgb(31, 120, 193)",
            "show": false
          },
          "targets": [{
            "expr": "(time() - process_start_time_seconds{job=\"prometheus\"})",
            "intervalFactor": 2,
            "refId": "A",
            "step": 4
          }],
          "thresholds": "",
          "title": "Uptime",
          "type": "singlestat",
          "valueFontSize": "80%",
          "valueMaps": [{
            "op": "=",
            "text": "N/A",
            "value": "null"
          }],
          "valueName": "current",
          "mappingTypes": [{
            "name": "value to text",
            "value": 1
          }, {
            "name": "range to text",
            "value": 2
          }],
          "rangeMaps": [{
            "from": "null",
            "to": "null",
            "text": "N/A"
          }],
          "mappingType": 1,
          "gauge": {
            "show": false,
            "minValue": 0,
            "maxValue": 100,
            "thresholdMarkers": true,
            "thresholdLabels": false
          }
        }, {
          "cacheTimeout": null,
          "colorBackground": false,
          "colorValue": false,
          "colors": ["rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)"],
          "datasource": "${DS_PROMETHEUS}",
          "editable": true,
          "error": false,
          "format": "none",
          "id": 6,
          "interval": null,
          "links": [],
          "maxDataPoints": 100,
          "nullPointMode": "connected",
          "nullText": null,
          "postfix": "",
          "postfixFontSize": "50%",
          "prefix": "",
          "prefixFontSize": "50%",
          "span": 3,
          "sparkline": {
            "fillColor": "rgba(31, 118, 189, 0.18)",
            "full": false,
            "lineColor": "rgb(31, 120, 193)",
            "show": true
          },
          "targets": [{
            "expr": "prometheus_local_storage_memory_series",
            "intervalFactor": 2,
            "refId": "A",
            "step": 4
          }],
          "thresholds": "1,5",
          "title": "Local Storage Memory Series",
          "type": "singlestat",
          "valueFontSize": "70%",
          "valueMaps": [],
          "valueName": "current",
          "mappingTypes": [{
            "name": "value to text",
            "value": 1
          }, {
            "name": "range to text",
            "value": 2
          }],
          "rangeMaps": [{
            "from": "null",
            "to": "null",
            "text": "N/A"
          }],
          "mappingType": 1,
          "gauge": {
            "show": false,
            "minValue": 0,
            "maxValue": 100,
            "thresholdMarkers": true,
            "thresholdLabels": false
          }
        }, {
          "cacheTimeout": null,
          "colorBackground": false,
          "colorValue": true,
          "colors": ["rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)"],
          "datasource": "${DS_PROMETHEUS}",
          "editable": true,
          "error": false,
          "format": "none",
          "id": 7,
          "interval": null,
          "links": [],
          "maxDataPoints": 100,
          "nullPointMode": "connected",
          "nullText": null,
          "postfix": "",
          "postfixFontSize": "50%",
          "prefix": "",
          "prefixFontSize": "50%",
          "span": 3,
          "sparkline": {
            "fillColor": "rgba(31, 118, 189, 0.18)",
            "full": false,
            "lineColor": "rgb(31, 120, 193)",
            "show": true
          },
          "targets": [{
            "expr": "prometheus_local_storage_indexing_queue_length",
            "intervalFactor": 2,
            "refId": "A",
            "step": 4
          }],
          "thresholds": "500,4000",
          "title": "Internal Storage Queue Length",
          "type": "singlestat",
          "valueFontSize": "70%",
          "valueMaps": [{
            "op": "=",
            "text": "Empty",
            "value": "0"
          }],
          "valueName": "current",
          "mappingTypes": [{
            "name": "value to text",
            "value": 1
          }, {
            "name": "range to text",
            "value": 2
          }],
          "rangeMaps": [{
            "from": "null",
            "to": "null",
            "text": "N/A"
          }],
          "mappingType": 1,
          "gauge": {
            "show": false,
            "minValue": 0,
            "maxValue": 100,
            "thresholdMarkers": true,
            "thresholdLabels": false
          }
        }, {
          "content": "\"Prometheus\nPrometheus\n\n

You're using Prometheus, an open-source systems monitoring and alerting toolkit originally built at SoundCloud. For more information, check out the Grafana and Prometheus projects.

", "editable": true, "error": false, "id": 9, "links": [], "mode": "html", "span": 3, "style": {}, "title": "", "transparent": true, "type": "text" }], "title": "New row" }, { "collapse": false, "editable": true, "height": 227, "panels": [{ "aliasColors": { "prometheus": "#C15C17", "{instance=\"localhost:9090\",job=\"prometheus\"}": "#C15C17" }, "bars": false, "datasource": "${DS_PROMETHEUS}", "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 3, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 2, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 9, "stack": false, "steppedLine": false, "targets": [{ "expr": "rate(prometheus_local_storage_ingested_samples_total[5m])", "interval": "", "intervalFactor": 2, "legendFormat": "{{job}}", "metric": "", "refId": "A", "step": 2 }], "timeFrom": null, "timeShift": null, "title": "Samples ingested (rate-5m)", "tooltip": { "shared": true, "value_type": "cumulative", "ordering": "alphabetical", "msResolution": false }, "type": "graph", "yaxes": [{ "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }, { "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }], "xaxis": { "show": true } }, { "content": "#### Samples Ingested\nThis graph displays the count of samples ingested by the Prometheus server, as measured over the last 5 minutes, per time series in the range vector. When troubleshooting an issue on IRC or Github, this is often the first stat requested by the Prometheus team. ", "editable": true, "error": false, "id": 8, "links": [], "mode": "markdown", "span": 2.995914043583536, "style": {}, "title": "", "transparent": true, "type": "text" }], "title": "New row" }, { "collapse": false, "editable": true, "height": "250px", "panels": [{ "aliasColors": { "prometheus": "#F9BA8F", "{instance=\"localhost:9090\",interval=\"5s\",job=\"prometheus\"}": "#F9BA8F" }, "bars": false, "datasource": "${DS_PROMETHEUS}", "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 2, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 5, "stack": false, "steppedLine": false, "targets": [{ "expr": "rate(prometheus_target_interval_length_seconds_count[5m])", "intervalFactor": 2, "legendFormat": "{{job}}", "refId": "A", "step": 2 }], "timeFrom": null, "timeShift": null, "title": "Target Scrapes (last 5m)", "tooltip": { "shared": true, "value_type": "cumulative", "ordering": "alphabetical", "msResolution": false }, "type": "graph", "yaxes": [{ "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }, { "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }], "xaxis": { "show": true } }, { "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 14, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 4, "stack": false, "steppedLine": false, "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile!=\"0.01\", quantile!=\"0.05\"}", "interval": "", "intervalFactor": 2, "legendFormat": "{{quantile}} ({{interval}})", "metric": "", "refId": "A", "step": 2 }], "timeFrom": null, "timeShift": null, "title": "Scrape Duration", "tooltip": { "shared": true, "value_type": "cumulative", "ordering": "alphabetical", "msResolution": false }, "type": "graph", "yaxes": [{ "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }, { "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }], "xaxis": { "show": true } }, { "content": "#### Scrapes\nPrometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. Target scrapes will show how frequently targets are scraped, as measured over the last 5 minutes, per time series in the range vector. Scrape Duration will show how long the scrapes are taking, with percentiles available as series. ", "editable": true, "error": false, "id": 11, "links": [], "mode": "markdown", "span": 3, "style": {}, "title": "", "transparent": true, "type": "text" }], "title": "New row" }, { "collapse": false, "editable": true, "height": "250px", "panels": [{ "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": null, "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 12, "legend": { "alignAsTable": false, "avg": false, "current": false, "hideEmpty": true, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 9, "stack": false, "steppedLine": false, "targets": [{ "expr": "prometheus_evaluator_duration_milliseconds{quantile!=\"0.01\", quantile!=\"0.05\"}", "interval": "", "intervalFactor": 2, "legendFormat": "{{quantile}}", "refId": "A", "step": 2 }], "timeFrom": null, "timeShift": null, "title": "Rule Eval Duration", "tooltip": { "shared": true, "value_type": "cumulative", "ordering": "alphabetical", "msResolution": false }, "type": "graph", "yaxes": [{ "show": true, "min": null, "max": null, "logBase": 1, "format": "percentunit", "label": "" }, { "show": true, "min": null, "max": null, "logBase": 1, "format": "short" }], "xaxis": { "show": true } }, { "content": "#### Rule Evaluation Duration\nThis graph panel plots the duration for all evaluations to execute. The 50th percentile, 90th percentile and 99th percentile are shown as three separate series to help identify outliers that may be skewing the data.", "editable": true, "error": false, "id": 15, "links": [], "mode": "markdown", "span": 3, "style": {}, "title": "", "transparent": true, "type": "text" }], "title": "New row" }], "time": { "from": "now-5m", "to": "now" }, "timepicker": { "now": true, "refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d"], "time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d"] }, "templating": { "list": [] }, "annotations": { "list": [] }, "refresh": false, "schemaVersion": 12, "version": 0, "links": [{ "icon": "info", "tags": [], "targetBlank": true, "title": "Grafana Docs", "tooltip": "", "type": "link", "url": "http://www.grafana.org/docs" }, { "icon": "info", "tags": [], "targetBlank": true, "title": "Prometheus Docs", "type": "link", "url": "http://prometheus.io/docs/introduction/overview/" }], "gnetId": 2, "description": "The official, pre-built Prometheus Stats Dashboard." } grafana-net-737-dashboard.json: | { "__inputs": [{ "name": "DS_PROMETHEUS", "label": "prometheus", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" }], "__requires": [{ "type": "panel", "id": "singlestat", "name": "Singlestat", "version": "" }, { "type": "panel", "id": "graph", "name": "Graph", "version": "" }, { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "3.1.0" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }], "id": null, "title": "Kubernetes Pod Resources", "description": "Shows resource usage of Kubernetes pods.", "tags": [ "kubernetes" ], "style": "dark", "timezone": "browser", "editable": true, "hideControls": false, "sharedCrosshair": false, "rows": [{ "collapse": false, "editable": true, "height": "250px", "panels": [{ "cacheTimeout": null, "colorBackground": false, "colorValue": true, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "editable": true, "error": false, "format": "percent", "gauge": { "maxValue": 100, "minValue": 0, "show": true, "thresholdLabels": false, "thresholdMarkers": true }, "height": "180px", "id": 4, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 4, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum (container_memory_working_set_bytes{id=\"/\",instance=~\"^$instance$\"}) / sum (machine_memory_bytes{instance=~\"^$instance$\"}) * 100", "interval": "", "intervalFactor": 2, "legendFormat": "", "refId": "A", "step": 2 }], "thresholds": "65, 90", "timeFrom": "1m", "timeShift": null, "title": "Memory Working Set", "transparent": false, "type": "singlestat", "valueFontSize": "80%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": true, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "percent", "gauge": { "maxValue": 100, "minValue": 0, "show": true, "thresholdLabels": false, "thresholdMarkers": true }, "height": "180px", "id": 6, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 4, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum(rate(container_cpu_usage_seconds_total{id=\"/\",instance=~\"^$instance$\"}[1m])) / sum (machine_cpu_cores{instance=~\"^$instance$\"}) * 100", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "65, 90", "timeFrom": "1m", "timeShift": null, "title": "Cpu Usage", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": true, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "percent", "gauge": { "maxValue": 100, "minValue": 0, "show": true, "thresholdLabels": false, "thresholdMarkers": true }, "height": "180px", "id": 7, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 4, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum(container_fs_usage_bytes{id=\"/\",instance=~\"^$instance$\"}) / sum(container_fs_limit_bytes{id=\"/\",instance=~\"^$instance$\"}) * 100", "interval": "10s", "intervalFactor": 1, "legendFormat": "", "metric": "", "refId": "A", "step": 10 }], "thresholds": "65, 90", "timeFrom": "1m", "timeShift": null, "title": "Filesystem Usage", "type": "singlestat", "valueFontSize": "80%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "bytes", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 9, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "20%", "prefix": "", "prefixFontSize": "20%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum(container_memory_working_set_bytes{id=\"/\",instance=~\"^$instance$\"})", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "title": "Used", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "bytes", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 10, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum (machine_memory_bytes{instance=~\"^$instance$\"})", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "title": "Total", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 11, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": " cores", "postfixFontSize": "30%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum (rate (container_cpu_usage_seconds_total{id=\"/\",instance=~\"^$instance$\"}[1m]))", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "timeShift": null, "title": "Used", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "none", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 12, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": " cores", "postfixFontSize": "30%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum (machine_cpu_cores{instance=~\"^$instance$\"})", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "title": "Total", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "bytes", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 13, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum(container_fs_usage_bytes{id=\"/\",instance=~\"^$instance$\"})", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "title": "Used", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "cacheTimeout": null, "colorBackground": false, "colorValue": false, "colors": [ "rgba(50, 172, 45, 0.97)", "rgba(237, 129, 40, 0.89)", "rgba(245, 54, 54, 0.9)" ], "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "format": "bytes", "gauge": { "maxValue": 100, "minValue": 0, "show": false, "thresholdLabels": false, "thresholdMarkers": true }, "height": "1px", "hideTimeOverride": true, "id": 14, "interval": null, "isNew": true, "links": [], "mappingType": 1, "mappingTypes": [{ "name": "value to text", "value": 1 }, { "name": "range to text", "value": 2 }], "maxDataPoints": 100, "nullPointMode": "connected", "nullText": null, "postfix": "", "postfixFontSize": "50%", "prefix": "", "prefixFontSize": "50%", "rangeMaps": [{ "from": "null", "text": "N/A", "to": "null" }], "span": 2, "sparkline": { "fillColor": "rgba(31, 118, 189, 0.18)", "full": false, "lineColor": "rgb(31, 120, 193)", "show": false }, "targets": [{ "expr": "sum (container_fs_limit_bytes{id=\"/\",instance=~\"^$instance$\"})", "interval": "10s", "intervalFactor": 1, "refId": "A", "step": 10 }], "thresholds": "", "timeFrom": "1m", "title": "Total", "type": "singlestat", "valueFontSize": "50%", "valueMaps": [{ "op": "=", "text": "N/A", "value": "null" }], "valueName": "current" }, { "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)", "thresholdLine": false }, "height": "200px", "id": 32, "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": 200, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 12, "stack": false, "steppedLine": false, "targets": [{ "expr": "sum(rate(container_network_receive_bytes_total{instance=~\"^$instance$\",namespace=~\"^$namespace$\"}[1m]))", "interval": "", "intervalFactor": 2, "legendFormat": "receive", "metric": "network", "refId": "A", "step": 240 }, { "expr": "- sum(rate(container_network_transmit_bytes_total{instance=~\"^$instance$\",namespace=~\"^$namespace$\"}[1m]))", "interval": "", "intervalFactor": 2, "legendFormat": "transmit", "metric": "network", "refId": "B", "step": 240 }], "timeFrom": null, "timeShift": null, "title": "Network", "tooltip": { "msResolution": false, "shared": true, "sort": 0, "value_type": "cumulative" }, "transparent": false, "type": "graph", "xaxis": { "show": true }, "yaxes": [{ "format": "Bps", "label": "transmit / receive", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "Bps", "label": null, "logBase": 1, "max": null, "min": null, "show": false }] }], "showTitle": true, "title": "all pods" }, { "collapse": false, "editable": true, "height": "250px", "panels": [{ "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": 3, "editable": true, "error": false, "fill": 0, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "height": "", "id": 17, "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "connected", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 12, "stack": false, "steppedLine": false, "targets": [{ "expr": "sum(rate(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\"}[1m])) by (pod_name)", "interval": "", "intervalFactor": 2, "legendFormat": "{{ pod_name }}", "metric": "container_cpu", "refId": "A", "step": 240 }], "timeFrom": null, "timeShift": null, "title": "Cpu Usage", "tooltip": { "msResolution": true, "shared": false, "sort": 2, "value_type": "cumulative" }, "transparent": false, "type": "graph", "xaxis": { "show": true }, "yaxes": [{ "format": "none", "label": "cores", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": false }] }, { "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "fill": 0, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 33, "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 12, "stack": false, "steppedLine": false, "targets": [{ "expr": "sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",instance=~\"^$instance$\",namespace=~\"^$namespace$\"}) by (pod_name)", "interval": "", "intervalFactor": 2, "legendFormat": "{{ pod_name }}", "metric": "", "refId": "A", "step": 240 }], "timeFrom": null, "timeShift": null, "title": "Memory Working Set", "tooltip": { "msResolution": false, "shared": false, "sort": 2, "value_type": "cumulative" }, "type": "graph", "xaxis": { "show": true }, "yaxes": [{ "format": "bytes", "label": "used", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": false }] }, { "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 16, "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": 200, "sort": "avg", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 12, "stack": false, "steppedLine": false, "targets": [{ "expr": "sum (rate (container_network_receive_bytes_total{image!=\"\",name=~\"^k8s_.*\",instance=~\"^$instance$\",namespace=~\"^$namespace$\"}[1m])) by (pod_name)", "interval": "", "intervalFactor": 2, "legendFormat": "{{ pod_name }} < in", "metric": "network", "refId": "A", "step": 240 }, { "expr": "- sum (rate (container_network_transmit_bytes_total{image!=\"\",name=~\"^k8s_.*\",instance=~\"^$instance$\",namespace=~\"^$namespace$\"}[1m])) by (pod_name)", "interval": "", "intervalFactor": 2, "legendFormat": "{{ pod_name }} > out", "metric": "network", "refId": "B", "step": 240 }], "timeFrom": null, "timeShift": null, "title": "Network", "tooltip": { "msResolution": false, "shared": false, "sort": 2, "value_type": "cumulative" }, "type": "graph", "xaxis": { "show": true }, "yaxes": [{ "format": "Bps", "label": "transmit / receive", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": false }] }, { "aliasColors": {}, "bars": false, "datasource": "${DS_PROMETHEUS}", "decimals": 2, "editable": true, "error": false, "fill": 1, "grid": { "threshold1": null, "threshold1Color": "rgba(216, 200, 27, 0.27)", "threshold2": null, "threshold2Color": "rgba(234, 112, 112, 0.22)" }, "id": 34, "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": 200, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "span": 12, "stack": false, "steppedLine": false, "targets": [{ "expr": "sum(container_fs_usage_bytes{image!=\"\",name=~\"^k8s_.*\",instance=~\"^$instance$\",namespace=~\"^$namespace$\"}) by (pod_name)", "interval": "", "intervalFactor": 2, "legendFormat": "{{ pod_name }}", "metric": "network", "refId": "A", "step": 240 }], "timeFrom": null, "timeShift": null, "title": "Filesystem", "tooltip": { "msResolution": false, "shared": false, "sort": 2, "value_type": "cumulative" }, "type": "graph", "xaxis": { "show": true }, "yaxes": [{ "format": "bytes", "label": "used", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": false }] }], "showTitle": true, "title": "each pod" }], "time": { "from": "now-3d", "to": "now" }, "timepicker": { "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] }, "templating": { "list": [{ "allValue": ".*", "current": {}, "datasource": "${DS_PROMETHEUS}", "hide": 0, "includeAll": true, "label": "Instance", "multi": false, "name": "instance", "options": [], "query": "label_values(instance)", "refresh": 1, "regex": "", "type": "query" }, { "current": {}, "datasource": "${DS_PROMETHEUS}", "hide": 0, "includeAll": true, "label": "Namespace", "multi": true, "name": "namespace", "options": [], "query": "label_values(namespace)", "refresh": 1, "regex": "", "type": "query" }] }, "annotations": { "list": [] }, "refresh": false, "schemaVersion": 12, "version": 8, "links": [], "gnetId": 737 } prometheus-datasource.json: | { "name": "prometheus", "type": "prometheus", "url": "http://prometheus:9090", "access": "proxy", "basicAuth": false } --- apiVersion: v1 kind: Service metadata: name: grafana namespace: monitoring labels: app: grafana component: core spec: type: NodePort ports: - port: 3000 selector: app: grafana component: core --- apiVersion: apps/v1 kind: Deployment metadata: name: grafana-core namespace: monitoring labels: app: grafana component: core spec: replicas: 1 selector: matchLabels: app: grafana component: core template: metadata: labels: app: grafana component: core spec: containers: - image: grafana/grafana:4.2.0 name: grafana-core imagePullPolicy: IfNotPresent resources: limits: cpu: 100m memory: 100Mi requests: cpu: 100m memory: 100Mi env: - name: GF_AUTH_BASIC_ENABLED value: "true" - name: GF_SECURITY_ADMIN_USER valueFrom: secretKeyRef: name: grafana key: admin-username - name: GF_SECURITY_ADMIN_PASSWORD valueFrom: secretKeyRef: name: grafana key: admin-password - name: GF_AUTH_ANONYMOUS_ENABLED value: "false" readinessProbe: httpGet: path: /login port: 3000 volumeMounts: - name: grafana-persistent-storage mountPath: /var/lib/grafana volumes: - name: grafana-persistent-storage emptyDir: {} --- apiVersion: batch/v1 kind: Job metadata: name: grafana-import-dashboards namespace: monitoring labels: app: grafana component: import-dashboards spec: template: metadata: name: grafana-import-dashboards labels: app: grafana component: import-dashboards spec: serviceAccountName: prometheus-k8s initContainers: - name: wait-for-grafana image: giantswarm/tiny-tools args: - /bin/sh - -c - > set -x; while [ $(curl -sw '%{http_code}' "http://grafana:3000" -o /dev/null) -ne 200]; do echo '.' sleep 15; done containers: - name: grafana-import-dashboards image: giantswarm/tiny-tools command: ["/bin/sh", "-c"] workingDir: /opt/grafana-import-dashboards args: - > for file in *-datasource.json ; do if [ -e "$file" ] ; then echo "importing $file" && curl --silent --fail --show-error \ --request POST http://${GF_ADMIN_USER}:${GF_ADMIN_PASSWORD}@grafana:3000/api/datasources \ --header "Content-Type: application/json" \ --data-binary "@$file" ; echo "" ; fi done ; for file in *-dashboard.json ; do if [ -e "$file" ] ; then echo "importing $file" && ( echo '{"dashboard":'; \ cat "$file"; \ echo ',"overwrite":true,"inputs":[{"name":"DS_PROMETHEUS","type":"datasource","pluginId":"prometheus","value":"prometheus"}]}' ) \ | jq -c '.' \ | curl --silent --fail --show-error \ --request POST http://${GF_ADMIN_USER}:${GF_ADMIN_PASSWORD}@grafana:3000/api/dashboards/import \ --header "Content-Type: application/json" \ --data-binary "@-" ; echo "" ; fi done env: - name: GF_ADMIN_USER valueFrom: secretKeyRef: name: grafana key: admin-username - name: GF_ADMIN_PASSWORD valueFrom: secretKeyRef: name: grafana key: admin-password volumeMounts: - name: config-volume mountPath: /opt/grafana-import-dashboards restartPolicy: Never volumes: - name: config-volume configMap: name: grafana-import-dashboards
kubectl apply -f grafana-all-in-one.yml

可以看到

configmap/grafana-import-dashboards created
service/grafana created
deployment.apps/grafana-core created
job.batch/grafana-import-dashboards created

查看一下服务

kubectl get svc -n monitoring

可以看到

NAME         TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
grafana      NodePort   10.1.8.205            3000:31582/TCP   102s
prometheus   NodePort   10.1.100.20           9090:30091/TCP   9m58s

访问 grafana:http://192.168.190.132:31582/ ,用户名和密码默认都是 admin

你可能感兴趣的:(12 kubernetes 监控(三))