【Grafana】通过阿里云日志服务监控Nginx访问日志显示统计信息

Grafana通过阿里云日志服务监控Nginx访问日志显示统计信息

前言

通过之前的文章,我们做到了Grafana的安装部署和初期的设置,没有了解过的同学可以参考下面的文章链接

https://blog.csdn.net/diandianxiyu_geek/article/details/97013330

https://blog.csdn.net/diandianxiyu_geek/article/details/96577082

接下来要做的是数据可视化的一个应用,nginx访问文件的监控,输出图表。

实操

修改nginx配置文件

如果你没有进行额外的配置,那么你的nginx日志是这样的

101.254.141.164 - - [19/Jul/2019:11:22:03 +0800] "GET /online/cart/info HTTP/1.1" 200 120 "https://servicewechat.com/wx855bd11a98860d4e/devtools/page-frame.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 Safari/601.1 wechatdevtools/1.02.1905151 MicroMessenger/6.7.3 Language/zh_CN webview/"
101.254.141.164 - - [19/Jul/2019:11:22:03 +0800] "GET /online/item/new?page=1&count=99 HTTP/1.1" 200 469 "https://servicewechat.com/wx855bd11a98860d4e/devtools/page-frame.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 Safari/601.1 wechatdevtools/1.02.1905151 MicroMessenger/6.7.3 Language/zh_CN webview/"
101.254.141.164 - - [19/Jul/2019:11:22:03 +0800] "GET /online/item/list?page=1&count=10 HTTP/1.1" 200 618 "https://servicewechat.com/wx855bd11a98860d4e/devtools/page-frame.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 Safari/601.1 wechatdevtools/1.02.1905151 MicroMessenger/6.7.3 Language/zh_CN webview/“

接下来修改配置文件,让日志记录更多信息

找到你的nginx

[root@VM_0_15_centos conf]# whereis  nginx
nginx: /usr/local/nginx /usr/local/nginx/sbin/nginx

找到配置文件

[root@VM_0_15_centos conf]# cd /usr/local/nginx/conf
[root@VM_0_15_centos conf]# ls
fastcgi.conf          fastcgi_params          koi-utf  mime.types          nginx.conf     nginx.conf.default  scgi_params          uwsgi_params          vhost
fastcgi.conf.default  fastcgi_params.default  koi-win  mime.types.default  nginx.conf_bk  proxy.conf          scgi_params.default  uwsgi_params.default  win-utf
[root@VM_0_15_centos conf]# cat nginx.conf
user www www;
worker_processes auto;
error_log /data/wwwlogs/error_nginx.log crit;
pid /var/run/nginx.pid;
worker_rlimit_nofile 51200;
events {
  use epoll;
  worker_connections 51200;
  multi_accept on;
}
http {
  include mime.types;
  default_type application/octet-stream;
  server_names_hash_bucket_size 128;
  client_header_buffer_size 32k;
  large_client_header_buffers 4 32k;
  client_max_body_size 1024m;
  client_body_buffer_size 10m;
  sendfile on;
  tcp_nopush on;
  keepalive_timeout 120;
  server_tokens off;
  tcp_nodelay on;
  fastcgi_connect_timeout 300;
  fastcgi_send_timeout 300;
  fastcgi_read_timeout 300;
  fastcgi_buffer_size 64k;
  fastcgi_buffers 4 64k;
  fastcgi_busy_buffers_size 128k;
  fastcgi_temp_file_write_size 128k;
  fastcgi_intercept_errors on;
 #注意这部分是新增的,日志配置
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $http_host '
                        '$status $request_length $body_bytes_sent "$http_referer" '
                        '"$http_user_agent"  $request_time $upstream_response_time';
  #Gzip Compression
  gzip on;
  gzip_buffers 16 8k;
  gzip_comp_level 6;
  gzip_http_version 1.1;
  gzip_min_length 256;
  gzip_proxied any;
  gzip_vary on;
  gzip_types
    text/xml application/xml application/atom+xml application/rss+xml application/xhtml+xml image/svg+xml
    text/javascript application/javascript application/x-javascript
    text/x-json application/json application/x-web-app-manifest+json
    text/css text/plain text/x-component
    font/opentype application/x-font-ttf application/vnd.ms-fontobject
    image/x-icon;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  ##If you have a lot of static files to serve through Nginx then caching of the files' metadata (not the actual files' contents) can save some latency.
  #open_file_cache max=1000 inactive=20s;
  #open_file_cache_valid 30s;
  #open_file_cache_min_uses 2;
  #open_file_cache_errors on;
######################## default ############################
  server {
    listen 80;
    server_name _;
    access_log /data/wwwlogs/access_nginx.log combined;
    root /data/wwwroot/default;
    index index.html index.htm index.php;
    #error_page 404 /404.html;
    #error_page 502 /502.html;
    location /nginx_status {
      stub_status on;
      access_log off;
      allow 127.0.0.1;
      deny all;
    }
    location ~ [^/]\.php(/|$) {
      #fastcgi_pass remote_php_ip:9000;
      fastcgi_pass unix:/dev/shm/php-cgi.sock;
      fastcgi_index index.php;
      include fastcgi.conf;
    }
    location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|flv|mp4|ico)$ {
      expires 30d;
      access_log off;
    }
    location ~ .*\.(js|css)?$ {
      expires 7d;
      access_log off;
    }
    location ~ ^/(\.user.ini|\.ht|\.git|\.svn|\.project|LICENSE|README.md) {
      deny all;
    }
  }
########################## vhost #############################
  include vhost/*.conf;
}

在上面的配置文件中我已经把日志配置加了进去,为了方便辨认再贴一遍

在http中添加对应日志配置->

 log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $http_host '
                        '$status $request_length $body_bytes_sent "$http_referer" '
                        '"$http_user_agent"  $request_time $upstream_response_time’;

上面的配置的意思是增加一个main的日志配置

接下来将nginx.conf的server中的access_log /data/wwwlogs/access_nginx.log combined; combined 替换为main,combined为nginx内置的不需要配置的默认文件格式。

vhost里的所有文件也要进行替换。

重启nginx

再查看日志,就变成了

101.254.141.164 - - [19/Jul/2019:11:38:36 +0800] "GET /online/item/list?page=1&count=10 HTTP/1.1" api4621d373.henibox.com 200 587 619 "https://servicewechat.com/wx855bd11a98860d4e/0/page-frame.html" "Mozilla/5.0 (Linux; Android 8.0.0; MIX Build/OPR1.170623.032; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Mobile Safari/537.36 MicroMessenger/7.0.5.1440(0x27000537) Process/appbrand2 NetType/WIFI Language/zh_CN"  0.351 0.351
101.254.141.164 - - [19/Jul/2019:11:38:36 +0800] "GET /online/pick/list?page=1&count=4 HTTP/1.1" api4621d373.henibox.com 200 586 634 "https://servicewechat.com/wx855bd11a98860d4e/0/page-frame.html" "Mozilla/5.0 (Linux; Android 8.0.0; MIX Build/OPR1.170623.032; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Mobile Safari/537.36 MicroMessenger/7.0.5.1440(0x27000537) Process/appbrand2 NetType/WIFI Language/zh_CN"  0.389 0.389

为了防止日志系统识别格式问题,可以将之前的日志文件转移到其他文件夹作为备份

nginx日志接入阿里云日志系统

在阿里云日志系统中添加一个项目,在项目中添加一个Logstore,然后接入配置向导

选择nginx访问日志
在这里插入图片描述
填写的日志格式就是我们之前修改的日志格式

在这里插入图片描述

选择需要应用的机器,这里作为测试选择测试机器

下一步就会自动识别文本,匹配对应的字段,并增加索引进行统计

在这里插入图片描述

到这一步,在阿里云日志服务的部分就完成了。

阿里云日志系统接入Grafana

首先接入数据源,安装插件

cd /var/lib/grafana/plugins
git clone https://github.com/aliyun/aliyun-log-grafana-datasource-plugin

在这里插入图片描述

查看新增一个数据源

在这里插入图片描述

下一步开始配置

在这里插入图片描述

然后配置变量,配置变量用于在仪表盘页面公共选择时间段

我们用简单粗暴的引入方式,先直接引入阿里云官方给出的demo

{
  "annotations": {
    "list": []
  },
  "description": "日志服务demo",
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "hideControls": false,
  "id": 2,
  "links": [],
  "rows": [
    {
      "collapse": false,
      "height": "250px",
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "logservice",
          "fill": 1,
          "id": 3,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "rightSide": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "pv",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "span": 3,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname| select approx_distinct(remote_addr) as uv ,count(1) as pv , __time__ - __time__%$$myinterval  as time group by __time__ - __time__%$$myinterval  order by time limit 1000",
              "refId": "A",
              "target": "uv,pv",
              "type": "timeserie",
              "xcol": "time",
              "ycol": "uv,pv"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "UV & PV",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "decimals": null,
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "decimals": null,
              "format": "short",
              "label": "",
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "logservice",
          "fill": 1,
          "id": 6,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 3,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select sum(body_byte_sent) as net_out, sum(request_length) as net_in  ,__time__ - __time__ % $$myinterval  as time group by __time__ - __time__ % $$myinterval limit 10000",
              "refId": "A",
              "target": "net_in,net_out",
              "type": "timeserie",
              "xcol": "time",
              "ycol": "net_in,net_out"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "入网、出网带宽",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "columns": [],
          "datasource": "logservice",
          "fontSize": "100%",
          "id": 15,
          "links": [],
          "pageSize": null,
          "scroll": true,
          "showHeader": true,
          "sort": {
            "col": 2,
            "desc": true
          },
          "span": 3,
          "styles": [
            {
              "alias": "Time",
              "dateFormat": "YYYY-MM-DD HH:mm:ss",
              "pattern": "Time",
              "type": "hidden"
            },
            {
              "alias": "",
              "colorMode": null,
              "colors": [
                "rgba(245, 54, 54, 0.9)",
                "rgba(237, 129, 40, 0.89)",
                "rgba(50, 172, 45, 0.97)"
              ],
              "decimals": 2,
              "pattern": "/.*/",
              "thresholds": [],
              "type": "number",
              "unit": "short"
            }
          ],
          "targets": [
            {
              "bucketAggs": [],
              "query": "* | select diff[1] as today, diff[2] as yestoday, concat( cast( round((diff[3] - 1)*100,2) as varchar),'%')  as growth from(select compare(pv, 86400) as diff  from(select count(1) as pv from log))",
              "refId": "A",
              "target": "today,yestoday,growth",
              "type": "timeserie",
              "xcol": "table",
              "ycol": "today,yestoday,growth"
            }
          ],
          "title": "环比比较",
          "transform": "timeseries_to_columns",
          "type": "table"
        },
        {
          "cacheTimeout": null,
          "colorBackground": false,
          "colorValue": false,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "datasource": "logservice",
          "format": "none",
          "gauge": {
            "maxValue": 100,
            "minValue": 0,
            "show": false,
            "thresholdLabels": false,
            "thresholdMarkers": true
          },
          "id": 17,
          "interval": null,
          "links": [],
          "mappingType": 1,
          "mappingTypes": [
            {
              "name": "value to text",
              "value": 1
            },
            {
              "name": "range to text",
              "value": 2
            }
          ],
          "maxDataPoints": 100,
          "nullPointMode": "connected",
          "nullText": null,
          "postfix": "",
          "postfixFontSize": "50%",
          "prefix": "",
          "prefixFontSize": "50%",
          "rangeMaps": [
            {
              "from": "null",
              "text": "N/A",
              "to": "null"
            }
          ],
          "span": 3,
          "sparkline": {
            "fillColor": "rgba(31, 118, 189, 0.18)",
            "full": false,
            "lineColor": "rgb(31, 120, 193)",
            "show": false
          },
          "tableColumn": "",
          "targets": [
            {
              "bucketAggs": [],
              "query": "* | select count(1) as total_pv",
              "refId": "A",
              "target": "total_pv",
              "type": "timeserie",
              "ycol": "total_pv"
            }
          ],
          "thresholds": "",
          "title": "总pv",
          "type": "singlestat",
          "valueFontSize": "80%",
          "valueMaps": [
            {
              "op": "=",
              "text": "N/A",
              "value": "null"
            }
          ],
          "valueName": "avg"
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 207,
      "panels": [
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 2,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 2,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select count(1) as pv ,method group by method",
              "refId": "A",
              "target": "method,pv",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "method,pv"
            }
          ],
          "title": "HTTP方法汇总",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        },
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 5,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 2,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select count(1) as pv ,status group by status",
              "refId": "A",
              "target": "status,pv",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "status,pv"
            }
          ],
          "title": "HTTP状态码",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        },
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 8,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 4,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select count(1) as pv , http_referer  group by http_referer order by pv desc",
              "refId": "A",
              "target": "referer,pv",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "http_referer,pv"
            }
          ],
          "title": "Top来源域名",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        },
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 11,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 4,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select count(1) as pv, case when http_user_agent like '%Android%' then 'Android' when http_user_agent like '%iPhone%' then 'iOS' else 'unKnown' end as http_user_agent  group by case when http_user_agent like '%Android%' then 'Android' when http_user_agent like '%iPhone%' then 'iOS' else 'unKnown' end   order by pv desc limit 10",
              "refId": "A",
              "target": "http_user_agent,pv",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "http_user_agent,pv"
            }
          ],
          "title": "客户端分布",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 362,
      "panels": [
        {
          "columns": [],
          "datasource": "logservice",
          "fontSize": "100%",
          "id": 9,
          "links": [],
          "pageSize": null,
          "scroll": true,
          "showHeader": true,
          "sort": {
            "col": 0,
            "desc": true
          },
          "span": 3,
          "styles": [
            {
              "alias": "Time",
              "dateFormat": "YYYY-MM-DD HH:mm:ss",
              "pattern": "Time",
              "type": "date"
            },
            {
              "alias": "",
              "colorMode": null,
              "colors": [
                "rgba(245, 54, 54, 0.9)",
                "rgba(237, 129, 40, 0.89)",
                "rgba(50, 172, 45, 0.97)"
              ],
              "decimals": 2,
              "pattern": "/.*/",
              "thresholds": [],
              "type": "number",
              "unit": "short"
            }
          ],
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select uri  as top_latency_url ,request_time order by request_time  desc limit 10",
              "refId": "A",
              "target": "top_latency_url,request_time",
              "type": "timeserie",
              "ycol": "top_latency_url,request_time"
            }
          ],
          "title": "延时最高的页面",
          "transform": "timeseries_to_columns",
          "type": "table"
        },
        {
          "columns": [],
          "datasource": "logservice",
          "fontSize": "100%",
          "id": 4,
          "links": [],
          "pageSize": null,
          "scroll": true,
          "showHeader": true,
          "sort": {
            "col": 0,
            "desc": false
          },
          "span": 4,
          "styles": [
            {
              "alias": "Time",
              "dateFormat": "YYYY-MM-DD HH:mm:ss",
              "pattern": "Time",
              "type": "date"
            },
            {
              "alias": "",
              "colorMode": null,
              "colors": [
                "rgba(245, 54, 54, 0.9)",
                "rgba(237, 129, 40, 0.89)",
                "rgba(50, 172, 45, 0.97)"
              ],
              "decimals": 2,
              "pattern": "/.*/",
              "thresholds": [],
              "type": "number",
              "unit": "short"
            }
          ],
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select count(1) as pv, split_part(uri,'?',1) as path  group by split_part(uri,'?',1) order by pv desc limit 20",
              "refId": "A",
              "target": "path,pv",
              "type": "timeserie",
              "ycol": "path,pv"
            }
          ],
          "title": "热门页面",
          "transform": "timeseries_to_columns",
          "type": "table"
        },
        {
          "columns": [],
          "datasource": "logservice",
          "fontSize": "100%",
          "id": 10,
          "links": [],
          "pageSize": null,
          "scroll": true,
          "showHeader": true,
          "sort": {
            "col": 0,
            "desc": false
          },
          "span": 5,
          "styles": [
            {
              "alias": "Time",
              "dateFormat": "YYYY-MM-DD HH:mm:ss",
              "pattern": "Time",
              "type": "date"
            },
            {
              "alias": "",
              "colorMode": null,
              "colors": [
                "rgba(245, 54, 54, 0.9)",
                "rgba(237, 129, 40, 0.89)",
                "rgba(50, 172, 45, 0.97)"
              ],
              "decimals": 2,
              "pattern": "/.*/",
              "thresholds": [],
              "type": "number",
              "unit": "short"
            }
          ],
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname not status:200| select count(1) as pv , uri  group by uri order by pv desc",
              "refId": "A",
              "target": "url,pv",
              "type": "timeserie",
              "ycol": "url,pv"
            }
          ],
          "title": "非200请求TOP页面",
          "transform": "timeseries_to_columns",
          "type": "table"
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 282,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "logservice",
          "fill": 1,
          "id": 7,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 3,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "bucketAggs": [],
              "query": "$hostname | select avg(request_time) as response_time, avg(upstream_response_time) as upstream_response_time  ,__time__ - __time__ % $$myinterval  as time group by __time__ - __time__ % $$myinterval limit 10000",
              "refId": "A",
              "target": "upstream_response_time,response_time",
              "type": "timeserie",
              "xcol": "time",
              "ycol": "upstream_response_time,response_time"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "前端平均延时和后端平均延时",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        },
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 12,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 5,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "*| select ip_to_province(remote_addr) as province, count(1) as pv group by province order by pv desc  limit 10",
              "refId": "A",
              "target": "province,pv",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "province,pv"
            }
          ],
          "title": "不同省份PV",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        },
        {
          "aliasColors": {},
          "cacheTimeout": null,
          "combine": {
            "label": "Others",
            "threshold": 0
          },
          "datasource": "logservice",
          "fontSize": "80%",
          "format": "short",
          "id": 13,
          "interval": null,
          "legend": {
            "show": true,
            "values": true
          },
          "legendType": "Under graph",
          "links": [],
          "maxDataPoints": 3,
          "nullPointMode": "connected",
          "pieType": "pie",
          "span": 4,
          "strokeWidth": 1,
          "targets": [
            {
              "bucketAggs": [],
              "query": "*| select ip_to_provider(remote_addr) as vendor, count(1) as pv group by vendor order by pv desc  limit 10",
              "refId": "A",
              "type": "timeserie",
              "xcol": "pie",
              "ycol": "vendor,pv"
            }
          ],
          "title": "网络供应商",
          "type": "grafana-piechart-panel",
          "valueName": "current"
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 250,
      "panels": [
        {
          "circleMaxSize": 30,
          "circleMinSize": 2,
          "colors": [
            "rgba(245, 54, 54, 0.9)",
            "rgba(237, 129, 40, 0.89)",
            "rgba(50, 172, 45, 0.97)"
          ],
          "datasource": "logservice",
          "decimals": 0,
          "esMetric": "Count",
          "hideEmpty": false,
          "hideZero": false,
          "id": 14,
          "initialZoom": 1,
          "locationData": "countries",
          "mapCenter": "(0°, 0°)",
          "mapCenterLatitude": 0,
          "mapCenterLongitude": 0,
          "maxDataPoints": 1,
          "showLegend": true,
          "span": 6,
          "stickyLabels": false,
          "targets": [
            {
              "bucketAggs": [],
              "query": "* | select count(1) as pv , ip_to_country_code(remote_addr) as country group by country",
              "refId": "A",
              "type": "timeserie",
              "xcol": "bar",
              "ycol": "country,pv"
            }
          ],
          "thresholds": "0,10",
          "title": "Panel Title",
          "type": "grafana-worldmap-panel",
          "unitPlural": "",
          "unitSingle": "",
          "valueName": "total"
        },
        {
          "cards": {
            "cardPadding": null,
            "cardRound": null
          },
          "color": {
            "cardColor": "#b4ff00",
            "colorScale": "sqrt",
            "colorScheme": "interpolateOranges",
            "exponent": 0.5,
            "mode": "spectrum"
          },
          "dataFormat": "timeseries",
          "datasource": "logservice",
          "heatmap": {},
          "highlightCards": true,
          "id": 18,
          "legend": {
            "show": false
          },
          "links": [],
          "span": 6,
          "targets": [
            {
              "bucketAggs": [],
              "query": "* | select __time__ - __time__ % $$myinterval as t, method,count(1) as pv  group by  t, method  order by t limit 10000",
              "refId": "A",
              "target": "method#:#pv",
              "type": "timeserie",
              "xcol": "t",
              "ycol": "method#:#pv"
            }
          ],
          "title": "热力图演示,method热力图",
          "tooltip": {
            "show": true,
            "showHistogram": false
          },
          "type": "heatmap",
          "xAxis": {
            "show": true
          },
          "xBucketNumber": null,
          "xBucketSize": null,
          "yAxis": {
            "decimals": null,
            "format": "short",
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true,
            "splitFactor": null
          },
          "yBucketNumber": null,
          "yBucketSize": null
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    },
    {
      "collapse": false,
      "height": 250,
      "panels": [
        {
          "content": "# 查看配置\n\n在本页面顶端,点击“share dashboard” 按钮  ->  Export    。 导出为json文件,该配置可以导入您的dashboard。",
          "id": 16,
          "links": [],
          "mode": "markdown",
          "span": 6,
          "title": "",
          "type": "text"
        },
        {
          "aliasColors": {},
          "bars": true,
          "dashLength": 10,
          "dashes": false,
          "datasource": "logservice",
          "fill": 1,
          "id": 19,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": false,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "span": 6,
          "stack": true,
          "steppedLine": false,
          "targets": [
            {
              "bucketAggs": [],
              "query": "* | select __time__ - __time__ % $$myinterval  as t, count(1) as pv , method group by t , method limit 10000",
              "refId": "A",
              "target": "method#:#pv",
              "type": "timeserie",
              "xcol": "t",
              "ycol": "method#:#pv"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "各个Method堆叠图",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ]
        }
      ],
      "repeat": null,
      "repeatIteration": null,
      "repeatRowId": null,
      "showTitle": false,
      "title": "Dashboard Row",
      "titleSize": "h6"
    }
  ],
  "schemaVersion": 14,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "auto": true,
        "auto_count": 30,
        "auto_min": "10s",
        "current": {
          "text": "30m",
          "value": "30m"
        },
        "hide": 0,
        "label": "time interval",
        "name": "myinterval",
        "options": [
          {
            "selected": false,
            "text": "auto",
            "value": "$__auto_interval"
          },
          {
            "selected": false,
            "text": "1m",
            "value": "1m"
          },
          {
            "selected": false,
            "text": "10m",
            "value": "10m"
          },
          {
            "selected": true,
            "text": "30m",
            "value": "30m"
          },
          {
            "selected": false,
            "text": "1h",
            "value": "1h"
          },
          {
            "selected": false,
            "text": "6h",
            "value": "6h"
          },
          {
            "selected": false,
            "text": "12h",
            "value": "12h"
          },
          {
            "selected": false,
            "text": "1d",
            "value": "1d"
          },
          {
            "selected": false,
            "text": "7d",
            "value": "7d"
          },
          {
            "selected": false,
            "text": "14d",
            "value": "14d"
          },
          {
            "selected": false,
            "text": "30d",
            "value": "30d"
          }
        ],
        "query": "1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",
        "refresh": 2,
        "type": "interval"
      },
      {
        "allValue": null,
        "current": {
          "text": "*",
          "value": "*"
        },
        "hide": 0,
        "includeAll": false,
        "label": "域名",
        "multi": false,
        "name": "hostname",
        "options": [
          {
            "selected": true,
            "text": "*",
            "value": "*"
          },
          {
            "selected": false,
            "text": "www.host.com",
            "value": "www.host.com"
          },
          {
            "selected": false,
            "text": "www.host0.com",
            "value": "www.host0.com"
          },
          {
            "selected": false,
            "text": "www.host1.com",
            "value": "www.host1.com"
          }
        ],
        "query": "*,www.host.com,www.host0.com,www.host1.com",
        "type": "custom"
      }
    ]
  },
  "time": {
    "from": "now-24h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "Nginx访问统计",
  "version": 79
}

注意上面的json文件,千万不要直接复制,这里需要修改的地方有两个,
一个是数据源,一定要改成自己前面配置的数据源的名称,不能用这个默认的
还有就是变量的域名部分,在这里直接改成自己的域名,就不用再去配置里的变量去改了

注意,由于这个官方的版本比较老,配置之后还存在字段兼容性的问题,需要按下面的进行修改

  • method —> request_method
  • url -> request_uri
  • body_byte_sent

我们可以在仪表盘的配置界面直接修改json文件

在这里插入图片描述

经过修改json并保存之后,我们就得到了nginx访问日志的监控数据

总结

我认为理解Grafana的点在于他是一个时序概念的数据可视化工具。
以后我会分享如何用它做业务相关的数据展示。

你可能感兴趣的:(服务器,云服务器,云主机,云服务,云服务器,阿里云,腾讯云,华为云)