ELK 日志系统搭建--监控nginx

1. logstash安装
   

下载路径: https://www.elastic.co/downloads/logstash（安装方法参考官网安装步骤）
要读取nginx日志，配置nginx日志格式
vim nginx.conf
修改nginx记录日志格式，从http模块下

log_format  main  '$remote_addr | $time_local | $request | $uri | '
                      '$status | $body_bytes_sent | $bytes_sent | $gzip_ratio | $http_referer | '
                      '"$http_user_agent" | $http_x_forwarded_for | $upstream_addr | $upstream_response_time | $upstream_status | $request_time';

修改完成后保存，使用./nginx -s reload 重新加载

/etc/logstash/conf.d下创建nginx日志配置文件
touch nginx_access.conf
sudo vim nginx_access.conf
input {
    file {
        path => [ "/usr/local/nginx/logs/adsapi.access.log" ]
         type => "nginx_access"
    }
}
filter {
   grok {
    match => [
                    "message", "%{IPORHOST:clientip} \| %{HTTPDATE:timestamp} \| (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|-) \| %{URIPATH:uripath} \| %{NUMBER:response} \| (?:%{NUMBER:body_bytes_sent}|-) \| (?:%{NUMBER:bytes_sent}|-) \| (?:%{NOTSPACE:gzip_ratio}|-) \| (?:%{QS:http_referer}|-) \| %{QS:user_agent} \| (?:%{QS:http_x_forwarded_for}|-) \| (%{URIHOST:upstream_addr}|-) \| (%{BASE16FLOAT:upstream_response_time}) \| %{NUMBER:upstream_status} \| (%{BASE16FLOAT:request_time})"
                ]
        }
 geoip {
      source => "clientip"
      target => "geoip"
      add_field => [ "[geoip][coordinates]","%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]","%{[geoip][latitude]}" ]
    }
    mutate {
     convert => [ "[geoip][coordinates]","float" ]
    }

    date {
      match => [ "timestamp","dd/MMM/yyyy:HH:mm:ss Z"]

    }
    mutate {
      remove_field => "timestamp"

    }
}


output {
    elasticsearch {
        hosts => ["127.0.*.*:9200"]
        index => "logstash-nginx-access-%{+YYYY.MM.dd}"
        user => "****" //下文安装kibana会设置
        password => "pwd"
    }
    stdout {

       }
}

2 . elasticsearch安装

下载地址：https://www.elastic.co/downloads/elasticsearch，安装步骤参见官网
安装完成后，从etc/elasticsearch/ 目录下
vim elasticsearch.yml

cluster.name: elk
node.name: es2
path.data: /data/elasticsearch（存储目录一定要给elasticsearch账户授权）
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: *.*.*.*(服务器ip)
http.port: 9200

启动服务sudo service elasticsearch start
查看启动日志，或直接查看启动后的进程状态是否成功
elasticsearch (pid 19206) is running.
浏览器输入：http://ip地址:9200/,给出响应结果

3 . kibana安装

下载地址：https://www.elastic.co/downloads/kibana
安装x-pack ,下载地址：https://www.elastic.co/downloads/x-pack
自己安装的在/user/share/ 目录下
从etc/kibana/ 目录下修改kibana.yml文件

sudo vim kibana.yml
server.name: "*.*.*.*"// (服务器ip地址)
elasticsearch.url: "http://*.*.*.*:9200"
elasticsearch.username: "username"
elasticsearch.password: "pwd"
增加：
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

配置完成后，三个服务依次启动 elasticsearch–kibana –logstash
service elasticsearch start
service kibana start
initctl start logstash
4. 要外网访问需要配置nginx.conf,访问地址到kibana

upstream elk {
    ip_hash;
    server 127.0.0.1:5601;
}

server {
    listen 80;
    server_name 域名;
    server_tokens off;

    client_body_timeout 5s;
    client_header_timeout 5s;

    location / {
        proxy_pass http://elk/;
        index index.html index.htm;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

5.  配置完成后，重新加载nginx,浏览器输入域名，填写安装x-pack的用户名和密码
6.  登录成功后，![这里写图片描述](https://img-blog.csdn.net/20171201115255853?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
7.  在Configure an index pattern功能下配置：logstash-nginx-access*
![这里写图片描述](https://img-blog.csdn.net/20171201115318889?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvcWluZ3RpYW4yMDAy/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)
8.创建成功后，选择discover模块就能查看到