docker+elk7.8实战之filebeat收集nginx日志
0. 前置条件
需要自行安装nginx环境,通过yum -y install nginx安装即可。
1.安装filebeat
-
如果没有安装kibana可以参考之前的文章https://blog.csdn.net/u010361276/article/details/107695787
-
如果安装有kibana可以在如下地址获取到安装filebeat及其他beat的方式和下载地址。（ip换成实际使用的ip）
http://192.168.0.203:5601/app/kibana#/home/tutorial/nginxLogs
- 具体安装方式，以centos7为例
# 通过rpm包安装
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.8.1-x86_64.rpm
sudo rpm -vi filebeat-7.8.1-x86_64.rpm
注意：
安装前可通过 rpm -qa|grep filebeat查看是否已经安装过filebeat(如果安装过会打印一系列已安装的文件名供查看和卸载使用)
如果需要卸载使用rpm -e xxx命令(xxx代表上一步输出的文件名)
2.配置filebeat
通过rpm文件安装的filebeat对应的配置文件目录在/etc/filebeat/,对应的日志目录在/var/log/filebeat/
主要的配置文件是filebeat.yml 和 modules.d文件夹中的一系列yml文件
在配置文件目录中存在一个filebeat.reference.yml文件，其中列举了几乎filebeat相关的所有配置及说明。
1) 修改filebeat.yml文件
# 这里是配置加载modules.d里面的哪些文件 以及是否定时加载配置文件
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 30s
# 这里配置这个filebeat的相关信息
name: filebeat-202
tags: ["filebeat", "203"]
# filebeat提供了多种输出可供选择,目前我们这里选择的是logstash 后续会做相关改造
# 这里是具体的logstash的IP地址和端口,请对应修改
output.logstash:
hosts: ["192.168.0.203:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# 下面的是配置日志相关的信息 包括日志级别及日志输出位置
logging.level: error
logging.to_stderr: false
logging.to_syslog: false
logging.to_eventlog: false
logging.to_files: true
logging.path: /var/log/filebeat
logging.name: filebeat
2) 修改模块配置
在/etc/filebeat/modules.d文件夹中默认存在了很多以.yml.disabled结尾的文件，我们可以参考这些配置yml文件
也可以直接重命名为yml文件
cd /etc/filebeat/modules.d
cp nginx.yml.disabled nginx.yml
说明: 开启模块的方式有多种，不局限于提到的这种。
3) logstash配置修改
在步骤2中我们设置了输出是logstash的5044端口，自然logstash的配置也要做相关修改。
在之前搭建的elk环境的logstash目录/opt/elk7/logstash/pipeline中我们新建一个conf文件，命名为logstash-beat-es.conf
# 输入配置 注意是beats不是beat 这里的beats是插件名
input {
beats {
port => 5044
}
}
# 过滤器 将日志解析成对应的具体字段值 使用了grok插件和内置的COMBINEDAPACHELOG格式匹配器
filter {
# 这里是ruby语法
if "nginx" == [service][type] {
grok {
# 注意这里的=> 和 ""
match => {"message" => "%{IP:req_ip} - (%{USERNAME:req_user}|-) \[%{HTTPDATE:req_timestamp}\] \"%{WORD:request_verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion
}\" %{NUMBER:resp_status:int} %{NUMBER:body_sent:int} \"-\" \"%{GREEDYDATA:httpagent}\" \"-\""}
}
}
}
# 这里是输出设置
output {
# 如果需要调试或者验证日志数据是否到了logstash
# 如下几行代码便是将收到的日志信息打印到控制台
stdout {
codec => rubydebug
}
# 设置输出到es的配置 和 之前的一样只是重新定义了索引名称
elasticsearch {
index => "beat-%{+YYYY.MM.dd}"
hosts => "http://192.168.0.203:9200"
# 注意这里是具体的es配置的用户名和密码 注意这里的""
user => "elastic"
password => "xxxxx"
}
}
添加了配置文件我们需要在pipeline.yml文件中创建对应的信息
添加如下代码即可
- pipeline.id: beat
path.config: "/usr/share/logstash/pipeline/logstash-beat.conf"
3.启动并验证
- logstash的启动 由于之前创建logstash容器的时候挂载的是pipeline文件夹 ,所以直接重启容器即可。
docker restart logstash
- 启动filebeat
# 设置filebeat随机启动
systemctl enable filebeat
# filebeat的启动
systemctl start filebeat
# 查看filebeat状态
systemctl status filebeat -l
如果看到如下输出表示正常
[root@localhost pipeline]# systemctl status filebeat -l
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
Active: active (running) since 日 2020-08-16 17:15:55 CST; 39min ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 4206 (filebeat)
Tasks: 21
Memory: 80.8M
CGroup: /system.slice/filebeat.service
└─4206 /usr/share/filebeat/bin/filebeat -environment systemd -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
8月 16 17:15:55 localhost.localdomain systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
8月 16 17:15:55 localhost.localdomain systemd[1]: Starting Filebeat sends log files to Logstash or directly to Elasticsearch....
注意：
Active的值必须是active(running)，如果是failed或者exits的话需要通过日志排查原因。
- 如果开启了stdout标准控制台输出 则可以通过logstash容器的日志查看具体内容
docker logs -f logstash --tail 200
访问nginx 如 curl http://192.168.0.203
观察logstash的日志输出，可以看到如下内容
{
"service" => {
"type" => "nginx"
},
"ident" => "-",
"timestamp" => "16/Aug/2020:18:10:41 +0800",
"tags" => [
[0] "filebeat",
[1] "203",
[2] "beats_input_codec_plain_applied"
],
"message" => "192.168.0.203 - - [16/Aug/2020:18:10:41 +0800] \"GET / HTTP/1.1\" 200 3700 \"-\" \"curl/7.29.0\" \"-\"",
"log" => {
"file" => {
"path" => "/var/log/nginx/access.log"
},
"offset" => 0
},
"referrer" => "\"-\"",
"@timestamp" => 2020-08-16T10:10:44.765Z,
"ecs" => {
"version" => "1.5.0"
},
"agent" => {
"name" => "filebeat-202",
"version" => "7.8.1",
"ephemeral_id" => "c33421ab-e2d9-4a6b-8bd5-94d3560c7c9b",
"hostname" => "localhost.localdomain",
"type" => "filebeat",
"id" => "219a0e91-c6f5-47d9-8539-00e89e163d9c"
},
"host" => {
"os" => {
"name" => "CentOS Linux",
"version" => "7 (Core)",
"kernel" => "3.10.0-862.9.1.el7.x86_64",
"family" => "redhat",
"platform" => "centos",
"codename" => "Core"
},
"architecture" => "x86_64",
"name" => "filebeat-202",
"ip" => [
[0] "192.168.0.203",
[1] "fe80::1c6b:d091:a6b1:d3e2",
[2] "172.17.0.1",
[3] "fe80::42:b4ff:fe99:5a47",
[4] "172.18.0.1",
[5] "fe80::fc89:7aff:fe78:5712",
[6] "fe80::20e6:5dff:fedb:7db1",
[7] "fe80::98d8:3fff:febb:1508",
[8] "fe80::6894:1cff:fed4:8f90"
],
"hostname" => "localhost.localdomain",
"mac" => [
[0] "00:0c:29:81:4c:19",
[1] "02:42:b4:99:5a:47",
[2] "02:42:8a:0b:e2:63",
[3] "fe:89:7a:78:57:12",
[4] "22:e6:5d:db:7d:b1",
[5] "9a:d8:3f:bb:15:08",
[6] "6a:94:1c:d4:8f:90"
],
"containerized" => false,
"id" => "35f66aae9fd84ef3af624c81426f1427"
},
"request" => "/",
"@version" => "1",
"event" => {
"timezone" => "+08:00",
"module" => "nginx",
"dataset" => "nginx.access"
},
"response" => "200",
"verb" => "GET",
"bytes" => "3700",
"httpversion" => "1.1",
"fileset" => {
"name" => "access"
},
"input" => {
"type" => "log"
},
"clientip" => "192.168.0.203",
"auth" => "-"
}
同样的我们通过浏览器访问nginx首页然后再看下日志输出,可以看到如下内容:
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"httpversion" => "1.1",
"request" => "/",
"fileset" => {
"name" => "access"
},
"httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
"tags" => [
[0] "filebeat",
[1] "203",
[2] "beats_input_codec_plain_applied"
],
"req_timestamp" => "18/Aug/2020:21:56:32 +0800",
"body_sent" => 0,
"@version" => "1",
"log" => {
"file" => {
"path" => "/var/log/nginx/access.log"
},
"offset" => 885
},
"request_verb" => "GET",
"service" => {
"type" => "nginx"
},
"req_user" => "-",
"req_ip" => "192.168.0.108",
"ecs" => {
"version" => "1.5.0"
},
"input" => {
"type" => "log"
},
"resp_status" => 304,
"@timestamp" => 2020-08-18T13:56:33.268Z
}
{
"httpversion" => "1.1",
"request" => "/",
"fileset" => {
"name" => "access"
},
"httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
"tags" => [
[0] "filebeat",
[1] "203",
[2] "beats_input_codec_plain_applied"
],
"req_timestamp" => "18/Aug/2020:21:56:32 +0800",
"body_sent" => 0,
"@version" => "1",
"log" => {
"file" => {
"path" => "/var/log/nginx/access.log"
},
"offset" => 673
},
"request_verb" => "GET",
"service" => {
"type" => "nginx"
},
"req_user" => "-",
"req_ip" => "192.168.0.108",
"ecs" => {
"version" => "1.5.0"
},
"input" => {
"type" => "log"
},
"resp_status" => 304,
"@timestamp" => 2020-08-18T13:56:32.268Z
}
{
"httpversion" => "1.1",
"request" => "/",
"fileset" => {
"name" => "access"
},
"httpagent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36 Edg/84.0.522.59",
"tags" => [
[0] "filebeat",
[1] "203",
[2] "beats_input_codec_plain_applied"
],
"req_timestamp" => "18/Aug/2020:21:56:31 +0800",
"body_sent" => 0,
"@version" => "1",
"log" => {
"file" => {
"path" => "/var/log/nginx/access.log"
},
"offset" => 461
},
"request_verb" => "GET",
"service" => {
"type" => "nginx"
},
"req_user" => "-",
"req_ip" => "192.168.0.108",
"ecs" => {
"version" => "1.5.0"
},
"input" => {
"type" => "log"
},
"resp_status" => 304,
"@timestamp" => 2020-08-18T13:56:32.268Z
}
通过kibana的discover模块查看日志信息
访问http://192.168.0.203:5601 登陆后点击discover模块选择beat-*索引
可以看到如下图的界面：
可以看到访问nginx的日志正常展示在界面上了，我们设置的filter也生效了。
至此filebeat收集nginx日志搭建完成。