2020-07-30
filebeat采集多个目录日志
预期效果：在kibana中建立不同的所以，通过索引可以查到不同日志的信息
（如果不同目录下的日志格式和类型一致，可以输出到logstash的同一端口，反之应该分开，即一个日志就需要写一个filebeat配置文件+一个logstash配置文件）
采集目标日志的路径：
C:\testlog\test.log（例：2020-07-30 15:05:54 | INFO | This is a test log13, kkkk55!）
C:\testlog1\test1.log（例：2020-07-30 14:56:30,674 [112] DEBUG performanceTrace 1120 http://api.114995.com:8082/api/Carpool/QueryMatchRoutes 183.205.134.240 null 972533 310000 TITTL00 HUAWEI 860485038452951 3.1.146 HUAWEI 5.1 113.552344 33.332737 发送响应完成 Exception:(null)）
（两个日志的内容格式不同）
filebeat配置(两个)
配置文件路径：D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64\filebeat.yml
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
- c:\testlog\*.log
#fields:
# level: debug
# review: 1
#fields:
#log_tag: erp
multiline:
pattern: '^\d{4}-\d{1,2}-\d{1,2}'
negate: true
match: after
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
#输出到logstash的5044端口
hosts: ["172.10.0.108:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
配置文件路径：D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64\filebeat1.yml
（拷贝第一个filebeat.yml文件重命名并修改内容）
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
- c:\testlog1\*.log
#fields:
# level: debug
# review: 1
#fields:
#log_tag: erp
multiline:
pattern: '^\d{4}-\d{1,2}-\d{1,2}'
negate: true
match: after
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
#输出到logstash的5045端口
hosts: ["172.10.0.108:5045"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
logstash配置(两个)
配置文件路径：D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\config\logstash.yml
input {
beats {
port => 5044
}
}
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} \| %{LOGLEVEL:log_level} \| (?.*)"
}
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
}
output {
stdout{ codec => rubydebug }
elasticsearch {
hosts => ["http://172.10.0.108:9200"]
index => "filebeat-%{+YYYY.MM.dd}"
}
}
配置文件路径：D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\config\logstash.yml（拷贝修改）
input {
beats {
port => 5045
}
}
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{NUMBER:thread:int}\] %{DATA:level} (?[a-zA-Z]+) %{NUMBER:executeTime:int} %{URI:url} %{IP:clientip} %{USERNAME:UserName} %{NUMBER:userid:int} %{NUMBER:AreaCode:int} (?[0-9a-zA-Z]+[-]?[0-9a-zA-Z]+) (?[0-9a-zA-Z]+[-]?[0-9a-zA-Z]+) %{NUMBER:DeviceId:int} (?[0-9a-z\.]+) %{NUMBER:Sdk:float} %{NUMBER:Lng:float} %{NUMBER:Lat:float} (?.*)"
}
remove_field => "message"
}
date {
match => ["timestamp","dd/MMM/YYYY:H:m:s Z"]
remove_field => "timestamp"
}
}
output {
stdout{ codec => rubydebug }
elasticsearch {
hosts => ["http://172.10.0.108:9200"]
index => "filebeat-test1-%{+YYYY.MM.dd}"
}
}
依次启动程序
powershell执行（一共开启4个powershell终端）
数据存储路径：D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\config
cd D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\bin
.\logstash.bat -f ../config/logstash.conf
数据存储路径：D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\config\logstash1（自己新建）
cd D:\Linux\data\ELK-Windows\elk\logstash-7.8.0\bin
.\logstash.bat -f ../config/logstash1.conf --path-data=..\config\logstash1
默认的数据存储路径是D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64\data
cd D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64
.\filebeat.exe -e -c .\filebeat.yml
默认的数据存储路径是D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64\data1
（注意：data1原本是没有的，需要自己创建）
cd D:\Linux\data\ELK-Windows\elk\filebeat-7.8.0-windows-x86_64
.\filebeat.exe -e -c .\filebeat1.yml --path-data=.\data1
（必须指定新的数据存储路径，否则执行失败）
测试
1.分别在两个日志文件中新加几条日志记录，并保存
2.观察http://172.10.0.108:9100/是否生成新的索引
a1.png
3.kibana新增索引，然后搜索数据
b1.png
c1.png