elk+filebeat 日志收集搭建
1．环境准备
a) 启动四台linux虚拟机
b) 安装JDK1.8
2. 下载并安装ES(x.x.x.130)
下载地址：
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.2.tar.gz
安装ES：
tar -xvf elasticsearch-6.5.2.tar.gz
配置ES：
cd elasticsearch-6.5.1/config/
vim elasticsearch.yml
cluster.name: my-test-es #es集群的名称
path.data: /application/es/data #自定义数据存放的文件夹
path.logs: /application/es/logs #自定义日志存放的文件夹
network.host: x.x.x.130 #es所在主机的ip地址
http.port: 9200 #es的端口号
启动ES：
cd elasticsearch-6.5.2/bin
./elasticsearch
3. 下载并安装kibana(x.x.x.129)
下载地址：
https://www.elastic.co/downloads
选择kibana 下载
安装kibana：
tar -xvf kibana-6.5.1-linux-x86_64.tar.gz
配置kibana
server.port: 5601 #kibana的端口号
server.host: "x.x.x.129" #kibana 所在服务器的ip
server.name: "centos001-kibana" #kibana 的名称
elasticsearch.url: http://x.x.x.130:9200 #es的地址
启动Kibana
cd kibana-6.5.1-linux-x86_64/bin/
./kibana
4. 下载安装logStash(x.x.x.131)
下载地址：
https://www.elastic.co/downloads
选择logStash下载
安装logStash
tar -xvf logstash-6.5.1.tar.gz
安装logStash插件
cd logstash-6.5.1
vim Gemfile
修改：source "https://ruby.taobao.org/"，替代source "https://rubygems.org"
. /bin/logstash-plugin install --no-verify
配置logStash
------------------------使用logStash采集日志信息配置开始-----------------------------------------
新建配置文件test.conf
cd config
vim test.conf
输入以下内容：
input{
file{
path => "/application/logstash/testLog/test.log"
start_position => beginning
}
}
filter{
grok{
match=>{
"message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}"
}
}
}
output{
elasticsearch {
hosts => ["http://x.x.x.130:9200"]
index => "logstash-testlog-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
#stdout{codec=>rubydebug}
}
新建test.log
55.3.244.1 GET /index.html 15824 0.043
------------------------使用logStash采集日志信息配置结束-----------------------------------------
------------------------使用filebeat采集日志信息，logStash过滤日志信息配置开始--------------
新建配置文件test.conf
cd config
vim test.conf
输入以下内容：
input{
beats {
port => 5044
}
}
filter{
grok{
match=>{
"message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}"
}
}
}
output{
elasticsearch {
hosts => ["http://x.x.x.130:9200"]
index => "logstash-testlog-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
#stdout{codec=>rubydebug}
}
-------------------------使用filebeat采集日志信息，logStash过滤日志信息配----------------------
启动logStash
cd logstash-6.5.1/
./bin/logstash -f config/test.conf
5. 下载安装filebeat
下载：
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.2-linux-x86_64.tar.gz
安装：
tar xzvf filebeat-6.5.2-linux-x86_64.tar.gz
配置：修改文件 filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/test.log
#- c:\programdata\elasticsearch\logs\*
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["x.x.x.131:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
新建test.log
55.3.244.1 GET /index.html 15824 0.043
启动filebeat
sudo chown root filebeat.yml
sudo ./filebeat -e
6.filebeat读取多个log文件，采用不同的过滤方式，输出到es的不同index中
Filebeat中的配置，filebeat.yml修改为：
filebeat.inputs:
- type: log
enabled: true
paths:
#- /var/log/log1.log
fields:
# level: debug
# review: 1
log_source: log1
- type: log
enabled: true
paths:
#- /var/log/log2.log
fields:
# level: debug
# review: 1
log_source: log2
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.6.131:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
logStash中的配置文件修改为：
input{
beats {
port => 5044
}
}
filter{
if [fields][log_source] == log1" {
grok {
match => {
"message"=>""#这里写自己的处理
}
}
}
if [fields][log_source] == "log2" {
grok {
match=>{
"message"=>""# 这里写自己的处理方式
}
}
}
mutate { #这里删除一些信息 可选
remove_field =>["message"]
remove_field =>["beat"]
remove_field =>["host"]
#remove_field =>["fields"]
remove_field =>["input"]
remove_field =>["prospector"]
}
}
output{
if [fields][log_source] == log1" {
elasticsearch {
hosts => ["http://x.x.x.130:9200"]
index => "log1"
#user => "elastic"
#password => "changeme"
}
}
if [fields][log_source] == "log2" {
elasticsearch {
hosts => ["http://x.x.x.130:9200"]
index => "log2"
#user => "elastic"
#password => "changeme"
}
}
}