centos7安装filebeat6.2 收集java日志

安装filebeat6.2

$ cd /usr/local/src
$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.1-linux-x86_64.tar.gz
$ tar xf filebeat-6.2.1-linux-x86_64.tar.gz
$ cd  filebeat-6.2.1-linux-x86_64
$ cp -r filebeat /usr/local/bin

创建filebeat配置文件

$ mkdir /etc/filebeat
$ cd /etc/filebeat/
$ vi filebeat.yml     
path.config: /etc/filebeat
path.logs: /var/log/filebeat
path.config: /etc/filebeat
path.logs: /var/log/filebeat
path.data: /var/lib/filebeat/data
filebeat.registry_file: /var/lib/filebeat/registry
filebeat.shutdown_timeout: 0
logging.level: info
logging.metrics.enabled: false
logging.files.rotateeverybytes: 104857600
logging.files.keepfiles: 10
logging.files.permissions: 0600

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
filebeat.config:
    prospectors:
        enabled: true
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: true
        reload.period: 10s

output.logstash:  ##提前把logstash配置好,开启负载均衡模式
    hosts: ["10.100.200.47:5044","10.100.200.66:5044"]
    index: filebeat-%{+yyyy.MM.dd}

    loadbalance: true

编辑 各个需要收集日志项目的配置文件，这里统一放到了prospectors.d中。

示例：

$ mkdir prospectors.d/
$ cd prospectors.d/
$ cat preproduct-htjfapp-account-provider.yml
- type: log
  enabled: true
  paths:
      - /data/WEBLOG/preproduct-xx-xx-provider/*/*.log
  scan_frequency: 10s
  fields_under_root: true
  
  
  fields:
      
      env: preproduct
      
      index: logstash-htjf
      
      logtype: preproduct-xx-xx-provider
      
      topic: logstash-htjf
 
      vm_host: 10.100.12.125
      
      
  tail_files: false
  close_inactive: 2h
  close_eof: false
  close_removed: true
  clean_removed: true
  close_renamed: false

使用superviosrd 守护进程

修改supervisord.conf配置文件

[program:filebeat]
command=/usr/local/bin/filebeat  -c /etc/filebeat/filebeat.yml
autostart=true
autorestart=true
startsecs=5
priority=1
user=root
stopasgroup=true
killasgroup=true

热加载supervisord

$ supervisorctl  update
$ supervisorctl  status
filebeat                         RUNNING   pid 16957, uptime 18:16:15

logstash 的配置文件（日志格式化）

$ cat htjf_spring_boot.conf
input {
    beats { 
        port => "5044"
        codec=> multiline {
        pattern => "^%{YEAR}.*"  ##不是年份开头的行都归到上一行
        negate  => true
        what    => "previous"
        }
    }
}

filter {
   grok  {
        patterns_dir => [ "/etc/logstash/patterns.d" ]
        match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{THREADID:threadId}\]\s+\[%{THREADNAME:traceId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}","message", "%{TIMESTAMP_ISO8601:timestamp}\s+\[%{THREADID_1:threadId}\]\s+%{LOGLEVEL:level}\s+%{JAVACLASS:javaclass}\s+\-\s+%{JAVAMESSAGE:javameassage}"]
        remove_field => [ "message","beat","timestamp","topic","hostname","name"]
   }
   date {
        match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"]
        locale => "cn"
    }
}

output {
        elasticsearch {
          hosts => ["192.168.60.117:9200","192.168.60.118:9200","192.168.60.119:9200"]
          index => "logstash-htjf-filebeat-%{+YYYY-MM}"
          user => elastic
          password => xxx
        }
       stdout { codec => rubydebug }
}

启动logstash

$ /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/htjf_spring_boot.conf --path.data=/var/log/htjf_spring_boot.log & > /dev/null

可以结合计划任务(crond) 执行logstash启动的守护进程

$ cat  /etc/logstash/scripts/logstash_auto.sh
#!/bin/bash
htjf_spring_boot_pid=`ps -ef |grep java |grep -v  grep|grep htjf_spring_boot |awk '{print $2}'`
if [ -z $htjf_spring_boot_pid ];then
        /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/htjf_spring_boot.conf --path.data=/var/log/htjf_spring_boot.log & > /dev/null
fi

$ crontab -l
#logstash auto
*/5 * * * *  /bin/sh /etc/logstash/scripts/logstash_auto.sh

$ /etc/init.d/crond  restart

kibana 展示收集日志

filebeat收集日志时，新增了字段，这里是vm_host。kibana日志展示中出现了此字段 说明成功啦。。