ELKF日志管理平台搭建教程（一）-轻量级的日志传输工具（Filebeat）

由于工作原因，需要搭建一套自己的ELKF日志管理平台收集系统日志并分析，Filebeat 已经完全替代了 Logstash-Forwarder 成为新一代的日志采集器，同时鉴于它轻量、安全等特点，越来越多人开始使用它。

本专栏是参照 开源搜索与分析Elastic官网 文档作为主要依据，系统应用示意图

  

上图出自于 SpringCloud实战 专栏，这两个专栏共同组成了上面的系统应用示意图。从上图可以看出Filebeat主要是在192.168.3.206上做日志搜集工作，搜集到的日志信息会被输出到Logstash进行过滤，过滤完成的数据会被存储到ElasticSearch中，然后Kibana会去ElasticSearch中查询数据并展示在监控平台上。

一、Filebeat简介

Filebeat是一个轻量级日志传输Agent，可以将指定日志转发到Logstash、Elasticsearch、Kafka、Redis等中。Filebeat占用资源少，而且安装配置也比较简单，支持目前各类主流OS及Docker平台，图示如下：

    

二、准备工作 

环境是Centos7，需要下载Filebeat安装包并上传服务器 

Filebeat官方下载地址

本专栏是基于官网最新版本进行搭建的，Filebeat版本是7.1.1，如图所示：

  

下载LINUX 64-BIT就可以了。 

三、Filebeat部署

我将Filebeat安装包上传到了/usr/local下面，图示如下：

  

然后我们解压它

$ tar -zxvf filebeat-7.1.1-linux-x86_64.tar.gz
$ mv ./filebeat-7.1.1-linux-x86_64 ./filebeat

 解压完成后修改配置文件

    $ vim ./filebeat/filebeat.yml

需要修改以下内容

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /usr/local/logs/eureka-server/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  fields:
    machine: master
    app: eureka-server

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #  hosts: ["localhost:9200"]

  # Enabled ilm (beta) to use index lifecycle management instead daily indices.
  #ilm.enabled: false

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.3.203:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

上面一共改动了三个地方，首先在file.inputs.paths标明你的系统日志所在位置，然后注释掉ElasticSearch output中内内容让它不直接输出到ElasticSearch，修改Logstash output中的output.logstash.hosts为你的Logstash所在机器，如图所示：

  

这里 filebeat.inputs.paths 配置你的eureka-server模块日志文件所在路径，这里可以用正则表达式匹配日志文件，这里可以配置多个日志路径。

我们如果要生成日志文件，需要在工程中添加logback配置文件，指定日志文件生成为止和命名规则

logback-spring.xml配置

将上述日志配置文件放到resources文件夹下就可以了。

  

大家注意，这里的 filebeat.inputs 是可以定义多个的，每个 filebeat.inputs 对应一个微服务模块日志配置，我们可以在fields下面自定义一些属性，然后在logstash的配置文件中获取到这个自定义属性，通过判断自定义属性来对每个模块做个性化索引配置。

  

  

因为我们的日志管理平台是需要通过filebeat将日志输入到logstash中进行过滤，所以我们的output配置logstash所在机器的ip地址。

启动/停止filebeat命令

    $ nohup ./filebeat -e -c filebeat.yml &
    $ ps -ef |grep filebeat
    $ kill -9  进程号

 到此Filebeat搭建完成。