docker版ELK(ElasticSearch+Logstash+ Kibana)+filebeat日志分析平台的搭建(二)之filebeat+logstash的配置

制作适合自己的filebeat镜像

上一篇：elk的搭建

1. 编写filebeat.yml文件

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /logs/*.log
    - /web/*.log
  tags: ["192mycs-web-log"]
  fields:
    document_type: 192mycs-web-log

- type: log
  enabled: true
  paths:
    - /logs/*.log
  tags: ["192logs"]
  fields:
    document_type: 192syslogs

#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: true

#==================== Elasticsearch template setting ==========================
#
#setup.template.settings:
#  index.number_of_shards: 3
#    #index.codec: best_compression
#      #_source.enabled: false

     
#============================== Dashboards ====================================
setup.dashboards.enabled: true

#=============================== Logstash output ================================
output.logstash:
  # The Logstash hosts
    hosts: ["192.168.10.201:5044"]

##=========================elasticsearch output ================================
#output.elasticsearch:
   # index: "mycs-java"
#    hosts: ["192.168.10.201:9200"]
#    username: "filebeat_internal"
#    password: "{pwd}"

##============================== Kibana ========================================
setup.kibana:
    host: "192.168.10.201:5601"

2. 编写Dockerfile

FROM docker.elastic.co/beats/filebeat:6.6.1
MAINTAINER menard 2019-3-7
SHELL ["/bin/bash", "-c"]
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN filebeat modules enable logstash
RUN filebeat modules enable elasticsearch
RUN filebeat modules enable redis
RUN filebeat modules enable mysql
RUN filebeat modules enable nginx
RUN filebeat modules enable system
RUN filebeat setup

3. 制作镜像 docker build -t menard/filebeat:6.6.1 .
   
   

编写docker-compose.yml文件

需要注意的是日志路径要挂载到容器里，不然没有用

  version: '2'
  services:
  filebeat:
    image: "menard/filebeat:6.6.1"
    hostname: filebeat
    container_name: filebeat
    volumes:
      - /filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /logs/:/logs/
      - /web/:/web/
    environment:
      - TZ=Asia/Shanghai
    restart: always

配置logstash
新建filebeat.conf,该文件需要挂载到logstash容器的/etc/logstash/conf.d/目录下，配置好后重启容器

input {
    beats {
        port => "5044"
    }
}
 filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "192.168.10.201:9200" ]
        index => "%{[fields][document_type]}-%{+YYYY.MM.dd}"
    }
}

启动filebeat

1.docker-compose up -d启动filebeat容器
2.通过head查看es索引

kibana创建索引和查看数据

下一篇：elasticsearch设置密码