docker filebeat 将日志多级目录和多维json数据日志同步到es

注

使用的时候先调试调试配置,调试成功在尝试写入es,如果es写入失败就是es账户.密码/白名单.和index未创建的问题,细节可以留言

setup.template.priority 模板优先级 调整这个可以配置一台机器多个filebeat 容器启动

多级目录日志和多维josn日志结构

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - '/home/data/logs/test/*.log'
      - '/home/data/logs/upload/logs/**/*.log'
    json.keys_under_root: true  
    json.add_error_key: true
    json.message_key: json  

processors:
  - decode_json_fields:
      fields: ["json"]
      process_array: false
      max_depth: 1
      target: ""

docker 直接启动

以docker形式启动filebeat

docker run -d \
  --name=filebeat \
  --user=root \
  --volume="/home/golang/gopath/filebeat/test/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="/home/data/logs/test:/home/data/logs/test/logs:ro" \
  docker.elastic.co/beats/filebeat:8.6.2 filebeat -e --strict.perms=false 


#调试
docker run  \
  --name=filebeat \
  --user=root \
  --volume="/home/golang/gopath/filebeat/test/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="/home/data/logs/test:/home/data/logs/test/logs:ro" \
  docker.elastic.co/beats/filebeat:8.6.2 filebeat -e --strict.perms=false 

filebeat 配置 filebeat.docker.yml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
    
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - '/home/data/logs/test/*.log'
      - '/home/data/logs/upload/logs/**/*.log'
    json.keys_under_root: true  
    json.add_error_key: true
    json.message_key: json  

processors:
  - decode_json_fields:
      fields: ["json"]
      process_array: false
      max_depth: 1
      target: ""

output.elasticsearch:
  ssl.verification_mode: none
  hosts: ["https://127.0.0.1:9200"]
  username: "test"
  password: "12345"
  index: "metricbeat-test"

setup.template.name: "metricbeattest"
setup.template.pattern: "metricbeat-*"
setup.template.priority: 50

filebeat 调试配置

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - '/home/data/logs/test/*.log'
      - '/home/data/logs/upload/logs/**/*.log'
    json.keys_under_root: false
    json.add_error_key: true
    json.message_key: log

processors:
  - decode_json_fields:
      fields: ["log"]
      process_array: false
      max_depth: 1
      target: ""
      
      
  # 丢掉不需要的字段
  - drop_fields:
      fields: ["agent","input","ecs","message","host", "log","error","level"]

# 配置输出到控制台
output.console:
  pretty: true

setup.template.name: "metricbeat-test"
setup.template.pattern: "metricbeat-*"
setup.template.priority: 50