Filebeat 收集 Elasticsearch 日志（qbit）

试验环境

CentOS 7.9
Elasticsearch 7.10.0
Kibana 7.10.0
Filebeat 7.10.0

试验步骤

ELK 安装

• EFK 的安装不赘述，Filebeat 安装配置官方文档：https://www.elastic.co/guide/...

Filebeat 配置

• 启用 elasticsearch 模块（elasticsearch.yml.disabled -> elasticsearch.yml）

filebeat modules elasticsearch

• 编辑模块配置文件 /etc/filebeat/modules.d/elasticsearch.yml

- module: elasticsearch
  server:
    enabled: true
    var.paths: 
      - /var/log/elasticsearch/elasticsearch_server.json

  gc:
    enabled: false

  audit:
    enabled: false

  slowlog:
    enabled: true
    var.paths: 
      - /var/log/elasticsearch/elasticsearch_index_search_slowlog.json

  deprecation:
    enabled: false

• 编辑 Filebeat 主配置文件 /etc/filebeat/filebeat.yml（语法与 logstash 一致）

# ============================== Filebeat inputs ===============================

filebeat.inputs:
- type: log
  enabled: false
- type: filestream
  enabled: false

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

# ================================== Outputs ===================================
setup.template.name: "filebeat-es"
setup.template.pattern: "filebeat-es-*"
setup.ilm.enabled: false
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
  indices:
    - index: "filebeat-es-log-%{+yyyy.MM.dd}"
      when.equals: 
        event.dataset: "elasticsearch.server"
    - index: "filebeat-es-slowlog-%{+yyyy.MM.dd}"
      when.equals: 
        event.dataset: "elasticsearch.slowlog"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

• 使用 filebeat -e 将日志打印到控制台便于测试
• 测试无误后启动 Filebeat 服务

sudo systemctl status filebeat.service

查看日志

• Kibana Dev Tools

GET _cat/indices/filebeat-es-*

• 创建 index pattern

• 在 Discover 中查看

本文出自 qbit snap