filebeat_OutPut 介绍

title: filebeat Output

date: 2017-06-11 05:22:31

categories: elk

tags: filebeat

Filbeat OutPut

Filebeat Elasticsearch OutPut

configure

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  template.enabled: true
  template.path: "filebeat.template.json"
  template.overwrite: false
  index: "filebeat"
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

定义ip port add portocalhttps

output.elasticsearch:
  hosts: ["localhost"]
  protocol: "https"
  username: "admin"
  password: "s3cr3t"

compression_level
- gzip 压缩级别 range（1-9）
work
- 发送事件到es的工作数量
index
- The index name to write events to.
- The default is "filebeat-%{+yyyy.MM.dd}" (for example,"filebeat-2015.04.26").
indices

支持条件，基于格式字符串的字段访问和名称映射的索引选择器规则的数组

- index 
    - 要使用的索引格式字符串。 如果使用的字段丢失，则规则失败。 
- mapping 
    - 映射字典为新名字
- default 
    - 默认字符串数值
- when  
    - 选择匹配条件

```
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "logs-%{+yyyy.MM.dd}"
  indices:
    - index: "critical-%{+yyyy.MM.dd}"
      when.contains:
        message: "CRITICAL"
    - index: "error-%{+yyyy.MM.dd}"
      when.contains:
        message: "ERR"

```

pipeline

格式字符串指定获取节点写入事件pipeline的id值

  output.elasticsearch:
    hosts: ["http://localhost:9200"]
    pipeline: my_pipeline_id

pipelines

filebeat.prospectors:
- paths: ["/var/log/app/normal/*.log"]
    fields:
    type: "normal"
- paths: ["/var/log/app/critical/*.log"]
    fields:
    type: "critical"

output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "filebeat-%{+yyyy.MM.dd}"
pipelines:
  - pipeline: critical_pipeline
       when.equals:
          type: "critical"
 - pipeline: normal_pipeline
       when.equals:
          type: "normal"

template

    output.elasticsearch:
      hosts: ["localhost:9200"]
      template.name: "filebeat"
      template.path: "filebeat.template.json"
      template.overwrite: false

templates.versions

output.elasticsearch:
  hosts: ["localhost:9200"]
  template.path: "filebeat.template.json"
  template.overwrite: false
  template.versions.2x.path: "filebeat.template-es2x.json

max_retries
- 当发送失败的时候，尝试多少次发送事件
bulk_max_size
- 单个Elasticsearch批量API索引请求中批量的最大事件数默认值为50
timeout
- The http request timeout in seconds for the Elasticsearch request
- The default is 90
flush_interval
- 在两个批量API索引请求之间等待新事件的秒数
ssl
- https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-ssl.html

Filebeat Logstash OutPut

需要logstash服务端安装beat插件使用lumberjack协议发送事件到logstash

output.logstash:
  hosts: ["localhost:5044"]

Metadata Fields
- @meatedata
  - Filebeat使用@metadata字段将元数据发送到Logstash
  - @metadata字段的内容只存在于Logstash中，不属于从Logstash发送的任何事件的一部分
  - 有关@metadata字段的更多信息，请参阅Logstash文档logstash doucument
```
  {
      ...
      "@metadata": { 
        "beat": "filebeat", 
        "type": "" 
      }   
  }
```

logstash to elasticsearch

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}

enabled
hosts
compression_level
worker

loadbalance

output.logstash:
  hosts: ["localhost:5044", "localhost:5045"]
  loadbalance: true
  index: filebeat

pipelining
- 异步处理事件，默认关闭
index
ssl
timeout
max_retries
bulk_max_size
- The maximum number of events to bulk in a single Logstash request
- The default is 2048.

kafka OutPut

The Kafka output sends the events to Apache Kafka.

output.kafka:
  # initial brokers for reading cluster metadata
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]

  # message topic selection + partitioning
  topic: '%{[type]}'
  partition.round_robin:
    reachable_only: false

  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

enabled
hosts
version
username
password
topic
topics
- topic
- mapping
- default
- when
partition
- random.group_events
- round_robin.group_events
- hash.hash
- hash.random
client_id
worker
codec
metadata
- refresh_frequency
- retry.max
- retry.backoff
max_retries
bulk_max_size
timeout
broker_timeout
channel_buffer_size
keep_alive
compression
max_message_bytes
required_acks
flush_interval
ssl

Redis OutPut

This output works with Redis 3.2.4.

output.redis:
  hosts: ["localhost"]
  password: "my_password"
  key: "filebeat"
  db: 0
  timeout: 5

enabled
hosts
port
index
key

    output.redis:
      hosts: ["localhost"]
      key: "%{[fields.list]:fallback}"

keys

key
mapping
default

when

  output.redis:
    hosts: ["localhost"]
    key: "default_list"
    keys:
      - key: "info_list"   # send to info_list if `message` field contains INFO
        when.contains:
          message: "INFO"
     - key: "debug_list"  # send to debug_list if `message` field contains DEBUG
       when.contains:
         message: "DEBUG"
     - key: "%{[type]}"
        mapping:
          "http": "frontend_list"
          "nginx": "frontend_list"
          "mysql": "backend_list"

passport
db
datatype

用于发布事件的Redis数据类型。如果数据类型为列表，则使用Redis RPUSH命令，并将所有事件添加到列表中，并在键下定义键。如果使用数据类型通道，则使用Redis PUBLISH命令，这意味着所有事件都被推送到Redis的pub / sub机制。通道的名称是键下定义的。默认值为列表。

codec
worker
loadbalance
timeout
max_retries
bulk_max_size
ssl
proxy_url
proxy_use_local_resolver

File OutPut

文件输出将事务转储到每个事务处于JSON格式的文件中。目前，该输出用于测试，但可以作为Logstash的输入

output.file:
  path: "/tmp/filebeat"
  filename: filebeat
  #rotate_every_kb: 10000
  #number_of_files: 7

enables
path
- 定义保存文件的路径
filename
- 定义保存文件的名字
rotate_every_kb
- 定义每个文件达到多少kb就开始切割
number_if_files
- 定义保存几份文件
codec

Console OutPut

output.console:
  pretty: true

pretty
codec
enabled
bulk_max_size

Codec Output

output.console:
  codec.json:
    pretty: true

output.console:
  codec.format:
    string: '%{[@timestamp]} %{[message]}'

Loggin OutPut

logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
  rotateeverybytes: 10MB 
  keepfiles: 7

Debugging

By default, Filebeat sends all its output to syslog. When you run Filebeat in the foreground, you can use the -e command line flag to redirect the output to standard error instead. For example:

filebeat -e

The default configuration file is filebeat.yml (the location of the file varies by platform). You can use a different configuration file by specifying
the -c flag. For example:

filebeat -e -c myfilebeatconfig.yml

You can increase the verbosity of debug messages by enabling one or more debug selectors. For example, to view the published transactions, you can start Filebeat with the publish selector like this:

filebeat -e -d "publish"

If you want all the debugging output (fair warning, it’s quite a lot), you can use *, like this:

filebeat -e -d "*"

support

https://www.elastic.co/support/matrix#show_compatibility