filebeat报错dropping too large message of size

filebeat报错:

dropping too large message of size 1714620.

原因:

kafka对每一条消息的大小进行了限制。

解决

kafka端

修改config/server.properties,添加以下配置

max_message_bytes=10000000
replica.fetch.max.bytes=10000000

修改后重启kafka

filebeat端

修改filebeat.yaml,添加以下配置
max_message_bytes: 10000000

output.kafka:
  hosts: ["kafka1:9092","kafka2:9092","kafka3:9092"]
  topic: 'topiclog'
  max_message_bytes: 10000000
  required_acks: -1

添加后重启filebeat

你可能感兴趣的:(kafka,filebeat,elk)