kafka max.partition.fetch.bytes

kafka-0.9 该配置项的解释

max.partition.fetch.bytes

The maximum amount of data per-partition the server will return. The maximum total memory used for a request will be #partitions * max.partition.fetch.bytes. This size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying to fetch a large message on a certain partition.

int

1048576

[0,...]

high

kafka-0.10.0 该配置项的解释

max.partition.fetch.bytes

The maximum amount of data per-partition the server will return. The maximum total memory used for a request will be #partitions * max.partition.fetch.bytes. This size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying to fetch a large message on a certain partition.

int

1048576

[0,...]

high

kafka-0.10.1 最新的未发布的版本

max.partition.fetch.bytes

The maximum amount of data per-partition the server will return. If the first message in the first non-empty partition of the fetch is larger than this limit, the message will still be returned to ensure that the consumer can make progress. The maximum message size accepted by the broker is defined viamessage.max.bytes (broker config) ormax.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size

int

1048576

[0,...]

high

这个绝对是一个大坑，之前在spark streaming里面就因为这个导致丢数据。当时的解决方案是——spark streaming里面是用kafka 底层API来取数据，所以直接改该部分代码，判断fetch失败原因，如果本次fetch失败是因为max.partition.fetch.bytes导致的，则将该值double，然后重试。

转载于:https://my.oschina.net/tyronecai/blog/787253