kafka CommitFailedError

2019独角兽企业重金招聘Python工程师标准>>>

线上kafka出现如下异常

2018-09-16 23:59:58,712-pid:20908-MainThread-consumer.py[line:782]-WARNING:Auto offset commit failed for group robot: CommitFailedError: Commit cannot be completed since the group has already
            rebalanced and assigned the partitions to another member.
            This means that the time between subsequent calls to poll()
            was longer than the configured max_poll_interval_ms, which
            typically implies that the poll loop is spending too much
            time message processing. You can address this either by
            increasing the rebalance timeout with max_poll_interval_ms,
            or by reducing the maximum size of batches returned in poll()
            with max_poll_records.

根据提示，老老实实修改max_poll_interval_ms，出现如上原因可能是消息的处理时间超过max_poll_interval_ms时间间隔，导致kafka认为该消费者挂掉，重新进行reblancing。

修改后max_poll_interval_ms>消息处理的时间即可

max.poll.interval.ms ，它表示最大的poll数据间隔，如果超过这个间隔没有发起pool请求，但heartbeat仍旧在发，就认为该consumer处于 livelock状态，进行reblancing

session.timeout.ms 消费者超时时间，如果消费者再超过了这个时间还没有发送心跳，那么kafka则会认为该消费者挂掉了，会重新进行reblancing。session.timeout.ms与消息的处理时间没有关系

转载于:https://my.oschina.net/u/732798/blog/2054811