kafka提交异常org.apache.kafka.clients.consumer.CommitFailedException

一、背景

       kafka使用版本为0.10.1.0，今天在测试kafkaConsumer的时候，代码如下

public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "192.168.80.132:9092");
        props.put("group.id", "testId");
        props.put("enable.auto.commit", "false");
        props.put("session.timeout.ms", "10000");
        props.put("max.poll.interval.ms", "3000");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList("kafka-topic-02"));
        try {
            while(true) {
                ConsumerRecords records = consumer.poll(100);
                System.out.println("*********开始休眠6s,模拟消息处理时间**********");
                Thread.sleep(6000);
                System.out.println("***************休眠结束*****************");
                consumer.commitSync();
            }
        }catch(Exception e) {
            e.printStackTrace();
        }finally {
            consumer.close();
        }

    一运行，报错了，报错如下，意思是我们手动提交的时候失败了，因为消费者组开启了rebalanced操作且将对应的分区分配给了其它消费者

*********开始休眠6s,模拟消息处理时间**********
***************休眠结束*****************
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:600)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:498)
    at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1104)
    at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1072)
    at com.tanjie.kafka.ConsumerDemo.main(ConsumerDemo.java:29)

      经过查看官网，发现参数max.poll.interval.ms(官网给得默认值为3000)的意思为，当我们从kafkaServer端poll消息时，poll()的调用之间的最大延迟。 这提供了消费者在获取更多记录之前可以空闲的时间量的上限。 如果在此超时到期之前未调用poll()，则认为使用者失败，并且消费者组将重新平衡以便将分区重新分配给其他消费者，而恰好这里我们设置了Thread.sleep(6000) > max.poll.interval.ms值，也就是我们在手动提交的时候，实际上分区信息已经被分配到整个消费者组里面的其它消费者了。

     了解了上面的原因，修改也很简单，第一是增大max.poll.interval.ms的值，不过我们在实际的生产中，可以测试一下，假如你每次poll 100条消息，每条消息处理N/s 那么最好将max.poll.interval.ms值设置为 > 100*N，这样就不会出现上面的异常了。

     经过修改后代码如下

public static void main(String[] args) {
		Properties props = new Properties();
		props.put("bootstrap.servers", "192.168.80.132:9092");
		props.put("group.id", "testId");
		props.put("enable.auto.commit", "false");
		props.put("session.timeout.ms", "10000");
		props.put("max.poll.interval.ms", "3000");
		props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
		props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
		KafkaConsumer consumer = new KafkaConsumer<>(props);
		consumer.subscribe(Arrays.asList("kafka-topic-02"));
		try {
			while(true) {
				ConsumerRecords records = consumer.poll(100);
				System.out.println("*********开始休眠2s,模拟消息处理时间**********");
				Thread.sleep(2000);
				System.out.println("***************休眠结束*****************");
				for(ConsumerRecord record : records) {
					System.out.println("offest:" + record.offset() + ";value: " + record.value());
				}
				consumer.commitSync();
			}
		}catch(Exception e) {
			e.printStackTrace();
		}finally {
			consumer.close();
		}
	}

     运行结果如下：

*********开始休眠2s,模拟消息处理时间**********
***************休眠结束*****************
*********开始休眠2s,模拟消息处理时间**********
***************休眠结束*****************
offest:18;value: 3f85df8b-4ec6-4b9d-be22-0c269ead55a9
offest:19;value: 34d8b20d-eb24-4cd5-a249-75b41a6fda5b
offest:20;value: dcc4bd2f-912d-4fda-b141-7c759ea3dc24
offest:21;value: d9a7db66-d9b5-4c0f-9287-f0197a7dcdf0
offest:14;value: 2dfab504-da05-4f38-a7be-cb1e0c7a8ea9
offest:15;value: c3170f1b-bbe6-4c58-8fda-c7a548ca4e3c
offest:18;value: 966faa1c-4cf1-44cd-a63e-818940fcb26a
offest:19;value: 7bb5b5aa-bd74-4c2c-9008-58714028a882
offest:20;value: 0f79cacf-c558-4696-985c-0ea0949acd4e
offest:21;value: 471f54a0-529f-4f54-b9fb-706c042839e2
*********开始休眠2s,模拟消息处理时间**********
***************休眠结束*****************
*********开始休眠2s,模拟消息处理时间**********
***************休眠结束*****************
*********开始休眠2s,模拟消息处理时间**********