Spring Boot Kafka优雅使用

前言

需要对Kafka有一定的了解

先抛出几个问题:

spring boot kafka使用

1. 如何配置多线程消费？
2. 如何手动管理Offset？
3. 如何命名多线程名称？

1. quick start

先快速开始体验一下

springBootVersion=1.5.17.RELEASE

jdk=1.8.0_161

// spring-kafka版本 2.2.0.RELEASE
compile("org.springframework.kafka:spring-kafka:2.2.0.RELEASE")

配置一下KafkaTemplate和KafkaConsumer

package com.dobest.bfas.collection.consumer;

import com.dobest.analytics.utils.IPUtil;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.*;
import org.springframework.kafka.listener.AbstractMessageListenerContainer;
import org.springframework.kafka.listener.ContainerProperties;

import java.util.HashMap;
import java.util.Map;

/**
 * @author: hujiansong
 */
@Configuration
public class KafkaTemplateConfig {

    @Bean
    public KafkaTemplate kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

    @Bean
    public ProducerFactory producerFactory() {
        return new DefaultKafkaProducerFactory<>(producerConfigs());
    }

    @Bean
    public Map producerConfigs() {
        Map props = new HashMap<>(16);
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.RETRIES_CONFIG, "3");
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 200);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
                "org.apache.kafka.common.serialization.StringSerializer");
        return props;
    }


    @Bean
    ConcurrentKafkaListenerContainerFactory
    kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(3);
//        factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
        return factory;
    }

    @Bean
    public ConsumerFactory consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map consumerConfigs() {
        Map props = new HashMap<>(16);
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "test.consumer");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
        props.put(ConsumerConfig.CLIENT_ID_CONFIG, IPUtil.getHostName() + "-h5-consumer");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return props;
    }
}

这样就配置好了，开袋即食用:

@Component
@Slf4j
public class SinkMessageListener implements CommandLineRunner {

    @Autowired
    private KafkaTemplate kafkaTemplate;

    // 指定topics，可以指定多个，最好每个topic对应一个
    @KafkaListener(id = "test-consumer", topics = {"realtime.analytics_h5_source"})
    public void listen(ConsumerRecord cr) throws Exception {
        log.debug(Thread.currentThread().getName() + "  {}", cr.value());
    }

    @Override
    public void run(String... args) throws Exception {
        log.info("start producer 10000 message ");
        for (int i = 0; i < 100; i++) {
            kafkaTemplate.send("realtime.analytics_h5_source", UUID.randomUUID().toString(), "i" + i);
        }
    }


}

回答问题

1. 如何配置多线程消费？

可以看到消费工厂，可以指定

factory.setConcurrency(3)

一行代码解决多线程消费问题

2. 如何手动管理Offset？

首先将

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

将自动提交设置为false

然后消费工厂设置提交模式

 factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);

AckMode 如下:

RECORD :当listener一读到消息，就提交offset

BATCH : poll() 函数读取到的所有消息,就提交offset

TIME : 当超过设置的ackTime ，即提交Offset

COUNT ：当超过设置的COUNT，即提交Offset

COUNT_TIME ：TIME和COUNT两个条件都满足，提交offset

MANUAL ： Acknowledgment.acknowledge()即提交Offset，和Batch类似

MANUAL_IMMEDIATE： Acknowledgment.acknowledge()被调用即提交Offset

count 和 time都可以通过factory的配置进行设置

 factory.getContainerProperties().setAckCount();
 factory.getContainerProperties().setAckTime();

设置完之后，就可以在Listen中回调

 @KafkaListener(id = "test-consumer", topics = {"realtime.analytics_h5_source"})
    public void listen(ConsumerRecord cr,Acknowledgment acknowledgment) throws Exception {
        log.debug(Thread.currentThread().getName() + "  {}", cr.value());
        // 注意: 比 quickstart 读了一个Acknowledgment参数
        acknowledgment.acknowledge();
    }

如果遇到Exception

java.lang.IllegalStateException: Consumer cannot be configured for auto commit for ackMode MANUAL_IMMEDIATE

修改配置，将自动提交关闭

props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

这样就可以手动提交offset了

3. 如何命名多线程?

大家都知道，见名知意的重要性，所以Spring-kafka可以对并发消费修改线程名字

@KafkaListener( topics = {"realtime.analytics_h5_source"})
    public void listen(ConsumerRecord cr) throws Exception {
    // 打印当前线程
        log.debug(Thread.currentThread().getName() + "  {}", cr.value());
    }

KafkaListener不加id得到的线程名字

link

KafkaListener 加id得到的线程名字:

@KafkaListener(id = "test-consumer", topics = {"realtime.analytics_h5_source"})
    public void listen(ConsumerRecord cr) throws Exception {
        log.debug(Thread.currentThread().getName() + "  {}", cr.value());
    }

link

总结

使用KafkaTemplate生产者步骤:

1. 配置producerConfigs
2. 配置producerFactory
3. 配置KafkaTemplate

使用KafkaLinstener消费者步骤

1. 配置consumerConfigs
2. 配置consumerFactory
3. 配置kafkaListenerContainerFactory

其中kafkaListenerContainerFactory可以设置并发数，offset提交模式。

参考

Spring-kafka-官方文档