spring boot2.x使用kafka消息队列(以及低版本springboot1.5x整合kafka后的问题)
项目中经常会用到kafka作为消息中间件，有用kafka记录收集日志，也有用kafka传消息异步处理业务逻辑，如果kafka处理业务逻辑，则要考虑相关业务是否要保证消息可靠性。记录一下springboot使用kafka的项目demo,以及配置文件如何配置。
一、依赖
<dependency>
<groupId>org.springframework.kafkagroupId>
<artifactId>spring-kafkaartifactId>
dependency>
注：
这个spring-kafka依赖里面包含了kafka-clients依赖，而kafka-client的版本要最好和kafka服务器版本一致，不然会因为版本不兼容，有连接的各种问题，下面的异常记录有写版本不一致遇到的问题
二、生产者
1.yaml配置
#自定义的配置，不是springboot提供的自动配置
mq:
kafka:
producer:
clusters: 192.168.240.42:9092,192.168.240.43:9092,192.168.240.44:9092
topic: KAFKA_TEST_TOPIC
2.引入上面的yaml的配置以及kafka的一些配置参数
@Data
@Configuration
@ConfigurationProperties("mq.kafka.producer")
public class KafkaProducerProperties {
/**
* 其他配置
*/
private Map<String, String> other;
/**
* 集群地址
*/
private String clusters;
/**
* 0 Producer 往集群发送数据不需要等到集群的返回，不确保消息发送成功。安全性最低但是效率最高。
* 1 Producer 往集群发送数据只要 Leader 应答就可以发送下一条，只确保 Leader 接收成功。
* -1 或 all Producer 往集群发送数据需要所有的ISR Follower 都完成从 Leader 的同步才会发送下一条，
* 确保 Leader 发送成功和所有的副本都成功接收。安全性最高，但是效率最低。
*/
private String acks = "all";
/**
* 重试次数
*/
private int retries = 0;
/**
* 将同分区数据合并为一个 batch 发送， 单位 bytes
*/
private String batchSize = "16384";
/**
* 发送延迟时间，当 batch 未满 batchSize 时，延迟 lingerMs 发送 batch
* 当数据达到 batchSize 将会被立即发送
*/
private String lingerMs = "1";
/**
* 缓冲大小，当缓冲满时，max.block.ms 会控制 send() && partitionsFor() 阻塞时长
*/
private String bufferMemory = "33554432";
/**
* bytes
*/
private String maxRequestSize = "5048576";
/**
* 压缩方式 none, gzip, snappy, lz4, or zstd.
*/
private String compressionType = "gzip";
/**
* key序列化方式，实现 org.apache.kafka.common.serialization.StringSerializer 的类
*/
private String keySerializerClass = "org.apache.kafka.common.serialization.StringSerializer";
/**
* value序列化方式，实现 org.apache.kafka.common.serialization.StringSerializer 的类
*/
private String valueSerializerClass = "org.apache.kafka.common.serialization.StringSerializer";
}
3.kafka的配置Config类
@Configuration
@EnableKafka
public class KafkaConfig {
@Autowired
private KafkaProducerProperties producerProperties;
@Bean
public ProducerFactory<Integer, String> producerFactory() {
return new DefaultKafkaProducerFactory<>(producerConfigs());
}
@Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>(16);
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, producerProperties.getClusters());
props.put(ProducerConfig.ACKS_CONFIG, producerProperties.getAcks());
props.put(ProducerConfig.RETRIES_CONFIG, producerProperties.getRetries());
props.put(ProducerConfig.BATCH_SIZE_CONFIG, producerProperties.getBatchSize());
props.put(ProducerConfig.LINGER_MS_CONFIG, producerProperties.getLingerMs());
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, producerProperties.getBufferMemory());
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, producerProperties.getMaxRequestSize());
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, producerProperties.getCompressionType());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, producerProperties.getKeySerializerClass());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, producerProperties.getValueSerializerClass());
if (producerProperties.getOther() != null){
props.putAll(producerProperties.getOther());
}
return props;
}
@Bean
public KafkaTemplate<Integer, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
注：这里的配置Config类主要是把配置注入KafkaTemplate，然后其他地方注入这个kafkaTemplate发送消息到kafka
4.发送kafka消息
@RestController
@RequestMapping(value = "/product")
public class ProductController {
@Value("${mq.kafka.producer.topic}")
private String topic;
@Autowired
private KafkaTemplate<Integer, String> kafkaTemplate;
@RequestMapping(value = "/sendMsg")
public String sendMsg(){
String msg = "发送kafka消息";
kafkaTemplate.send(topic,msg);
return msg;
}
}
注:
SpringBoot2.0已经提供了Kafka的自动配置，可以在application.properties文件中配置,所以也可以不用像我上面这样写工厂类，自己注入kafka配置
5.springboot2.0的kafka自动配置
#============== kafka ===================
# 指定kafka server的地址，集群配多个，中间，逗号隔开
spring.kafka.bootstrap-servers=127.0.0.1:9092
#=============== provider =======================
# 写入失败时，重试次数。当leader节点失效，一个repli节点会替代成为leader节点，此时可能出现写入失败，
# 当retris为0时，produce不会重复。retirs重发，此时repli节点完全成为leader节点，不会产生消息丢失。
spring.kafka.producer.retries=0
# 每次批量发送消息的数量,produce积累到一定数据，一次发送
spring.kafka.producer.batch-size=16384
# produce积累数据一次发送，缓存大小达到buffer.memory就发送数据
spring.kafka.producer.buffer-memory=33554432
#可以设置的值为：all, -1, 0, 1
spring.kafka.producer.acks=1
# 指定消息key和消息体的编解码方式
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
三、消费者
1.我们在消费者服务引入yaml配置
#自定义配置
mq:
kafka:
consumer:
groupId: kafka-server
clusters: 192.168.240.42:9092,192.168.240.43:9092,192.168.240.44:9092
topic: KAFKA_TEST_TOPIC
2.读取上面的配置以及kafka消费端的配置
@Data
@Configuration
@ConfigurationProperties(prefix = "mq.kafka.consumer")
public class KafkaConsumerProperties {
private Map<String, String> other;
private String clusters ;//通过yaml的配置注入
private String groupId ;
private boolean enableAutoCommit = true;//是否自动提交
private String sessionTimeoutMs = "120000";
private String requestTimeoutMs = "160000";
private String fetchMaxWaitMs = "5000";
private String autoOffsetReset = "earliest";
private String maxPollRecords = "10";//批量拉取消息数量
private String maxPartitionFetch = "5048576";
private String keyDeserializerClass = "org.apache.kafka.common.serialization.StringDeserializer";
private String valueDeserializerClass = "org.apache.kafka.common.serialization.StringDeserializer";
}
3.KafkaConfig配置类
@Configuration
@EnableKafka
public class KafkaConfig {
@Autowired
private KafkaConsumerProperties consumerProperties;
@Bean
ConcurrentKafkaListenerContainerFactory<Integer, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
// //设置为批量消费，每个批次数量在Kafka配置参数中设置ConsumerConfig.MAX_POLL_RECORDS_CONFIG
factory.setBatchListener(true);
//设置手动提交ackMode ,手动批量提交
// factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
return factory;
}
@Bean
public ConsumerFactory<Integer, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
@Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>(16);
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, consumerProperties.getClusters());
props.put(ConsumerConfig.GROUP_ID_CONFIG, consumerProperties.getGroupId());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, consumerProperties.isEnableAutoCommit());
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, consumerProperties.getSessionTimeoutMs());
props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, consumerProperties.getRequestTimeoutMs());
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, consumerProperties.getFetchMaxWaitMs());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, consumerProperties.getAutoOffsetReset());
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, consumerProperties.getMaxPollRecords());
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, consumerProperties.getMaxPartitionFetch());
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, consumerProperties.getKeyDeserializerClass());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, consumerProperties.getValueDeserializerClass());
if (consumerProperties.getOther() != null){
props.putAll(consumerProperties.getOther());
}
return props;
}
}
4.使用@KafkaListener来监听kafka服务器的topic的消息
@Slf4j
@Component
public class KafkaMessageListener {
@KafkaListener(topics = "${mq.kafka.consumer.topic}",groupId = "${mq.kafka.consumer.groupId}")
public void applyStatusSync(ConsumerRecords<Integer, String> records) {
log.info("kafka poll ：count:{}", records.count());
for (ConsumerRecord<Integer, String> record : records) {
log.info("接收到的kafka 数据 {}", record.value());
}
}
//手动提交偏移量
// @KafkaListener(topics = "${mq.kafka.topic}",groupId = "${mq.kafka.consumer.groupId}")
// public void applyStatusSync(ConsumerRecords records, Acknowledgment ack) {
// log.info("kafka poll ：count:{}", records.count());
// for (ConsumerRecord record : records) {
// log.info("接收到的kafka 数据 {}", record.value());
// }
ack.acknowledge();
// }
}
四、测试验证
1.分别启动生产者、消费者两个服务
2.服务都正常启动，调用一下生产者发kafka消息的接口
3.可以消费端已经打印了kafka接收到的消息，验证没有问题
4.项目目录
五、异常问题记录
1.在springboot1.5.10也是引入了spring-kafka，配置也是如上面的，但是项目启动后一直连接不上kafka,提示Bootstrap broker 192.168.xxx.xx:9092 disconnected
分析原因：是因为springboot1.5.10中kafka-client的版本为0.10.1.1,低版本的kafka-client连不上kafka服务器，是kafka-client和kafka服务器版本不对应的兼容性问题
解决方法：在引入的spring-kafka中用exclusion排除kafka-client，然后重新引入一个0.10.2.1的依赖，重启，成功连上kafka服务器
2.在kafka消费端的监听器，看到有拉取回消息，但是一直提示数据转换异常
MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [org.apache.kafka.clients.consumer.ConsumerRecords]
Bean [com.xinlin.demo.comsumer.listen.KafkaMessageListener@1f43cab7]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [org.apache.kafka.clients.consumer.ConsumerRecords] for GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}], failedMessage=GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [org.apache.kafka.clients.consumer.ConsumerRecords] for GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}], failedMessage=GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1327) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1316) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1232) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1203) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1123) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:938) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:751) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:700) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [org.apache.kafka.clients.consumer.ConsumerRecords] for GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}], failedMessage=GenericMessage [payload=发送kafka消息, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@416f4587, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=null, kafka_receivedPartitionId=7, kafka_receivedTopic=KAFKA_TEST_TOPIC, kafka_receivedTimestamp=1596119315437}]
at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:292) ~[spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:79) ~[spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:50) ~[spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1278) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:1261) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1222) [spring-kafka-2.2.8.RELEASE.jar:2.2.8.RELEASE]
分析原因：其实提示错误可能有两个原因导致的
1）一个是设置了不允许批量消费
factory.setBatchListener(false);但是监听器写的是ConsumerRecords
factory.setBatchListener(true);
2）除了上面的原因，还有一个原因就是spring-kafka的版本太低了，低版本的批量拉取消息不是用ConsumerRecords
我在springboot2.1.7引入spring-kafka没有问题，因为这里的spring-kafka版本是2.2.8，但是在springboot1.5.10引入的spring-kafka版本是1.1.7
3.设置手动提交ack，spring1.5和spring2.1配置轻微区别
//springboot1.5的配置方式：
//设置手动提交ackMode ,手动批量提交
factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL);
//springboot2.0的配置方式：
//设置手动提交ackMode ,手动批量提交
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
注：
只是AbstractMessageListenerContainer.AckMode.MANUAL和ContainerProperties.AckMode.MANUAL的枚举类的位置区别