spring boot中使用kafka详解(踩完坑又爬了出来)
一.概述
本文会使用两种方式使用kafka：直接使用kafka-client连接kafka服务；另一种是使用spring-kafka框架来连接kafka。
二.Compatibility（兼容性）：
springboot：2.2.1.RELEASE
gradle：
compile ('org.apache.kafka:kafka-clients:2.4.0')
compile ('org.springframework.kafka:spring-kafka:2.3.3.RELEASE')
在使用kafka的时候因为版本兼容性的问题没少踩坑。主要的版本兼容体现在springboot与spring-kafka之间，spring-kafka和kafka-client之间，kafka-client和kafka服务端之间的兼容。
spring官网给出的spring-kafka与kafka-clients的兼容对比参照表是这样的：
具体的版本依赖也可以参考mvnrepository。
根据博主的实践感觉kafka服务端只要是在0.10.0.1以上，kafka-clients高版本都是支持的，kafka官网也说了服务端和客户端之间是版本兼容的，但是亲测发现低于0.10.0.1版本的服务端是真的高版本兼容不了。
三.异常现象参考：
1.如果在启动kafka的时候配的ip是连接不上的ip那么在启动或者发送消息到kafka的时候就会出现类似这样的异常：
2020-01-29 17:02:49.034 -DEBUG [] [ad | producer-7] o.a.k.c.NetworkClient : [Producer clientId=producer-7] Initiating connection to node -12.kafka-client和服务端不兼容在发送message到kafka的时候就会一直输出如下log，一直到超时：
2020-01-29 15:19:48.006 -DEBUG [] [ad | producer-7] o.a.k.c.NetworkClient : [Producer clientId=producer-7] Disconnecting from node -1 due to request timeout.
2020-01-29 15:19:48.006 -DEBUG [] [ad | producer-7] kClient$DefaultMetadataUpdater : [Producer clientId=producer-7] Give up sending metadata request since no node is available
2020-01-29 15:19:48.056 -DEBUG [] [ad | producer-7] kClient$DefaultMetadataUpdater : [Producer clientId=producer-7] Give up sending metadata request since no node is available
2020-01-29 15:19:48.106 -DEBUG [] [ad | producer-7] kClient$DefaultMetadataUpdater : [Producer clientId=producer-7] Give up sending metadata request since no node is available
2020-01-29 15:19:48.157 -DEBUG [] [ad | producer-7] kClient$DefaultMetadataUpdater : [Producer clientId=producer-7] Initialize connection to node 127.0.0.1:9092 (id: -1 rack: null) for sending metadata request
2020-01-29 15:19:48.157 -DEBUG [] [ad | producer-7] o.a.k.c.NetworkClient : [Producer clientId=producer-7] Initiating connection to node 127.0.0.1:9092 (id: -1 rack: null) using address /127.0.0.1
2020-01-29 15:19:48.158 -DEBUG [] [ad | producer-7] o.a.k.c.n.Selector : [Producer clientId=producer-7] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
2020-01-29 15:19:48.158 -DEBUG [] [ad | producer-7] o.a.k.c.NetworkClient : [Producer clientId=producer-7] Completed connection to node -1. Fetching API versions.
2020-01-29 15:19:48.158 -DEBUG [] [ad | producer-7] o.a.k.c.NetworkClient : [Producer clientId=producer-7] Initiating API versions fetch from node -1.3.kafka-client和spring-kafka版本不匹配，项目不能成功启动
2020-01-29 17:08:48.853 -DEBUG [] [ad | producer-6] o.a.k.c.p.i.Sender : Starting Kafka producer I/O thread.
2020-01-29 17:08:48.853 - INFO [] [ main] .a.k.c.u.AppInfoParser$AppInfo : Kafka version : 0.10.0.1
2020-01-29 17:08:48.853 - INFO [] [ main] .a.k.c.u.AppInfoParser$AppInfo : Kafka commitId : a7a17cdec9eaa6c5
2020-01-29 17:08:48.853 -DEBUG [] [ main] o.a.k.c.p.KafkaProducer : Kafka producer started
2020-01-29 17:08:48.859 -DEBUG [] [ main] pathLoggingApplicationListener : Application started with classpath: unknown4.springboot和spring-kafka不匹配
java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration.kafkaTemplate
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:60) ~[spring-boot-autoconfigure-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:184) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:144) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:120) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:337) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:242) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:275) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:95) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-5.2.1.RELEASE.jar:5.2.1.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.1.RELEASE.jar:2.2.1.RELEASE]四.直接使用kafka-client
本文重点讲Producer（发送消息到kafka）。
1.启动kafka服务，可以参照博主的另一篇https://mp.csdn.net/postedit/103386177，给出本次使用的docker-compose文件
version: '2'
services:
zookeeper:
container_name: zookeeper
hostname: zookeeper
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
container_name: kafka
hostname: kafka
image: wurstmeister/kafka:0.10.0.1-1
ports:
- "9092:9092"
- "9093:9093"
links:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_HOST_NAME: 10.48.24.**（此处使用本地ip）
JMX_PORT: 9093
KAFKA_ADVERTISED_PORT: 9092
KAFKA_DELETE_TOPIC_ENABLE: 'true'
KAFKA_LOG_RETENTION_HOURS: 1
KAFKA_MESSAGE_MAX_BYTES: 10000000
KAFKA_REPLICA_FETCH_MAX_BYTES: 10000000
KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS: 60000
KAFKA_NUM_PARTITIONS: 2
KAFKA_DELETE_RETENTION_MS: 1000
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka-manager:
container_name: kafka-manager
hostname: kafka-manager
image: sheepkiller/kafka-manager
ports:
- "9000:9000"
links:
- zookeeper
- kafka
environment:
ZK_HOSTS: zookeeper:2181
APPLICATION_SECRET: letmein
kafdrop:
container_name: kafdrop
hostname: kafdrop
image: thomsch98/kafdrop
ports:
- "9010:9010"
environment:
ZK_HOSTS: zookeeper:2181
LISTEN: 9010
首先要对kafka-client进行配置：
package com.neuralyzer.common.kafka;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Properties;
import java.util.concurrent.Future;
import static com.neuralyzer.common.NeuralyzerLog.errorLog;
import static com.neuralyzer.common.NeuralyzerLog.serviceLog;
public class kafkaProducerTemplate {
private static KafkaProducer kafkaProducer;
private static Properties producerConfigs() {
Properties props = new Properties();
props.put("bootstrap.servers", "127.0.0.1:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
return props;
}
static {
Properties props = producerConfigs();
kafkaProducer = new KafkaProducer<>(props);
}
public static void sendMessage(String topicName, String message) {
try {
serviceLog.info("Send data to kafka, topic name:{}, message:{}",topicName, message);
ProducerRecord msg = new ProducerRecord<>(topicName, message);
Future future = kafkaProducer.send(msg);
RecordMetadata recordMetadata = future.get();
} catch (Exception e) {
errorLog.error("kafka send happen error.", e);
}
}
}
配置参数详细情况可参考https://www.cnblogs.com/cxuanBlog/p/11862698.html
五.使用sprng-kafka
因为spring-boot提供了对spring-kafka的自动装配，所以直接在application配置文件中配置就可以，当然也可以手动配置。
kafka:
bootstrap-servers: 127.0.0.1:9092
listener:
concurrency: 3
template:
default-topic: neuralyzer-st-backup
producer:
retries: 0
batch-size: 10
buffer-memory: 33554432
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
properties:
linger.ms: 1package com.neuralyzer.common.kafka;
import lombok.SneakyThrows;
import lombok.extern.log4j.Log4j2;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.ProducerListener;
import org.springframework.stereotype.Component;
import static com.neuralyzer.common.NeuralyzerLog.errorLog;
import static com.neuralyzer.common.NeuralyzerLog.serviceLog;
@Log4j2
@Component
public class KafkaProducer {
private final KafkaTemplate kafkaTemplate;
@Autowired
public KafkaProducer(KafkaTemplate kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void sendMessage(String topicName, String message) {
serviceLog.info("Send data to kafka, topic name:{}, message:{}",topicName, message);
kafkaTemplate.send(topicName, message);
kafkaTemplate.setProducerListener(new ProducerListener() {
@SneakyThrows
@Override
public void onError(ProducerRecord producerRecord, Exception exception) {
errorLog.error("Send data to kafka happen exception", exception);
throw exception;
}
});
}
}