现象: 服务端接收到的消息集中在固定partition上 原因: kafka生产端默认按照消息key来路由具体的broker机器
解决: 不要指定key,如果需要保证业务上的顺序,就指定key为业务变量
错误代码示例:
String key = "aa";
producer.send(new ProducerRecord<>("TOIPICA",key,"这是内容")) //代码的问题在于,把消息的key硬编码了,导致消息固定往partition发送
分区策略源码 :DefaultPartitioner#partition
如果要保证消息的有序性,可以把保证有序的一类消息设置为同一个key,这样就能保证这类消息发送到同一个 partition 上,但一定要避免,所有的消息公用一个key 当时也可以通过自定义分区策略的方式,参考 生产者基本概念解释#指定分区
可能的原因:topic的partion数量小于应用数量,kafka消息并行依赖partition的个数,一个topic有多个partition,而每个partition只能分配给一个消费组下的具体实例,当partition数量小于消费组实例的个数时,自然就不够分了
解决:扩大topic的分区数,或者保证每台机器消费均衡的前提下减少consumer的并行数
现象:某系统2020-04-21 21:18:24的消息,21:18:42才被消费
原因:项目发布consumer消费下来的数据未处理成功, session timeout默认30s,导致30s后才能进行rebalance,rebalance后才进行消费之前的消息,导致消息消费延迟
解决:优雅起停,进程关闭的时候回调KafkaConsumer.close方法
示例代码:
@Bean(destroyMethod = "shutdown")
public KafkaConsumerDealer commonKafkaDealer() {
topicCallback.put(KafkaTopic.NOTIFY_SMS_TEMPLATE.name(), commSmsWrapperWorker);
KafkaConsumerDealer dealer = new KafkaConsumerDealer(threadName, servers, groupId,
topicCallback, envConstant.getWaitTime());
dealer.init();
return dealer;
}
现象:offset提交失败
原因:业务线程拉取offset超过session timeout默认配置的30s
解决:限制consumer拉取批次数或者优化业务处理性能
实例代码:
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,10); //设置每批次,拉取消息量,效率抵消,慎用
Consumer consumer = new KafkaConsumer<>(props);
现象:通过线程池多线程方式处理KafkaConsumer拉取的消息,应用关闭,未处理完线程池里面的数据,出现消息丢失
条件:1、应用关闭;2、消费者设置offset自动提交;3、线程池处理KafkaConsumer拉取的消息
原因:消费者设置自动提交, 调用KafkaConsumer#close的时候会自动提交一次offset, 而此时线程池有未处理完的 KafkaConsumer 拉取的消息
解决: 在关闭consumer后,优雅的进行线程池关闭,保证线程池拉取的数据处理完成
较之spring kafka ,原生客户端offset仅提供了自动offset的维护,手动offset需要自己实现,但整体维护成本低,客户端实现灵活
jar引入
org.apache.kafka
kafka-clients
0.10.0.1
版本
建议 0.10.0.1 版本与服务端保持一致
快速使用
生产者示例
@Configuration
public class KafkaConfig {
@Autowired
private Environment environment;
@Bean(name = "producer", destroyMethod = "close")
public KafkaProducer producer() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
// 0、1(默认的设置) 和 all
// 0 发送出去就不管了
// 1 意思就是说只要Partition Leader接收到消息而且写入本地磁盘了,就认为成功了,
// 不管他其他的Follower有没有同步过去这条消息了
// all Partition Leader接收到消息之后,还必须要求ISR列表里跟Leader保持同步的那些Follower都要把消息同步过去,
// 才能认为这条消息是写入成功了。
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, 3);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
//构造Producer对象,注意,该对象是线程安全的,一般来说,一个进程内一个Producer对象即可;
//如果想提高性能,可以多构造几个对象,但不要太多,最好不要超过5个
return new KafkaProducer<>(props);
}
}
//发送代码示例
ProducerRecord record = new ProducerRecord(this.topic, JSON.toJSONString(pushInfoDto));
Future future = producer.send(record);
消费者示例
@Configuration
public class KafkaConsumerConfig {
@Bean
public NewKafkaConsumerDealer kafkaDealer(
UicPushInfoReceiveWorker uicPushInfoReceiveWorker) {
//
String servers = "";
String groupId = "";
final String threadName = "KafkaDealer-";
Map topicCallback = Maps.newHashMap();
topicCallback.put("TOPICA", uicPushInfoReceiveWorker);
NewKafkaConsumerDealer dealer = new NewKafkaConsumerDealer(threadName, servers, groupId,
topicCallback);
return dealer;
}
}
//特别注意
NewKafkaConsumerDealer最好在容器关闭的时候关闭
public class NewKafkaConsumerDealer {
private static final Logger LOG = LoggerFactory.getLogger(
NewKafkaConsumerDealer.class);
private static final String THREAD_POOL_NAME_PREFIX = "NewKafkaConsumerDealer-Worker-";
private Consumer consumer;
private String threadName;
// 消费线程
private Thread consumerThread;
private volatile boolean running = false;
private BlockingQueue blockingDeque = new ArrayBlockingQueue(256);
private Properties props;
private ExecutorService pool = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors() * 2,
Runtime.getRuntime().availableProcessors() * 2, 120, TimeUnit.SECONDS,
blockingDeque, NamedThreadFactory.create(THREAD_POOL_NAME_PREFIX), new CallerRunsPolicy()
);
private final Map callbacks;
public NewKafkaConsumerDealer(String threadName, String servers,
String groupId
, Map callbacks) {
this.threadName = threadName;
this.callbacks = Collections.unmodifiableMap(callbacks);
Assert.notEmpty(callbacks);
props = new Properties();
props.put("bootstrap.servers", servers);
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("group.id", groupId);
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "3000");
props.put("session.timeout.ms", "30000");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 100);
LOG.info("consumer|init|success|topic:{}|servers:{}|groupId:{}", this.callbacks.keySet(),
servers,
groupId);
}
public void start() {
consumer = new KafkaConsumer<>(props);
consumer.subscribe(Lists.newArrayList(this.callbacks.keySet()));
running = true;
consumerThread = new Thread(() -> {
while (running) {
try {
//读取数据,读取超时时间为100ms
ConsumerRecords records = consumer.poll(500);
for (ConsumerRecord record : records) {
Callback callback = callbacks.get(record.topic());
if (callback == null) {
LOG.error("invalid topic|{}", record.topic());
continue;
}
pool.submit(new Worker(record, callback));
}
} catch (WakeupException exception) {
// 捕获 WakeupException consumer 准备退出
LOG.info("WakeupException 准备关闭consumer");
// 跳出循环,关闭consumer
break;
} catch (Exception exception) {
LOG.error("error cause by ", exception);
}
}
consumer.close();
boolean isTerminated = ExecutorUtils.shutdownAndAwaitTermination(pool, 1, TimeUnit.SECONDS);
LOG.info("consumer close success,poll terminated:{}", isTerminated);
});
consumerThread.setName(threadName);
consumerThread.start();
}
public void close() {
if (!running) {
LOG.info("consumer 不是运行状态,不需要关闭,threadName[{}]", threadName);
return;
}
Stopwatch stopwatch = Stopwatch.createStarted();
running = false;
consumer.wakeup();
try {
// 等待counsumer线程执行完成
consumerThread.join();
} catch (InterruptedException e) {
LOG.error("consumerThread.join InterruptedException", e);
}
LOG.info("consumer线程执行完成,threadName[{}],Spend time[{}]", threadName, stopwatch);
}
/**
* 添加消费者配置,会覆盖默认配置
*/
public void addConsumerProperties(String key, Object value) {
props.put(key, value);
}
}