kafka 如何动态设置并发

kafka 并发

springboot setConcurrency 决定kafka 并发，kafka 的分区决定了并发上限 ，concurrency<=分区数 ，如果大于分区数并发会等于分区

每个分区都是一个有序的、不可变的记录序列，这些记录不断附加到结构化提交日志。分区中的记录被分配给一个顺序 ID 号，称为唯一标识分区内每个记录的偏移量。

KafkaBootstrapConfiguration

@SuppressWarnings(“rawtypes”)
@Bean(name = KafkaListenerConfigUtils.KAFKA_LISTENER_ANNOTATION_PROCESSOR_BEAN_NAME)
@Role(BeanDefinition.ROLE_INFRASTRUCTURE)
public KafkaListenerAnnotationBeanPostProcessor kafkaListenerAnnotationProcessor() {
return new KafkaListenerAnnotationBeanPostProcessor();
}

@Bean(name = KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME)
public KafkaListenerEndpointRegistry defaultKafkaListenerEndpointRegistry() {
return new KafkaListenerEndpointRegistry();
}

ConcurrentKafkaListenerContainerFactory

ConcurrentMessageListenerContainer 会进行创建 3 个数 KafkaMessageListenerContainer（多个）

KafkaMessageListenerContainer（多个）

for (int i = 0; i < this.concurrency; i++) {
    KafkaMessageListenerContainer container;
    if (topicPartitions == null) {
        container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties);
    }
    else {
        container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties,
                partitionSubset(containerProperties, i));
    }
    if (getBeanName() != null) {
        container.setBeanName(getBeanName() + "-" + i);
    }
    if (getApplicationEventPublisher() != null) {
        container.setApplicationEventPublisher(getApplicationEventPublisher());
    }
    container.setClientIdSuffix("-" + i);
    container.start();
    this.containers.add(container);
}

KafkaMessageListenerContainer（多个）中包含 ListenerConsumer

SimpleAsyncTaskExecutor（ CustomizableThreadCreator AsyncListenableTaskExecutor TaskExecutor） 用于启动执行consumer

如何动态设置并发

 @GetMapping("/stop")
    public String stop(){
        KafkaListenerEndpointRegistry  kafkaListenerEndpointRegistry=
                context.getBean(KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME,
                        KafkaListenerEndpointRegistry.class);

        Collection containers= kafkaListenerEndpointRegistry.getListenerContainers();
        ConcurrentMessageListenerContainer concurrentMessageListenerContainer
                =(ConcurrentMessageListenerContainer)containers.iterator().next();
        concurrentMessageListenerContainer.stop();
        return "";
    }

    @GetMapping("/start")
    public String start(int num){
        KafkaListenerEndpointRegistry  kafkaListenerEndpointRegistry=
                context.getBean(KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME,
                        KafkaListenerEndpointRegistry.class);

        Collection containers= kafkaListenerEndpointRegistry.getListenerContainers();
        ConcurrentMessageListenerContainer concurrentMessageListenerContainer
                =(ConcurrentMessageListenerContainer)containers.iterator().next();
        concurrentMessageListenerContainer.setConcurrency(num);
        concurrentMessageListenerContainer.start();
        return "";
    }

如上，通过获取到KafkaListenerEndpointRegistry 然后可以获取到对应的管理的 containers ,通过重启ConcurrentMessageListenerContainer，重新创建MessageListenerContainer数量达到动态修改业务并发，但是注意同一组的topic将发生重平衡。