(十三)ATP应用测试平台——springboot集成kafka案例实战

前言

kafka消息中间键也是我们经常要用到的功能，尤其是在大数据、高并发的项目中，如日志收集、业务数据分发等等。其最核心的俩大功能作用是:①削峰填谷②异步解耦。本节我们主要介绍一下如何在springboot项目中集成kafka消息中间键，实现简单的数据分发以及消费的案例。

正文

• kafka集群搭建

快速搭建一个kafka集群，我们这里以docker环境为例，搭建一个kafka集群，具体搭建过程请参考以下博客，这里不再赘述。

（六）史上最强ELK集群搭建系列教程——kafka集群搭建_北溟的博客-CSDN博客

• springboot项目引入kafka依赖

	org.springframework.kafka
	spring-kafka

• 引入kafka配置

spring:
  #kafka配置
  kafka:
    #kafka集群地址
    bootstrap-servers: 192.168.56.10:9091,192.168.56.10:9092,192.168.56.10:9093
    producer:
      #批量发送的数据量大小
      batch-size: 16384
      #可用发送数量的最大缓存
      buffer-memory: 33554432
      #key序列化器
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      #value序列化器
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      properties:
        linger.ms: 1
    consumer:
      enable-auto-commit: false
      auto-commit-interval: 100ms
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties:
        session.timeout.ms: 15000
      group-id: atp-group-id

• 创建一个kafka生产者生产消息

①创建一个kafka生产者类KafkaProducer生产消息

package com.yundi.atp.platform.kafka;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Component;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.util.concurrent.ListenableFutureCallback;

/**
 * @Author: yanp
 * @Description: kafka生产者
 * @Date: 2021/11/3 18:31
 * @Version: 1.0.0
 */
@Component
@Slf4j
public class KafkaProducer {
    @Autowired
    private KafkaTemplate kafkaTemplate;

    public void sendMsg(String topic, String msg) {
        log.info("开始发送kafka消息:[topic:{},msg:{}]", topic, msg);
        ListenableFuture> sendMsg = kafkaTemplate.send(topic, msg);
        //消息确认
        sendMsg.addCallback(new ListenableFutureCallback>() {
            @Override
            public void onFailure(Throwable throwable) {
                log.error("send error:[ex:{},topic:{},msg:{}]", throwable, topic, msg);
            }

            @Override
            public void onSuccess(SendResult stringStringSendResult) {
                log.info("send success:[topic:{},msg:{}]", topic, msg);
            }
        });
    }
}

 ②通过web服务发送生产消息测试

package com.yundi.atp.platform.module.test.controller;

import com.yundi.atp.platform.common.Result;
import com.yundi.atp.platform.kafka.KafkaProducer;
import io.swagger.annotations.Api;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * @Author: yanp
 * @Description:
 * @Date: 2021/11/4 17:32
 * @Version: 1.0.0
 */
@Api(tags = {"Springboot集成Kafka测试"})
@RestController
@RequestMapping(value = "/kafka")
public class KafkaContoller {
    @Autowired
    private KafkaProducer kafkaProducer;

    @GetMapping(value = "/sendKafkaTestMsg/{topic}/{msg}")
    public Result sendKafkaTestMsg(@PathVariable(value = "topic")String topic,@PathVariable(value = "msg")String msg){
        kafkaProducer.sendMsg(topic,msg);
        return Result.success();
    }
}

③通过测试页面发送生产者消息

 

• 创建kafka消费者消费消息

package com.yundi.atp.platform.kafka;

import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

/**
 * @Author: yanp
 * @Description: kafka消费者
 * @Date: 2021/11/3 18:32
 * @Version: 1.0.0
 */
@Component
@Slf4j
public class KafkaConsumer {

    /**
     * 消费者：监听test主题，分组：groupId1
     */
    @KafkaListener(topics = {"test"}, groupId = "groupId1")
    public void consumer1(ConsumerRecord record) {
        log.info("consumer1 start consume message:[topic:{},msg:{}]", record.topic(), record.value());
    }

    /**
     * 消费者：监听atp主题，分组：groupId1
     */
    @KafkaListener(topics = {"atp"}, groupId = "groupId1")
    public void consumer2(ConsumerRecord record) {
        log.info("consumer2 start consume message:[topic:{},msg:{}]", record.topic(), record.value());
    }

    /**
     * 消费者：监听atp主题，分组：groupId3
     */
    @KafkaListener(topics = {"atp"}, groupId = "groupId2")
    public void consumer3(ConsumerRecord record) {
        log.info("consumer3 start consume message:[topic:{},msg:{}]", record.topic(), record.value());
    }


}

• 结果测试

①发送生产者消息

②消费者消费消息

结语

ok,到这里关于springboot集成kafka案例实战就结束了，我们下期见。。。