kafka多线程消费

1、zookeeper集群搭建：https://blog.csdn.net/qq_31289187/article/details/80933365

2、kafka集群搭建：https://blog.csdn.net/qq_31289187/article/details/80955228

3、kafka生成消息：https://blog.csdn.net/qq_31289187/article/details/81809014

4、kafka多线程消费：offset从zookeeper中得到，一个线程对应一个partition，这样消费速度很快，而且消息的顺序可控，线程数量和partition一样，多了浪费资源，少了效率很低，也可以不通过zookeeper来消费，kafka0.9以后的版本就可以将offset记录到对应消费group到对应的broker上。

5、pom.xml

    4.0.0

    com.cn.dl
    kafka-consumer1
    1.0
    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                
                    1.6
                    1.6
                
            
        
    
    
        
            org.apache.kafka
            kafka_2.10
            0.8.2.2
        
        
            com.alibaba
            fastjson
            1.2.43
        
    

6、KafkaConsumterMain

package com.dl.cn;

import java.io.IOException;
import java.util.Properties;

/**
 * Created by tiger on 2018/8/20.
 */
public class KafkaConsumterMain {
    public static void main(String[] args) throws IOException {
        String topic = "user-info";
        int threadNum = 2;
        Properties properties = ReadPropertiesUtils.readConfig("config.properties");
        KafkaConsumterServer kafkaConsumterDemo = new KafkaConsumterServer(topic,threadNum,properties);
        kafkaConsumterDemo.consumer();
    }
}

7、KafkaConsumterServer

package com.dl.cn;

import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * Created by tiger on 2018/8/20.
 */
public class KafkaConsumterServer {
    private String topic;
    private Properties properties;
    private int threadNum;

    public KafkaConsumterServer(String topic,int threadNum,Properties properties) {
        this.topic = topic;
        this.threadNum = threadNum;
        this.properties = properties;
    }
    /**
     * 创建固定线程池消费消息
     * 线程和partition一对一
     * */
    public void consumer() {
        Map topicCountMap = new HashMap();
        topicCountMap.put(topic, new Integer(threadNum));
        ConsumerConfig consumerConfig = new ConsumerConfig(properties);
        ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(consumerConfig);
        Map>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List> streams = consumerMap.get(topic);
        //创建固定数量的线程池
        ExecutorService executor = Executors.newFixedThreadPool(threadNum);
        for (KafkaStream stream : streams) {
            executor.submit(new KafkaConsumerThread(stream));
        }
    }
}

8、KafkaConsumerThread

package com.dl.cn;

import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.message.MessageAndMetadata;

/**
 * Created by tiger on 2018/8/20.
 */
public class KafkaConsumerThread implements Runnable{
    private KafkaStream stream;

    public KafkaConsumerThread(KafkaStream stream) {
        this.stream = stream;
    }

    @Override
    public void run() {
        ConsumerIterator it = stream.iterator();
        while (it.hasNext()) {
            MessageAndMetadata mam = it.next();
            System.out.println(Thread.currentThread().getName() + ">>>partition[" + mam.partition() + "],"
                    + "offset[" + mam.offset() + "], " + new String(mam.message()));
        }
    }
}

9、ReadPropertiesUtils

package com.dl.cn;

import java.io.IOException;
import java.io.InputStream;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.Set;

/**
 * Created by tiger on 2018/8/18.
 */
public class ReadPropertiesUtils {
    /**
     * 读取properties配置文件
     * @param configFileName
     * @exception
     * @return
     * */
    public static Properties readConfig(String configFileName) throws IOException {
        Map config = new HashMap();
        InputStream in = ReadPropertiesUtils.class.getClassLoader().getResourceAsStream(configFileName);
        Properties properties = new Properties();
        properties.load(in);
        return properties;
    }
}

10、config.properties

 

11、测试结果：

线程1对应partition0，线程2对应partition1，两者互不干扰

12、

auto.offset.reset=smallest，意思是从topic最早数据开始消费
auto.offset.reset=largest，是从topic最新数据开始消费

在zk中可以看到消费组

比如在代码中用到tiger7777这个消费者组

在代码中看到线程2最后消费的消息offset=1755

线程1最后消费的消息offset=2243

zookeeper中记录的offset值

生产者不断生产数据，消费者不断消费数据

 

将tiger7777，中partition对应的offset的值更新为200，然后重新启动

消费者，发现消息从offset=200开始重新消费，而且发现只有一个线程在继续消费