Kafka开发实战(三)-Kafka API使用

上一篇Kafka开发实战(二)-集群环境搭建文章中,我们已经搭建起了Kafka集群了,接下来我们通过代码演示如何发布、订阅消息。
1、添加maven依赖

<dependency>
            <groupId>org.apache.kafkagroupId>
            <artifactId>kafka-clientsartifactId>
            <version>0.9.0.1version>
        dependency>

我使用的Kafka版本是0.9.0.1,下面来看看Kafka Producer代码

2、KafkaProducer

package com.ricky.codelab.kafka;

import java.io.IOException;
import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;

import com.ricky.codelab.kafka.util.PropertyUtils;

public class KafkaProducerDemo {

    private int total = 1000000;

    public static void main(String[] args) {

        new KafkaProducerDemo().send();
    }

    public void send(){

        long start = System.currentTimeMillis();
        System.out.println("Kafka Producer send msg start,total msgs:"+total);

        // set up the producer
        Producer producer = null;
        try {
            Properties props = PropertyUtils.load("producer_config.properties");
            producer = new KafkaProducer<>(props);

            for (int i = 0; i < total; i++){
                producer.send(new ProducerRecord("hello",
                        String.valueOf(i), String.format("{\"type\":\"test\", \"t\":%d, \"k\":%d}", System.currentTimeMillis(), i)));

                // every so often send to a different topic
                if (i % 1000 == 0) {
                    producer.send(new ProducerRecord("test", String.format("{\"type\":\"marker\", \"t\":%d, \"k\":%d}", System.currentTimeMillis(), i)));
                    producer.send(new ProducerRecord("hello", String.format("{\"type\":\"marker\", \"t\":%d, \"k\":%d}", System.currentTimeMillis(), i)));

                    producer.flush();
                    System.out.println("Sent msg number " + i);
                }

            }

        } catch (IOException e) {
            e.printStackTrace();
        }finally{
            producer.close();
        }

        System.out.println("Kafka Producer send msg over,cost time:"+(System.currentTimeMillis()-start)+"ms");
    }
}


KafkaProducer在构造时需要传入一个Properties对象来传入Kafka相关配置信息,producer_config.properties如下:

bootstrap.servers=172.18.19.206:9092,172.18.19.207:9092,172.18.19.208:9092
acks=all
retries=0
batch.size=16384
linger.ms=1
buffer.memory=33554432
auto.commit.interval.ms=1000
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

3、KafkaConsumer

package com.ricky.codelab.kafka;

import java.io.IOException;
import java.util.Arrays;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import com.ricky.codelab.kafka.util.PropertyUtils;

public class KafkaConsumerDemo {

    public static void main(String[] args) {

        new KafkaConsumerDemo().consume();
    }

    public void consume() {

        JsonParser jsonParser = new JsonParser();  

        // and the consumer
        KafkaConsumer consumer = null;
        try {
            Properties props = PropertyUtils.load("consumer_config.properties");
            consumer = new KafkaConsumer<>(props);

            //subscribe topics
            consumer.subscribe(Arrays.asList("hello", "test"));

            while (true) {
                ConsumerRecords records = consumer.poll(100);
                for (ConsumerRecord record : records){

//                  System.out.printf("offset -> %d, key -> %s, value -> %s",
//                          record.offset(), record.key(), record.value());

                    switch (record.topic()) {
                        case "hello":

                            JsonObject jObj = (JsonObject)jsonParser.parse(record.value()); 
                            switch (jObj.get("type").getAsString()) {
                            case "test":

                                long latency = System.currentTimeMillis() - jObj.get("t").getAsLong();
                                System.out.println(latency);

                                break;
                            case "marker":

                                break;
                            default:
                                break;
                            }
                        break;
                        case "test":

                            break;
                        default:
                            throw new IllegalStateException("Shouldn't be possible to get message on topic " + record.topic());
                    }
                }
            }
        } catch (IOException e) {
            e.printStackTrace();
        }finally{
            if(consumer!=null){
                consumer.close();
            }
        }
    }
}

KafkaConsumer在构造时也需要传入java.util.Properties对象,来告诉它Kafka相关的配置信息,consumer_config.properties如下:

bootstrap.servers=172.18.19.206:9092,172.18.19.207:9092,172.18.19.208:9092
group.id=test
enable.auto.commit=true
auto.commit.interval.ms=1000
session.timeout.ms=30000
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer

在KafkaProducer和KafkaConsumer都使用到了PropertyUtils类,它的作用是将.properties文件转换为java.util.Properties对象,代码如下:

package com.ricky.codelab.kafka.util;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;

import org.apache.commons.io.IOUtils;

public class PropertyUtils {

    private PropertyUtils(){

    }

    public static Properties load(File file) throws IOException{

        InputStream in = null;
        try {
            in = new FileInputStream(file);
            Properties props = new Properties();
            props.load(in);

            return props;

        }finally{
            IOUtils.closeQuietly(in);
        }
    }

    public static Properties load(String path) throws IOException{

        InputStream in = null;
        try {
            in = PropertyUtils.class.getClassLoader().getResourceAsStream(path);
            Properties props = new Properties();
            props.load(in);

            return props;

        }finally{
            IOUtils.closeQuietly(in);
        }
    }
}

4、打包执行
将KafkaProducerDemo.java、KafkaConsumerDemo.java 分别单独打成CLI jar包(KafkaProducer.jar和KafkaConsumer.jar),然后分别启动 KafkaProducer.jar和KafkaConsumer.java,就能看到控制台输出信息了。


点此下载完整项目代码


参考资料:
KafkaProducer javadoc:http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

KafkaConsumer javadoc:http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

你可能感兴趣的:(MQ)