深入浅出系列之 -- kafka消费者的三种语义模型

    本文主要详解kafka client的使用，包括kafka消费者的三种消费语义at-most-once，at-least-once，和exact-once message，生产者的使用等。

 

创建主题

bin/kafka-topics.sh --zookeeper localhost：2181 --create --topic normal --partitions 2 --rerelication-factor 1

 

生产者

public class ProducerExample {
   public static void main(String[] str) throws InterruptedException, IOException {
           System.out.println("Starting ProducerExample ...");
           sendMessages();
   }
   private static void sendMessages() throws InterruptedException, IOException {
           Producer producer = createProducer();
           sendMessages(producer);
           // Allow the producer to complete sending of the messages before program exit.
           Thread.sleep(20);
   }
   private static Producer createProducer() {
       Properties props = new Properties();
       props.put("bootstrap.servers", "localhost:9092");
       props.put("acks", "all");
       props.put("retries", 0);
       // Controls how much bytes sender would wait to batch up before publishing to Kafka.
       props.put("batch.size", 10);
       props.put("linger.ms", 1);
       props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
       props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
       return new KafkaProducer(props);
   }
   private static void sendMessages(Producer producer) {
       String topic = "normal-topic";
       int partition = 0;
       long record = 1;
       for (int i = 1; i <= 10; i++) {
           producer.send(
               new ProducerRecord(topic, partition,                                 Long.toString(record),Long.toString(record++)));
       }
   }
}

 

消费者

消费者注册到卡夫卡有多种方式：

订阅：这种方式在新增的话题或者分区或者消费者增加或者消费者减少的时候，会进行消费者组内消费者的再平衡。

分配：这种方式注册的消费者不会进行重新平衡。

上面两种方式都是可以实现，三种消费语义的。具体API的使用请看下文。

1.最多一次kafka消费者

最多一次消费语义是kafka消费者的默认实现。配置这种消费者最简单的方式是

1）enable.auto.commit设置为真。

2）auto.commit.interval.ms设置为一个较低的时间范围。

3）consumer.commitSync（）不要调用该方法。

由于上面的配置，就可以使得kafka有线程负责按照指定间隔提交偏移。但是这种方式会使得kafka消费者有两种消费语义：

消费语义最多一次 :

    消费者的偏移已经提交，但是消息还在处理，这个时候挂了，再重启的时候会从上次提交的偏移处消费，导致上次在处理的消息部分丢失。

消费语义最少一次:

    消费者已经处理完了，但是偏移还没提交，那么这个时候消费者挂了，就会导致消费者重复消费消息处理。但是由于auto.commit.interval.ms设置为一个较低的时间范围，会降低这种情况出现的概率。

代码如下：

public class AtMostOnceConsumer {
       public static void main(String[] str) throws InterruptedException {
           System.out.println("Starting  AtMostOnceConsumer ...");
           execute();
       }
       private static void execute() throws InterruptedException {
               KafkaConsumer consumer = createConsumer();
               // Subscribe to all partition in that topic. 'assign' could be used here
               // instead of 'subscribe' to subscribe to specific partition.
               consumer.subscribe(Arrays.asList("normal-topic"));
               processRecords(consumer);
       }
       private static KafkaConsumer createConsumer() {
               Properties props = new Properties();
               props.put("bootstrap.servers", "localhost:9092");
               String consumeGroup = "cg1";
               props.put("group.id", consumeGroup);
               // Set this property, if auto commit should happen.
               props.put("enable.auto.commit", "true");
               // Auto commit interval, kafka would commit offset at this interval.
               props.put("auto.commit.interval.ms", "101");
               // This is how to control number of records being read in each poll
               props.put("max.partition.fetch.bytes", "135");
               // Set this if you want to always read from beginning.
               // props.put("auto.offset.reset", "earliest");
               props.put("heartbeat.interval.ms", "3000");
               props.put("session.timeout.ms", "6001");
               props.put("key.deserializer",
                       "org.apache.kafka.common.serialization.StringDeserializer");
               props.put("value.deserializer",
                       "org.apache.kafka.common.serialization.StringDeserializer");
               return new KafkaConsumer(props);
       }
       private static void processRecords(KafkaConsumer consumer)  {
               while (true) {
                       ConsumerRecords records = consumer.poll(100);
                       long lastOffset = 0;
                       for (ConsumerRecord record : records) {
                               System.out.printf("\n\roffset = %d, key = %s, value = %s", record.offset(),                                             record.key(), record.value());
                               lastOffset = record.offset();
                        }
               System.out.println("lastOffset read: " + lastOffset);
               process();
               }
       }
       private static void process() throws InterruptedException {
               // create some delay to simulate processing of the message.
               Thread.sleep(20);
       }
}

2.至少一次kafka消费者

实现最少一次消费语义的消费者也很简单。

1）设置enable.auto.commit为假

2）消息处理完之后手动调用consumer.commitSync（）

这种方式就是要手动在处理完该次轮询得到消息之后，调用偏移异步提交函数consumer.commitSync（）。建议是消费者内部实现密等，来避免消费者重复处理消息进而得到重复结果。最多一次发生的场景是消费者的消息处理完并输出到结果库（也可能是部分处理完），但是偏移还没提交，这个时候消费者挂掉了，再重启的时候会重新消费并处理消息。

代码如下：

public class AtLeastOnceConsumer {
   public static void main(String[] str) throws InterruptedException {
           System.out.println("Starting AutoOffsetGuranteedAtLeastOnceConsumer ...");
           execute();
    }
   private static void execute() throws InterruptedException {
           KafkaConsumer consumer = createConsumer();
           // Subscribe to all partition in that topic. 'assign' could be used here
           // instead of 'subscribe' to subscribe to specific partition.
           consumer.subscribe(Arrays.asList("normal-topic"));
           processRecords(consumer);
    }
    private static KafkaConsumer createConsumer() {
           Properties props = new Properties();
           props.put("bootstrap.servers", "localhost:9092");
           String consumeGroup = "cg1";
           props.put("group.id", consumeGroup);
           // Set this property, if auto commit should happen.
           props.put("enable.auto.commit", "true");
           // Make Auto commit interval to a big number so that auto commit does not happen,
           // we are going to control the offset commit via consumer.commitSync(); after processing             // message.
           props.put("auto.commit.interval.ms", "999999999999");
           // This is how to control number of messages being read in each poll
           props.put("max.partition.fetch.bytes", "135");
           props.put("heartbeat.interval.ms", "3000");
           props.put("session.timeout.ms", "6001");
           props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
           props.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
           return new KafkaConsumer(props);
   }
    private static void processRecords(KafkaConsumer consumer) throws {
           while (true) {
                   ConsumerRecords records = consumer.poll(100);
                   long lastOffset = 0;
                   for (ConsumerRecord record : records) {
                       System.out.printf("\n\roffset = %d, key = %s, value = %s", record.offset(),                                         record.key(), record.value());
                       lastOffset = record.offset();
                   }
                   System.out.println("lastOffset read: " + lastOffset);
                   process();
                   // Below call is important to control the offset commit. Do this call after you
                   // finish processing the business process.
                   consumer.commitSync();
           }
   }
   private static void process() throws InterruptedException {
       // create some delay to simulate processing of the record.
       Thread.sleep(20);
   }
}

3.使用subscribe实现Exactly-once 

使用subscribe实现Exactly-once很简单，具体思路如下：

1）将enable.auto.commit设置为假。

2）不调用consumer.commitSync（）。

3）使用SUBCRIBE定于话题。

4）实现一个ConsumerRebalanceListener，在该监听器内部执行consumer.seek（topicPartition，偏移），从指定的主题/分区的偏移处启动。

5）在处理消息的时候，要同时控制保存住每个消息的偏移量。以原子事务的方式保存偏移和处理的消息结果。传统数据库实现原子事务比较简单。但对于非传统数据库，比如HDFS或者nosql的，为了实现这个目标，只能将偏移与消息保存在同一行。

6）实现密等，作为保护层。

代码如下：

public class ExactlyOnceDynamicConsumer {
      private static OffsetManager offsetManager = new OffsetManager("storage2");
       public static void main(String[] str) throws InterruptedException {
               System.out.println("Starting ExactlyOnceDynamicConsumer ...");
               readMessages();
       }
       private static void readMessages() throws InterruptedException {
               KafkaConsumer consumer = createConsumer();
               // Manually controlling offset but register consumer to topics to get dynamically
               //  assigned partitions. Inside MyConsumerRebalancerListener use
               // consumer.seek(topicPartition,offset) to control offset which messages to be read.
               consumer.subscribe(Arrays.asList("normal-topic"),
                               new MyConsumerRebalancerListener(consumer));
               processRecords(consumer);
       }
       private static KafkaConsumer createConsumer() {
               Properties props = new Properties();
               props.put("bootstrap.servers", "localhost:9092");
               String consumeGroup = "cg3";
               props.put("group.id", consumeGroup);
               // Below is a key setting to turn off the auto commit.
               props.put("enable.auto.commit", "false");
               props.put("heartbeat.interval.ms", "2000");
               props.put("session.timeout.ms", "6001");
               // Control maximum data on each poll, make sure this value is bigger than the maximum                   // single message size
               props.put("max.partition.fetch.bytes", "140");
               props.put("key.deserializer",                                 "org.apache.kafka.common.serialization.StringDeserializer");
               props.put("value.deserializer",                         "org.apache.kafka.common.serialization.StringDeserializer");
               return new KafkaConsumer(props);
       }
       private static void processRecords(KafkaConsumer consumer) {
           while (true) {
                   ConsumerRecords records = consumer.poll(100);
                   for (ConsumerRecord record : records) {
                           System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(),                                     record.key(), record.value());
                           // Save processed offset in external storage.
                           offsetManager.saveOffsetInExternalStore(record.topic(), record.partition(),                                             record.offset());
                   }
              }
       }
}
public class MyConsumerRebalancerListener implements                                 org.apache.kafka.clients.consumer.ConsumerRebalanceListener {
       private OffsetManager offsetManager = new OffsetManager("storage2");
       private Consumer consumer;
       public MyConsumerRebalancerListener(Consumer consumer) {
               this.consumer = consumer;
       }
       public void onPartitionsRevoked(Collection partitions) {
               for (TopicPartition partition : partitions) {
                   offsetManager.saveOffsetInExternalStore(partition.topic(), partition.partition(),                     consumer.position(partition));
               }
       }
       public void onPartitionsAssigned(Collection partitions) {
               for (TopicPartition partition : partitions) {
                       consumer.seek(partition,                             offsetManager.readOffsetFromExternalStore(partition.topic(),                             partition.partition()));
               }
       }
}
/**
* The partition offset are stored in an external storage. In this case in a local file system where 
* program runs.
*/
public class OffsetManager {
       private String storagePrefix;
       public OffsetManager(String storagePrefix) {
               this.storagePrefix = storagePrefix;
       }
   /**
       * Overwrite the offset for the topic in an external storage.
       *
       * @param topic - Topic name.
       * @param partition - Partition of the topic.
       * @param offset - offset to be stored.
       */
       void saveOffsetInExternalStore(String topic, int partition, long offset) {
           try {
               FileWriter writer = new FileWriter(storageName(topic, partition), false);
               BufferedWriter bufferedWriter = new BufferedWriter(writer);
               bufferedWriter.write(offset + "");
               bufferedWriter.flush();
               bufferedWriter.close();
           } catch (Exception e) {
                   e.printStackTrace();
                   throw new RuntimeException(e);
           }
       }
       /**
           * @return he last offset + 1 for the provided topic and partition.
       */
       long readOffsetFromExternalStore(String topic, int partition) {
               try {
                       Stream stream = Files.lines(Paths.get(storageName(topic, partition)));
                       return Long.parseLong(stream.collect(Collectors.toList()).get(0)) + 1;
               } catch (Exception e) {
                   e.printStackTrace();
               }
               return 0;
       }
       private String storageName(String topic, int partition) {
           return storagePrefix + "-" + topic + "-" + partition;
       }
}

4.使用指定实现完全一次

使用assign实现Exactly-once也很简单，具体思路如下：

1）将enable.auto.commit设置为假。

2）不调用consumer.commitSync（）。

3）调用指定注册卡夫卡消费者到卡夫卡

4）初次启动的时候，调用consumer.seek（topicPartition，偏移）来指定偏移量。

5）在处理消息的时候，要同时控制保存住每个消息的偏移量。以原子事务的方式保存偏移和处理的消息结果。传统数据库实现原子事务比较简单。但对于非传统数据库，比如HDFS或者nosql的，为了实现这个目标，只能将偏移与消息保存在同一行。

6）实现密等，作为保护层。

代码如下：

public class ExactlyOnceStaticConsumer {
       private static OffsetManager offsetManager = new OffsetManager("storage1");
       public static void main(String[] str) throws InterruptedException, IOException {
               System.out.println("Starting ExactlyOnceStaticConsumer ...");
               readMessages();
       }
       private static void readMessages() throws InterruptedException, IOException {
               KafkaConsumer consumer = createConsumer();
               String topic = "normal-topic";
               int partition = 1;
               TopicPartition topicPartition =
                               registerConsumerToSpecificPartition(consumer, topic, partition);
               // Read the offset for the topic and partition from external storage.
               long offset = offsetManager.readOffsetFromExternalStore(topic, partition);
               // Use seek and go to exact offset for that topic and partition.
               consumer.seek(topicPartition, offset);
               processRecords(consumer);
       }
       private static KafkaConsumer createConsumer() {
               Properties props = new Properties();
               props.put("bootstrap.servers", "localhost:9092");
               String consumeGroup = "cg2";
               props.put("group.id", consumeGroup);
               // Below is a key setting to turn off the auto commit.
               props.put("enable.auto.commit", "false");
               props.put("heartbeat.interval.ms", "2000");
               props.put("session.timeout.ms", "6001");
               // control maximum data on each poll, make sure this value is bigger than the maximum                 // single message size
               props.put("max.partition.fetch.bytes", "140");
               props.put("key.deserializer",                                     "org.apache.kafka.common.serialization.StringDeserializer");
               props.put("value.deserializer",                                     "org.apache.kafka.common.serialization.StringDeserializer");
               return new KafkaConsumer(props);
       }
       /**
           * Manually listens for specific topic partition. But, if you are looking for example of how to                * dynamically listens to partition and want to manually control offset then see
           * ExactlyOnceDynamicConsumer.java
           */
        private static TopicPartition registerConsumerToSpecificPartition(
                   KafkaConsumer consumer, String topic, int partition) {
                   TopicPartition topicPartition = new TopicPartition(topic, partition);
                   List partitions = Arrays.asList(topicPartition);
                   consumer.assign(partitions);
                   return topicPartition;
         }
           /**
               * Process data and store offset in external store. Best practice is to do these operations
               * atomically. 
               */
           private static void processRecords(KafkaConsumer consumer) throws {
                   while (true) {
                          ConsumerRecords records = consumer.poll(100);
                           for (ConsumerRecord record : records) {
                                   System.out.printf("offset = %d, key = %s, value = %s\n", record.offset(),                                                 record.key(), record.value());
                                   offsetManager.saveOffsetInExternalStore(record.topic(), record.partition(),                                                 record.offset());
                           }
                   }
           }
}

 

------------------------------------------------------------------------------------------------------------------------------------------------------------------------

      考虑一千次，不如去做一次；犹豫一万次，不如实践一次；华丽的跌倒，胜过无谓的彷徨，将来的你，一定会感谢现在奋斗的你。欢迎大家加入大数据交流群：725967421     一起交流，一起进步！！

------------------------------------------------------------------------------------------------------------------------------------------------------------------------