分布式消息队列kafka系列介绍 — 核心API介绍及实例

原文地址:http://www.inter12.org/archives/834

 

一 PRODUCER的API

 

1.Producer的创建,依赖于ProducerConfig
public Producer(ProducerConfig config);

2.单个或是批量的消息发送
public void send(KeyedMessage message);
public void send(List> messages);

3.关闭Producer到所有broker的连接
public void close();

 

二 CONSUMER的高层API

主要是Consumer和ConsumerConnector,这里的Consumer是ConsumerConnector的静态工厂类
class Consumer {
public static kafka.javaapi.consumer.ConsumerConnector createJavaConsumerConnector(config: ConsumerConfig);
}

具体的消息的消费都是在ConsumerConnector中
创建一个消息处理的流,包含所有的topic,并根据指定的Decoder
public Map>>
createMessageStreams(Map topicCountMap, Decoder keyDecoder, Decoder valueDecoder);

创建一个消息处理的流,包含所有的topic,使用默认的Decoder
public Map>> createMessageStreams(Map topicCountMap);

获取指定消息的topic,并根据指定的Decoder
public List>
createMessageStreamsByFilter(TopicFilter topicFilter, int numStreams, Decoder keyDecoder, Decoder valueDecoder);

获取指定消息的topic,使用默认的Decoder
public List> createMessageStreamsByFilter(TopicFilter topicFilter);

提交偏移量到这个消费者连接的topic
public void commitOffsets();

关闭消费者
public void shutdown();

高层的API中比较常用的就是public List> createMessageStreamsByFilter(TopicFilter topicFilter);和public void commitOffsets();

 

三 CONSUMER的简单API–SIMPLECONSUMER

批量获取消息
public FetchResponse fetch(request: kafka.javaapi.FetchRequest);

获取topic的元信息
public kafka.javaapi.TopicMetadataResponse send(request: kafka.javaapi.TopicMetadataRequest);

获取目前可用的偏移量
public kafka.javaapi.OffsetResponse getOffsetsBefore(request: OffsetRequest);

关闭连接
public void close();

对于大部分应用来说,高层API就已经足够使用了,但是若是想做更进一步的控制的话,可以使用简单的API,例如消费者重启的情况下,希望得到最新的offset,就该使用SimpleConsumer.

 

四 KAFKA HADOOP CONSUMER API

提供了一个可水平伸缩的解决方案来结合hadoop的使用参见

https://github.com/linkedin/camus/tree/camus-kafka-0.8/

 

五 实战

maven依赖:

org.apache.kafka
kafka_2.10
0.8.0


生产者代码:

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
 
import java.util.Properties;
 
/**
 * 
 * Created by zhaoming on 14-5-4 下午3:23
 * 
*/ public class KafkaProductor { public static void main(String[] args) throws InterruptedException { Properties properties = new Properties(); properties.put("zk.connect", "127.0.0.1:2181"); properties.put("metadata.broker.list", "localhost:9092"); properties.put("serializer.class", "kafka.serializer.StringEncoder"); ProducerConfig producerConfig = new ProducerConfig(properties); Producer producer = new Producer(producerConfig); // 构建消息体 KeyedMessage keyedMessage = new KeyedMessage("test-topic", "test-message"); producer.send(keyedMessage); Thread.sleep(1000); producer.close(); } } 消费端代码 import java.io.UnsupportedEncodingException; import java.util.List; import java.util.Properties; import java.util.concurrent.TimeUnit; import kafka.consumer.*; import kafka.javaapi.consumer.ConsumerConnector; import kafka.message.MessageAndMetadata; import org.apache.commons.collections.CollectionUtils; /** *
 * Created by zhaoming on 14-5-4 下午3:32
 * 
*/ public class kafkaConsumer { public static void main(String[] args) throws InterruptedException, UnsupportedEncodingException { Properties properties = new Properties(); properties.put("zookeeper.connect", "127.0.0.1:2181"); properties.put("auto.commit.enable", "true"); properties.put("auto.commit.interval.ms", "60000"); properties.put("group.id", "test-group"); ConsumerConfig consumerConfig = new ConsumerConfig(properties); ConsumerConnector javaConsumerConnector = Consumer.createJavaConsumerConnector(consumerConfig); //topic的过滤器 Whitelist whitelist = new Whitelist("test-topic"); List> partitions = javaConsumerConnector.createMessageStreamsByFilter(whitelist); if (CollectionUtils.isEmpty(partitions)) { System.out.println("empty!"); TimeUnit.SECONDS.sleep(1); } //消费消息 for (KafkaStream partition : partitions) { ConsumerIterator iterator = partition.iterator(); while (iterator.hasNext()) { MessageAndMetadata next = iterator.next(); System.out.println("partiton:" + next.partition()); System.out.println("offset:" + next.offset()); System.out.println("message:" + new String(next.message(), "utf-8")); } } } }

  



 

你可能感兴趣的:(分布式消息队列kafka系列介绍 — 核心API介绍及实例)