Kafka有四个核心的API:
main线程通过拦截器(interceptors)、序列化器(serializer)和分区器(partitioner)之后,将消息发送给共享变量RecordAccumulator,Sender线程从RecordAccumulator中拉取消息到kafka broker。以下是涉及到的参数:
创建一个3副本5分区的名称为consumer的topic(下面语句中采用创建再修改分区的方式做的)。
bin/kafka-topics.sh --zookeeper hadoop100:2181/kafka --create --replication-factor 3 --partitions 1 --topic consumer
bin/kafka-topics.sh --zookeeper hadoop100:2181/kafka --alter --topic consumer --partitions 5
maven导包
<dependency>
<groupId>org.apache.kafkagroupId>
<artifactId>kafka-clientsartifactId>
<version>2.4.1version>
dependency>
直接贴代码吧,代码逻辑也比较简单。
package www.whuhhh.cn;
import org.apache.kafka.clients.producer.*;
import scala.Int;
import java.util.Properties;
public class CustomProducer {
public static void main(String[] args){
Properties props = new Properties();
props.put("bootstrap.servers", "hadoop100:9092");//连接的集群,broker-list。9092是其端口号
props.put("acks", "all");
props.put("retries", 1);//重试次数
props.put("batch.size", 16384);//批次大小
props.put("linger.ms", 1);
props.put("buffer.memory", 33553332);//缓冲物大小
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<String, String>(props);
for(int i = 0; i < 20; i++){
producer.send(new ProducerRecord<String, String>("consumer", Integer.toString(i), "test-" + Integer.toString(i)), new Callback() {
@Override
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
if(e == null){
System.out.println("SUCCESS -> " + recordMetadata.offset() + ":" + recordMetadata.partition());
}else{
e.printStackTrace();
}
}
});
}
producer.close();
}
}
开启消费者:
bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --from-beginning --topic consumer
在IDEA的打印结果如图:
在消费者端打印出了全部的数据,这里就不贴图了。可以看到的是打印出了offset和partition分区。
同步发送的意思就是,一条消息发送之后,会阻塞当前线程,直至返回ack。由于 send方法返回的是一个 Future对象,根据 Futrue对象的特点,我们也可以实现同 步发送的效果,只需在调用 Future对象的 get方发即可。同步发送并不常用
package www.whuhhh.cn;
import org.apache.kafka.clients.producer.*;
import scala.Int;
import java.util.Properties;
public class CustomProducer {
public static void main(String[] args)throws ExecutionException, InterruptedException{
Properties props = new Properties();
props.put("bootstrap.servers", "hadoop100:9092");//连接的集群,broker-list。9092是其端口号
props.put("acks", "all");
props.put("retries", 1);//重试次数
props.put("batch.size", 16384);//批次大小
props.put("linger.ms", 1);
props.put("buffer.memory", 33553332);//缓冲物大小
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<String, String>(props);
for(int i = 0; i < 20; i++){
producer.send(new ProducerRecord<String, String>("consumer", Integer.toString(i), "test-" + Integer.toString(i)), new Callback() {
@Override
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
if(e == null){
System.out.println("SUCCESS -> " + recordMetadata.offset() + ":" + recordMetadata.partition());
}else{
e.printStackTrace();
}
}
}).get();
}
producer.close();
}
}
1)导入依赖
<dependency>
<groupId>org.apache.kafkagroupId>
<artifactId>kafka-clientsartifactId>
<version>2.4.1version>
dependency>
2)编写代码
需要用到的类:
为了使我们能够专注于自己的业务逻辑,Kafka提供了自动提交offset的功能。
自动提交offset的相关参数:
enable.auto.commit:是否开启自动提交offset功能
auto.commit.interval.ms:自动提交offset的时间间隔
以下为自动提交offset的代码:
package com.atguigu.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.util.Arrays;
import java.util.Properties;
public class CustomConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "hadoop102:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("first"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
}
}
虽然自动提交offset十分简介便利,但由于其是基于时间提交的,开发人员难以把握offset提交的时机。因此Kafka还提供了手动提交offset的API。
手动提交offset的方法有两种:分别是commitSync(同步提交)和commitAsync(异步提交)。两者的相同点是,都会将本次poll的一批数据最高的偏移量提交;不同点是,commitSync阻塞当前线程,一直到提交成功,并且会自动失败重试(由不可控因素导致,也会出现提交失败);而commitAsync则没有失败重试机制,故有可能提交失败。
1)同步提交offset
由于同步提交offset有失败重试机制,故更加可靠,以下为同步提交offset的示例。
package com.atguigu.kafka.consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.util.Arrays;
import java.util.Properties;
public class CustomComsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "hadoop102:9092");//Kafka集群
props.put("group.id", "test");//消费者组,只要group.id相同,就属于同一个消费者组
props.put("enable.auto.commit", "false");//关闭自动提交offset
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("first"));//消费者订阅主题
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);//消费者拉取数据
for (ConsumerRecord<String, String> record : records) {
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
consumer.commitSync();//同步提交,当前线程会阻塞知道offset提交成功
}
}
}
2)异步提交offset
虽然同步提交offset更可靠一些,但是由于其会阻塞当前线程,直到提交成功。因此吞吐量会收到很大的影响。因此更多的情况下,会选用异步提交offset的方式。
以下为异步提交offset的示例:
package com.atguigu.kafka.consumer;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;
import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
public class CustomConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "hadoop102:9092");//Kafka集群
props.put("group.id", "test");//消费者组,只要group.id相同,就属于同一个消费者组
props.put("enable.auto.commit", "false");//关闭自动提交offset
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("first"));//消费者订阅主题
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);//消费者拉取数据
for (ConsumerRecord<String, String> record : records) {
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
consumer.commitAsync(new OffsetCommitCallback() {
@Override
public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
if (exception != null) {
System.err.println("Commit failed for" + offsets);
}
}
});//异步提交
}
}
}
3)数据漏消费和重复消费分析
无论是同步提交还是异步提交offset,都有可能会造成数据的漏消费或者重复消费。先提交offset后消费,有可能造成数据的漏消费;而先消费后提交offset,有可能会造成数据的重复消费。
借用羊哥的一句话,走的慢才能走的更远。