分析比较KafkaWordCount及DierctKafkaWordCount
val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).
需要注意的地方:
(1) Kafka中的Topic分区和Spark Streaming中RDDs的分区没有关系.因此,增加KafkaUtils.creatStream()中topic的分区仅仅只能增加使用这个从单一receiver进行consume的thread数量(only increases the number of threads using which topics that are consumed within a single receiver.),它并不能提高Spark在处理数据过程中的并行处理能力.
(2)能够为不同groups和topics创建多个Kafka输入DStream,这些DStream能够被多个receivers接收并行处理.
(3)如果使用Write Ahead Logs来备份数据,input stream的storage level应该被设置为StorageLevel.MEMORY_AND_DISK_SER
2,Approach 2:Direct Approache(No Receiers),没有Receiver直接接收数据
从Spark 1.3开始,引入了无receiver的direct方式,保证了强端对端关系.不同于使用receivers去接收数据,这种方法每隔一段时间就去查询Kafka最近生成的每一个主题+分区(topic+partition),并根据该查询结果,来处理每一个batch的数据处理范围.
当处理数据的Job运行起来,Kafka简单的consumer API被用来从Kafka的defined ranges中读取数据(类似于从文件系统读取数据).
相比于基于receiver的方式,无receiver方式具有如下优点:
(1)更容易实现并行处理.不需要创建更多的Kafka input streamings并union它们.使用directStream,Spark Streaming会创建与Kafka分区一样多的RDD分区进行consume,这些会并行从Kafka读取数据.因此Kafka分区与RDD分区之间有了一对一的关系.
(2)高效.在Approach1中,需要使用Write Ahead Log机制来保证数据0丢失,但是这样会导致数据重复保存.(分别在Kafka和Write Ahead Log中).Approach2不使用receiver,所以不需要Write Ahead Logs.丢失的数据可以在Kafka中进行恢复.
(3)Exactly-once.Approach1中采用典型的方法去consume Kafka数据,同时使用Write Ahead Logs去保证数据的0丢失,这样实现了at-least once.只有很少的可能会出现数据在failure后被consume两次.在Approach2中,使用简单Kafka API不使用Zookeeper,在Spark Streaming的checkpoints位置会跟踪记录数据的offsets
这种方法的一个不足之处是,不会更新Zookeeper中的offsets,因此基于Zookeeper的Kafka监控工具不会显示这个过程.然而你可以在每个batch中手动更新Zookeeper.
在编程中使用Approach2:
import org.apache.spark.streaming.kafka._
val directKafkaStream = KafkaUtils.createDirectStream[[key class],[value class],[key decoder class],[value decoder class]](streamingContext,[map of kafka parameters],[set of topics to consume])
在DirectKafkaWordCount中:
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
在Kafka的参数中,必须指定metadata.broker.list或者bootstrap.servers.默认清空下,会从Kafka分区最新的offset开始consuming.如果将Kafka的auto.offset.reset参数设置为smallest,会从最小的offset开始consuming
还可以通过使用KafkaUtils.createDirectStream的其他配置参数从任意的offset开始consuming.更进一步,如果想从Kafka的每一个batch接收Kafka offsets,按如下方式进行:
// Hold a reference to the current offset ranges, so it can be used downstream
var offsetRanges = Array[OffsetRange]()
directKafkaStream.transform { rdd =>
offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
rdd
}.map {
...
}.foreachRDD { rdd =>
for (o <- offsetRanges) {
println(s"${o.topic} ${o.partition} ${o.fromOffset} ${o.untilOffset}")
}
...
}
使用这种方法可以更新Zookeeper,从而可以是基于Zookeeper的Kafka监控工具显示streaming程序运行过程中的监控清空.
Note that the typecast to HasOffsetRanges will only succeed if it is done in the first method called on the directKafkaStream, not later down a chain of methods. You can use transform() instead of foreachRDD() as your first method call in order to access offsets, then call further Spark methods. However, be aware that the one-to-one mapping between RDD partition and Kafka partition does not remain after any methods that shuffle or repartition, e.g. reduceByKey() or window().
Another thing to note is that since this approach does not use Receivers, the standard receiver-related (that is, configurations of the formspark.streaming.receiver.* ) will not apply to the input DStreams created by this approach (will apply to other input DStreams though). Instead, use the configurations spark.streaming.kafka.*. An important one is spark.streaming.kafka.maxRatePerPartition which is the maximum rate (in messages per second) at which each Kafka partition will be read by this direct API.