【Spark】Spark Streaming 指定消费Topic中某个Partition的数据

最近接触到一个需求：通过Driect API消费Kafka topic中指定partiton中的数据。这个partition中的数据是用户通过某种算法过滤出的有效数据，其他partition数据不作处理，代码片段实现如下：

val kafkaParams = Map[String, String]("metadata.broker.list" -> "ip:port", "auto.offset.reset" -> "largest")

val kc = new KafkaCluster(kafkaParams)
val arrTopicPartition = Array(
  TopicAndPartition("topic1", 1),
  TopicAndPartition("topic2", 0))

val arrLastOffsets  = for (tp <- arrTopicPartition)
  yield kc.getLatestLeaderOffsets(Set(tp)).right.get(tp).offset

val messages = KafkaUtils.createDirectStream[
  String, String, StringDecoder, StringDecoder,(String, Int, String)](
  ssc,
  kafkaParams,
  Map(arrTopicPartition(0) -> arrLastOffsets(0),
    arrTopicPartition(1) -> arrLastOffsets(1)),
  (m: MessageAndMetadata[String, String]) => (m.topic, m.partition, m.message()))

var offsetRanges = Array.empty[OffsetRange]
messages.transform { rdd =>
  offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
  rdd
}.foreachRDD { rdd =>
  for (o <- offsetRanges) {
    println(s"${o.topic} ${o.partition} ${o.fromOffset} ${o.untilOffset} ${rdd.collect().foreach(r => print(r + "##"))}")
  }
}

实现消费topic1中partiton为1和topic2中partition为0 的数据。