Spark_Streaming 对接Kafka的好多坑

	本地	远程
Spark	1.5.2	2.4.3(不过现在还没提交到集群运行)
Kafka	0.8.2.1	0.8.

Spark_Streaming:
// 创建Receiver流的一个要点
// “zookeeper.connect” -> “spark1:2181,spark2:2181,spark3:2181,spark4:2181”,
val stream:ReceiverInputDStream[(String, String)] = KafkaUtils.createStream[String,String,
StringDecoder,StringDecoder](ssc,kafkaParams,
topicMap,StorageLevel.MEMORY_AND_DISK_SER)

附上0.8的消费者版本:

  class consuemr{
    var consumer:kafka.consumer.ConsumerConnector=_
    def test(): Unit ={
        import kafka.utils.VerifiableProperties
        val props = new Properties()
        // zookeeper 配置
        props.put("zookeeper.connect", "spark1:2181,spark2:2181")
        // group 代表一个消费组
        props.put("group.id", "jd-group")
        // zk连接超时
        props.put("zookeeper.session.timeout.ms", "4000")
        props.put("zookeeper.sync.time.ms", "200")
        props.put("auto.commit.interval.ms", "1000")
        props.put("auto.offset.reset", "smallest")
        // 序列化类
        props.put("serializer.class", "kafka.serializer.StringEncoder")
        val config = new ConsumerConfig(props)
        consumer = kafka.consumer.Consumer.create(config)
        val topicCountMap = new util.HashMap[String,Int]()
        topicCountMap.put("",1)
        val keyDecoder = new StringDecoder(new VerifiableProperties)
        val valueDecoder = new StringDecoder(new VerifiableProperties)
        val consumerMap = consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder)
        val stream:KafkaStream[String,String] = consumerMap("yy-topic").head
        val it = stream.iterator()
        while(it.hasNext()){
            println(it.next().message())
        }
    }
}

============ 可能问题:
缺少:LKafka/…/Connector:
Spark-Streaming-Kafka 版本不对
NotClassFound:…/Logging
Spark 1.5 以上缺少Logging模块