CDH集成Kafka时配置的版本问题

问题

当提交使用了Kafka的Spark应用到CDH集群时，抛出异常如下

2020-01-27 11:33:48,289 ERROR [TestApp] - java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V
2020-01-27 11:33:48,291 ERROR [TestApp] - 	at com.aspire.cmbsms.LogCollector$.getEarliestOffsets(LogCollector.scala:491)
2020-01-27 11:33:48,291 ERROR [TestApp] - 	at com.aspire.cmbsms.LogCollector$.getStartingOffsetsToConsume(LogCollector.scala:432)
2020-01-27 11:33:48,291 ERROR [TestApp] - 	at com.aspire.cmbsms.LogCollector$.collectImpl(LogCollector.scala:84)
2020-01-27 11:33:48,291 ERROR [TestApp] - 	at com.aspire.cmbsms.LogCollector$.main(LogCollector.scala:45)
2020-01-27 11:33:48,291 ERROR [TestApp] - 	at com.aspire.cmbsms.LogCollector.main(LogCollector.scala)
2020-01-27 11:33:48,292 ERROR [TestApp] - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-01-27 11:33:48,292 ERROR [TestApp] - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-01-27 11:33:48,292 ERROR [TestApp] - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-01-27 11:33:48,292 ERROR [TestApp] - 	at java.lang.reflect.Method.invoke(Method.java:498)
2020-01-27 11:33:48,292 ERROR [TestApp] - 	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)

原因

CDH Spark2中使用的Kafka版本与Spark应用pom.xml中使用的不同。
pom.xml中使用了spark-streaming-kafka-0-10_2.11
而默认的CDH Spark2配置中使用的是 spark-streaming-kafka-0-9_2.11

解决方法

CDH集成了Spark2，将spark_kafka_version设置为0.10