spark2.2 批量往es7.1.1中插入数据的时候问题

报的错为:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.elasticsearch.spark.deploy.yarn.security.EsServiceCredentialProvider.credentialsRequired(Lorg/apache/spark/SparkConf;Lorg/apache/hadoop/conf/Configuration;)Z java.lang.AbstractMethodError: org.elasticsearch.spark.deploy.yarn.security.EsServiceCredentialProvider.credentialsRequired(Lorg/apache/spark/SparkConf;Lorg/apache/hadoop/conf/Configuration;)Z at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:81) at org.apache.spark.deploy.yarn.security.ConfigurableCredentialManager$$anonfun$obtainCredentials$2.apply(ConfigurableCredentialManager.scala:80)

因为cdh5.13以后 会默认给spark2开启Kerberos模式,es7.1.1跟spark整合的包默认会以Kerberos的形式访问spark2,所以会报这个错误,解决方法:

在spark中设置spark.yarn.security.credentials.elasticsearch.enabled为false即可

 

另外访问hdfs hive等都会默认以这个形式,解决方法可参考官网

https://www.elastic.co/guide/en/elasticsearch/hadoop/current/kerberos.html

你可能感兴趣的:(es,Spark,sparksql)