Spark连接需Kerberos认证的HBase
Prerequisite
- krb5.conf 或 krb5.ini
- xx.keytab
- core-site.xml
- hbase-core.xml
Codes
hBaseConfig.addResource("hbase-site.xml")
hBaseConfig.addResource("core-site.xml")
UserGroupInformation.setConfiguration(hBaseConfig);
UserGroupInformation ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("[email protected]", keyTab);
UserGroupInformation.setLoginUser(ugi);
ugi.doAs(new PrivilegedAction() {
public Void run() {
connection = ConnectionFactory.createConnection(conf);
//hbase operation
return null;
}
}); Submit cmd
spark-submit --class com.myclass \
--master yarn \
--deploy-mode cluster \
--num-executors 2 \
--driver-memory 1G \
--executor-memory 1G \
--executor-cores 2 \
--queue myqueue \
--conf spark.app.name=MyAPP \
--conf spark.streaming.kafka.maxRatePerPartition=100 \
--conf spark.streaming.stopGracefullyOnShutDown=true \
--conf spark.yarn.security.credentials.hbase.enable=true \
--conf spark.driver.extraJavaOptions='-Dlog4j.configuration=log4j -Djava.security.krb5.conf=krb5.conf -Dkeytab.file=mykeytab.keytab' \
--conf spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j -Djava.security.krb5.conf=krb5.conf -Dkeytab.file=mykeytab.keytab' \
--conf spark.yarn.dist.archives=/path/core-site.xml,/path/hbase-site.xml,/path/krb5.conf,/path/mykeytab.keytab,/path/log4j.properties / \
/path/myproject-1.0-SNAPSHOT.jarTIPS
1. spark yarn 模式下提交时可用--files, --conf spark.yarn.dist.archives, --conf spark.yarn.dist.files 后接逗号分隔符的文件全路径将文件提交到每个运行节点.
2. spark yarn 模式下测试情况来看只能用ugi.doAs形式访问HBase,否则会报错认证不通过.
3. 如果使用submit命令中使用了--keytab和--principal,--conf spark.yarn.dist.archives提交的keytab文件会找不到. 详情参照spark官方文档security和yarnReferences
Spark can’t connect to HBase using Kerberos in Cluster mode
HBase中的TTL应用
Spark读写Hbase的二种方式对比
Spark Streaming接收kafka数据，输出到HBase
Spark Streaming with HBase
SparkStreamingHBaseExample
Spark读写Hbase的二种方式对比
kafka-spark-streaming-to-hbase
New in Cloudera Labs: SparkOnHBase
Spark官档Security
Spark官档Yarn
Spark官档Configuration