Kerberos集群下SPARK任务的用户代理测试

隶属于文章系列：大数据安全实战 https://www.jianshu.com/p/76627fd8399c

---

• 在参数中传入testyarn-nopwd的Kerberos凭证，使用集群上有的用户提交

bin/spark-submit      \
 --principal [email protected]  \
--keytab /etc/hadoop/conf/testyarn-nopwd.keytab 
--proxy-user hive 
--class org.apache.spark.examples.SparkPi 
--master yarn --deploy-mode client  
./examples/jars/spark-examples_2.11-2.2.0.jar 
10

在界面上看到：testyarn-nopwd

image.png

• 使用当前终端已有的hdfs的凭证

bin/spark-submit      --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client  ./examples/jars/spark-examples_2.11-2.2.0.jar 10
#使用的当前已有的凭证：hdfs  在运行记录上看到也是hdfs用户的运行记录

image.png

• 使用当前登录的hdfs用户的凭证，代理hive用户

bin/spark-submit     --proxy-user hive --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client  ./examples/jars/spark-examples_2.11-2.2.0.jar 10
#在界面上看到：hive

image.png

• 没凭证时，代理集群上存在的hive用户

kdestroy
bin/spark-submit    --proxy-user hive     --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client  ./examples/jars/spark-examples_2.11-2.2.0.jar 10

18/04/24 14:17:52 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.20.178.28:4040
18/04/24 14:17:52 INFO spark.SparkContext: Added JAR file:/var/opt/spark-2.2.0-bin-hadoop2.7/./examples/jars/spark-examples_2.11-2.2.0.jar at spark://172.20.178.28:61994/jars/spark-examples_2.11-2.2.0.jar with timestamp 1524550672801
18/04/24 14:17:53 INFO client.RMProxy: Connecting to ResourceManager at v-hadoop-kbds.sz.kingdee.net/172.20.178.28:8032
18/04/24 14:17:53 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

就报错了,说明使用了代理机制的时候，也要有Kerberos凭证。

• 有凭证时，使用集群上不存在的用户代理：

bin/spark-submit    --proxy-user hellokitty --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client  ./examples/jars/spark-examples_2.11-2.2.0.jar 10

## 报错：
18/04/24 14:51:41 INFO yarn.Client:
         client token: N/A
         diagnostics: Application application_1524468264642_0007 failed 2 times due to AM Container for appattempt_1524468264642_0007_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://v-hadoop-kbds.sz.kingdee.net:8088/cluster/app/application_1524468264642_0007Then, click on links to logs of each attempt.
Diagnostics: Application application_1524468264642_0007 initialization failed (exitCode=255) with output: User hellokitty not found

• 总结
  使用代理机制提交spark任务的时候， 必须有有Kerberos凭证，proxy-user必须在集群上存在。

image.png