spark提交任务采用yarn集群提交方法
spark提交任务采用yarn集群提交方法
先启动 hadoop集群
再启动yarn
再启动spark集群
spark提交任务命令
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /home/hadoop3/app/spark/examples/jars/spark-examples_2.11-2.3.1.jar 10命令的执行没有错误，但因为使用的是yarn-cluster模式，结果不会输出到控制台（主要包括yarn-Client和yarn-Cluster两种模式，yarn-client模式将会在控制台中输出计算结果）。
注意到上述的：http://slave1:8088/proxy/application_1534472886935_0001/，在slave1主机上的浏览器中打开，
可以看到hadoop yarn集群的运行情况：
yarn集群的端口为8088
http://192.168.10.201:8088
查看地址:
http://slave1:8088/proxy/application_1534472886935_0001/
把slave1转为对应IP地址为：
http://192.168.10.201:8088/cluster/app/application_1534472886935_0001
主要内容为：
User: hadoop3
Name: org.apache.spark.examples.SparkPi
Application Type: SPARK
Application Tags:
YarnApplicationState: FINISHED
Queue: default
FinalStatus Reported by AM: SUCCEEDED
Started: Fri Aug 17 16:09:38 +0800 2018
Elapsed: 1mins, 32sec
Tracking URL: History
Diagnostics: 表示执行成功。
查看输出结果地址：LOG
http://192.168.10.201:8042/node/containerlogs/container_e23_1534472886935_0001_01_000001/hadoop3
日志页面内容如下所示：
Logs for container_e23_1534472886935_0001_01_000001
ResourceManager
RM Home
NodeManager
Tools
stderr : Total file length is 557 bytes.
stdout : Total file length is 19933 bytes.
具体提交执行情况如下所示：
[hadoop3@master app]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster /home/hadoop3/app/spark/examples/jars/spark-examples_2.11-2.3.1.jar 10
2018-08-17 16:08:56 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-08-17 16:09:08 INFO ConfiguredRMFailoverProxyProvider:100 - Failing over to rm2
2018-08-17 16:09:08 INFO Client:54 - Requesting a new application from cluster with 3 NodeManagers
2018-08-17 16:09:08 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-08-17 16:09:08 INFO Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2018-08-17 16:09:08 INFO Client:54 - Setting up container launch context for our AM
2018-08-17 16:09:08 INFO Client:54 - Setting up the launch environment for our AM container
2018-08-17 16:09:08 INFO Client:54 - Preparing resources for our AM container
2018-08-17 16:09:10 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-08-17 16:09:15 INFO Client:54 - Uploading resource file:/tmp/spark-7702c5f2-a1d0-4ceb-80c3-b800c6cbd75f/__spark_libs__360199046225972247.zip -> hdfs://mycluster/user/hadoop3/.sparkStaging/application_1534472886935_0001/__spark_libs__360199046225972247.zip
2018-08-17 16:09:36 INFO Client:54 - Uploading resource file:/home/hadoop3/app/spark/examples/jars/spark-examples_2.11-2.3.1.jar -> hdfs://mycluster/user/hadoop3/.sparkStaging/application_1534472886935_0001/spark-examples_2.11-2.3.1.jar
2018-08-17 16:09:37 INFO Client:54 - Uploading resource file:/tmp/spark-7702c5f2-a1d0-4ceb-80c3-b800c6cbd75f/__spark_conf__2307104984446206168.zip -> hdfs://mycluster/user/hadoop3/.sparkStaging/application_1534472886935_0001/__spark_conf__.zip
2018-08-17 16:09:38 INFO SecurityManager:54 - Changing view acls to: hadoop3
2018-08-17 16:09:38 INFO SecurityManager:54 - Changing modify acls to: hadoop3
2018-08-17 16:09:38 INFO SecurityManager:54 - Changing view acls groups to:
2018-08-17 16:09:38 INFO SecurityManager:54 - Changing modify acls groups to:
2018-08-17 16:09:38 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop3); groups with view permissions: Set(); users with modify permissions: Set(hadoop3); groups with modify permissions: Set()
2018-08-17 16:09:38 INFO Client:54 - Submitting application application_1534472886935_0001 to ResourceManager
2018-08-17 16:09:38 INFO YarnClientImpl:273 - Submitted application application_1534472886935_0001
2018-08-17 16:09:39 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:39 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1534493378198
final status: UNDEFINED
tracking URL: http://slave1:8088/proxy/application_1534472886935_0001/
user: hadoop3
2018-08-17 16:09:40 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:41 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:42 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:43 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:44 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:46 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:47 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:48 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:49 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:50 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:51 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:52 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:53 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:54 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:55 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:56 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:57 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:58 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:09:59 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:00 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:01 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:02 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:03 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:04 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:05 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:06 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:07 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:08 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:09 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:10 INFO Client:54 - Application report for application_1534472886935_0001 (state: ACCEPTED)
2018-08-17 16:10:11 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:11 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.10.201
ApplicationMaster RPC port: 0
queue: default
start time: 1534493378198
final status: UNDEFINED
tracking URL: http://slave1:8088/proxy/application_1534472886935_0001/
user: hadoop3
2018-08-17 16:10:12 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:13 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:14 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:15 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:16 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:17 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:18 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:19 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:20 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:21 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:22 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:23 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:24 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:25 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:26 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:27 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:28 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:29 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:30 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:31 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:32 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:33 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:34 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:35 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:36 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:37 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:38 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:39 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:40 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:41 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:42 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:43 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:44 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:45 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:46 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:47 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:48 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:49 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:50 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:51 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:52 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:53 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:54 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:55 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:56 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:57 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:58 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:10:59 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:00 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:01 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:03 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:04 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:05 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:06 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:07 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:08 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:09 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:10 INFO Client:54 - Application report for application_1534472886935_0001 (state: RUNNING)
2018-08-17 16:11:11 INFO Client:54 - Application report for application_1534472886935_0001 (state: FINISHED)
2018-08-17 16:11:11 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.10.201
ApplicationMaster RPC port: 0
queue: default
start time: 1534493378198
final status: SUCCEEDED
tracking URL: http://slave1:8088/proxy/application_1534472886935_0001/
user: hadoop3
2018-08-17 16:11:11 INFO ShutdownHookManager:54 - Shutdown hook called
2018-08-17 16:11:11 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-7702c5f2-a1d0-4ceb-80c3-b800c6cbd75f
2018-08-17 16:11:11 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-01737925-5237-45b1-a86a-66a660aa7be2
[hadoop3@master app]$