spark on yarn提交任务时一直显示ACCEPTED,过一个小时后就会出现任务失败,但在提交时shell终端显示的日志并没有报错,logs文件夹中也没有日志产生。注:spark on yarn是不需要启动spark集群的,只需要在提交任务的机器配置spark就可以了,因为任务是由hadoop来执行的,spark只负责任务的提交。
任务提交命令为
bin/spark-submit --class org.apache.spark.examples.JavaWordCount\
--master yarn-client \
--num-executors 25 \
--executor-memory 1g \
--executor-cores 1 \
lib/spark-examples-1.0.1-hadoop2.2.0.jar \
hdfs://192.168.1.11:9000/test
spark-env.sh配置如下:
export SPARK_JAR=hdfs://192.168.1.11:9000/test1/spark-assembly-1.0.1-hadoop2.2.0.jar
export SPARK_YARN_APP_JAR=hdfs://192.168.1.11:9000/test1/spark-examples-1.0.1-hadoop2.2.0.jar
export HADOOP_CONF_DIR=/hadoop/etc/hadoop
export SPARK_EXECUTOR_INSTANCES=1000
export SPARK_EXECUTOR_CORES=1
export SPARK_EXECUTOR_MEMORY=10000M
export SPARK_DRIVER_MEMORY=20000M
export SPARK_YARN_APP_NAME=spark
查看日志发现报错:
14/09/04 17:10:44 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/09/04 17:10:45 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/09/04 17:10:46 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/09/04 17:10:47 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
这个问题很明显没有找到resourcemanager,我们可以通过以下方法来解决:
1 在spark-env.sh中添加
export SPARK_YARN_USER_ENV="CLASSPATH=hadoop-2.3.0/etc/hadoop"(修改成自己的hadoop路径)
2 自己编写JavaWordCount程序,在程序中配置集群信息
SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
sparkConf.set("mapreduce.framework.name", "yarn");
sparkConf.set("mapreduce.jobtracker.address", "192.168.1.10:9001");
sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");
sparkConf.set("yarn.resourcemanager.admin.address", "192.168.1.10:8033");
sparkConf.set("yarn.resourcemanager.address", "192.168.1.10:8032");
sparkConf.set("yarn.resourcemanager.resource-tracker.address", "192.168.1.10:8031");
sparkConf.set("yarn.resourcemanager.scheduler.address", "192.168.1.10:8030");
sparkConf.set("yarn.resourcemanager.hostname", "192.168.1.10");
然后打包,在提交spark任务时使用自己打包的jar包即可。