ERROR cluster.StandaloneSchedulerBackend: Application has been killed. Reason: Master removed app...

hadoop集群搭建完毕，spark-standalone集群搭建结束后，能正常启动

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://hadoop002:7077 \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10

这时候开始配置历史服务：

spark-shell停止后，集群监控hadoop002:4040就看不到历史任务的运行情况，所以开发时配置历史服务器记录任务运行情况。
修改【spark-defaults.conf.template】去掉template，并加上

spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://hadoop002:8020/directory #directory要保证事先存在于hdfs上

修改【spark-env.sh】文件，添加日志配置

export SPARK_HISTORY_OPTS="
-Dspark.history.ui.port=18080 
-Dspark.history.fs.logDirectory=hdfs://hadoop002:8020/directory 
-Dspark.history.retainedApplications=30" 

[root]分发配置：standlone目录下，xsync conf/
[childe-h]启动hadoop集群，并保证directory目录的存在

sbin/start-dfs.sh
#存在则不用创建。首次创建后，之后对dfs进行-format格式化才需要再次创建
hadoop fs -mkdir /directory 

[childe-h]启动集群：standlone目录下，
sbin/start-all.sh sbin/start-history-server.sh #启动历史服务器

启动报错

21/5/19 16:30:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/5/19 16:30:33 ERROR cluster.StandaloneSchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED
21/5/19 16:30:33 ERROR netty.Inbox: Ignoring error
org.apache.spark.SparkException: Exiting due to error from cluster scheduler: Master removed our application: FAILED
    at org.apache.spark.scheduler.TaskSchedulerImpl.error(TaskSchedulerImpl.scala:459)
    at org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend.dead(StandaloneSchedulerBackend.scala:139)
    at ...
19/11/15 16:28:31 ERROR spark.SparkContext: Error initializing SparkContext.
...

有个worker日志，内容如下：

java.io.IOException: Failed to create directory /soft/spark/work/app-...
    at org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.applyOrElse(Worker.scala:450)
    at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
    at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
...

worker无法创建日志，到目录下一看，worker是root所属文件，而其他是自己用户的所属文件。

两种解决方法。

一个是把这个文件改用户，改成要使用的用户

chown xxx.xxx worker -R

另一个是修改spark-env.sh文件并分发，我在这个文件中指定了work目录，如下

# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR      Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR       Where log files are stored.  (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR       Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING  A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS      The scheduling priority for daemons. (Default: 0)
# - SPARK_NO_DAEMONIZE  Run the proposed command in the foreground. It will not output a PID file.
 
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
#主节点的IP
export SPARK_MASTER_IP=b1
#主节点的端口号，用来与worker通信
export SPARK_MASTER_PORT=7077
#每一个worker进程所能管理的核数
export SPARK_WORKER_CORES=2
#每一个worker进程所能管理的内存数
export SPARK_WORKER_MEMORY=1G
#worker的工作目录区
export SPARK_WORKER_DIR=/usr/local/spark/spark-standlone