spark on yarn部署方案

1.在spark中spark-env.sh添加如下：
export HADOOP_CONF_DIR=/home/hadoop/hadoop/hadoop-2.7.6/etc/hadoop/

2.2. 拷贝 yarn-site.xml, hdfs-site.xml, core-site.xml 配置文件到$SPARK_HOME 下，重点是 yarn-site.xml，因为在搭建 spark ha 集群的时候，就已经把 core-site.xml 和 hdfs-site.xml 放置 在这个目录下了。

所以： $SPAR{K}_{H}ONE/conf目录下有hadoop的三个配置文件Core-site.xmlHdfs-site.xmlYarn-site.xml3.验证：spark-shell--masteryarn--executor-memory512m--total-executor-cores14.遇到报错：如下org.apache.spark.SparkException:Yarnapplicationhasalreadyended!Itmighthavebeenkilledorunabletolaunchapplicationmaster.5.在$HADOOP_HOME/etc/hadoop目录下
修改yarn-site.xml,添加如下

yarn.nodemanager.vmem-check-enabled
false
Whether virtual memory limits will be enforced forcontainers


yarn.nodemanager.vmem-pmem-ratio
4
Ratio between virtual memory to physical memory whensetting memory limits for containers

6.重新启动hadoop集群和spark集群
7.成功显示：

8.异常报错：
/home/hadoop/spark/spark-2.3.1-bin-hadoop2.7/bin/spark-shell: line 44: 6590 Killed

跟$SPARK_HONE/conf目录下 spark-env.sh 下的
export HADOOP_CONF_DIR=/home/hadoop/hadoop/hadoop-2.7.6/etc/hadoop/有关
9.使用spark-submit
~/spark/spark-2.3.1-bin-hadoop2.7/bin/spark-submit
–class org.apache.spark.examples.SparkPi
–master yarn
–deploy-mode client
–executor-memory 512m
–total-executor-cores 1
~/spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar
100
正确结果如下：