Spark安装（Yarn模式）

一、解压

链接：https://pan.baidu.com/s/1O8u1SEuLOQv2Yietea_Uxg 
提取码：mb4h

tar -zxvf /opt/software/spark-3.0.3-bin-hadoop3.2.tgz -C /opt/module/spark-yarn

mv spark-3.0.3-bin-hadoop3.2/ spark-yarn

二、配置环境变量

vim /etc/profile

#SPARK_HOME
export SPARK_HOME=/opt/module/spark-yarn
export PATH=$PATH:$SPARK_HOME/bin

使变量生效

source /etc/profile

三、配置文件

vim /opt/module/hadoop-3.1.3/etc/hadoop/yarn-site.xml

     yarn.nodemanager.pmem-check-enabled
     false

 

     yarn.nodemanager.vmem-check-enabled
     false

四、 分发刚刚修改的配置文件

scp /opt/module/hadoop-3.1.3/etc/hadoop/yarn-site.xml bigdata-slave1:/opt/module/hadoop-3.1.3/etc/hadoop/

scp /opt/module/hadoop-3.1.3/etc/hadoop/yarn-site.xml bigdata-slave2:/opt/module/hadoop-3.1.3/etc/hadoop/

五、修改spark-env.sh
 

cd /opt/module/spark-yarn/conf/

[root@bigdata-master conf]# mv spark-env.sh.template spark-env.sh
[root@bigdata-master conf]# vim spark-env.sh
YARN_CONF_DIR=/opt/module/hadoop-3.1.3/etc/hadoop

六、启动hadoop

start-all.sh

七、求PI

spark-submit --class org.apache.spark.examples.SparkPi --master yarn ./examples/jars/spark-examples_2.12-3.0.3.jar  10