[Spark]spark3.0.0 on yarn 安装测试

spark3.0.0  on yarn 安装测试

版本：

hive 3.1.1 

spark 3.0.0 

hadoop3.0.0

从spark3.0.0版本开始支持hadoop3。hive3 也支持hadoop，完美的组合。

 

3.1.下载spark3.0.0

https://mirror.bit.edu.cn/apache/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz

版本支持：scala2.12版本(其实不装scala也可以运行，但是低版本应该不行)，java8+ 

 

3.2.spark3配置

 

3.2.1.spark-env.sh 配置

export SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://xx/user/spark/applicationHistory_xx0001" export HADOOP_HOME=~/hadoop export SPARK_HOME=/home/xx/spark-3.0.0-bin-hadoop3.2 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native export SPARK_DIST_CLASSPATH=$(hadoop classpath) export SPARK_CLASSPATH=$SPARK_CLASSPATH:$HADOOP_HOME/share/hadoop/yarn/:$HADOOP_HOME/share/hadoop/yarn/lib/:$HADOOP_HOME/share/hadoop/common/:$HADOOP_HOME/share/hadoop/common/lib/:$HADOOP_HOME/share/hadoop/hdfs/:$HADOOP_HOME/share/hadoop/hdfs/lib/:$HADOOP_HOME/share/hadoop/mapreduce/:$HADOOP_HOME/share/hadoop/mapreduce/lib/:$HADOOP_HOME/share/hadoop/tools/lib/:$SPARK_HOME/lib export SPARK_LOCAL_IP=172.x.x.2

3.2.2.spark-default.conf

spark.eventLog.enabled           true spark.yarn.historyServer.address 172.23.4.152:12345 spark.eventLog.compress true spark.history.ui.port 12345 spark.eventLog.dir hdfs://xx/user/spark/applicationHistory_szhadoop310001 spark.history.fs.logDirectory hdfs://xx/user/spark/applicationHistory_szhadoop310001 spark.port.min  30000 spark.port.max  40000 spark.executor.extraLibraryPath /home/hadoop/cdh5/hadoop/latest/lib/native spark.shuffle.consolidateFiles true spark.executor.extraJavaOptions -Xss24m spark.driver.extraJavaOptions -Xss24m spark.yarn.submit.file.replication 3 spark.sql.parquet.compression.codec     gzip spark.history.fs.cleaner.enabled true spark.history.fs.cleaner.interval 1d spark.history.fs.cleaner.maxAge 3d

3.2.3. 相关配置软连接到spark/conf目录

core-site.xml 

hdfs-site.xml 

hive-site.xml 

mapred-site.xml 

yarn-site.xml 

hadoop和hive的5个配置文件。

如果想查看hive的表，记住要拷贝hive-site.xml

 

3.3测试

spark-shell --master yarn  --deploy-mode client --queue root --num-executors 10 --driver-memory 3g --executor-memory 4g --executor-cores 3 --conf spark.speculation=true --conf spark.speculation.quantile=0.75 --conf spark.speculation.multiplier=1.5 --conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true --jars /home/xx/module/hive/lib/mysql-connector-java-5.1.38.jar

scala> spark.sql("show tables").show(false)

+--------+---------+-----------+

|database|tableName|isTemporary|

+--------+---------+-----------+

|default |student  |false      |

+--------+---------+-----------+

 

 

scala> import spark.sql

import spark.sql

 

scala> sql("select * from student").show(false)

+---+----+                                                                      

|id |name|

+---+----+

|1  |tony|

|1  |tony|

+---+----+

 

测试通过，接下来做业务方面任务的验证。