spark2-release-HDP-2.6.4.0-91-tag在CentOS7上的源码编译

Spark2编译准备

下载：

• https://github.com/hortonworks/spark2-release/releases/tag/HDP-2.6.4.0-91-tag
• https://github.com/hortonworks/spark2-release/archive/HDP-2.6.4.0-91-tag.tar.gz

[root@gaia tarpackage]# wget https://github.com/hortonworks/spark2-release/archive/HDP-2.6.4.0-91-tag.tar.gz
[root@gaia tarpackage]# tar -zxvf HDP-2.6.4.0-91-tag.tar.gz -C /opt/
[root@gaia tarpackage]# cd /opt/spark2-release-HDP-2.6.4.0-91-tag/

maven环境搭建

参考博客：https://blog.csdn.net/Happy_Sunshine_Boy/article/details/91375219

• maven 设置

[root@gaia spark2-release-HDP-2.6.4.0-91-tag]# vim /etc/profile
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
[root@gaia spark2-release-HDP-2.6.4.0-91-tag]# source /etc/profile

Scala环境搭建

参考博客：https://blog.csdn.net/Happy_Sunshine_Boy/article/details/92806387

Spark2编译

设置 make-distribution.sh

[root@gaia spark2-release-HDP-2.6.4.0-91-tag]# vim ./dev/make-distribution.sh
VERSION=2.2.0
SCALA_VERSION=2.11.8
SPARK_HADOOP_VERSION=2.7.3-hdp2.6.4
SPARK_HIVE=1.2

执行编译命令

[root@gaia spark2-release-HDP-2.6.4.0-91-tag]# ./dev/make-distribution.sh --name 2.2.0-hdp2.6.4 --tgz -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pyarn

• 参考官网：http://spark.apache.org/docs/latest/building-spark.html