1、spark官网下载spark,可以可以在win系统中下载后上传至linux,也可以linux中直接下载: wget http://mirror.bit.edu.cn/apache/spark/spark-2.4.2/spark-2.4.2-bin-hadoop2.7.tgz
2、下载后解压至自己要安装的位置:tar -zxvf spark-2.4.2-bin-hadoop2.7.tgz,重命名为spark,
3、配置spark:
(1)、将conf下的 slaves.template复制为slaves,进入slaves,将localhost删除,新增子节点主机名,我的子节点为node-2、node-3。
(2)、将conf下的spark-env.sh.template复制为spark-env.sh,进入spark-env.sh,新增:
SPARK_MASTER_HOST=node-1
SPARK_MASTER_PORT=7077
4、将整个spark文件拷贝到另外两台子节点机器上:
scp -r spark root@node-2:/usr/centos/
scp -r spark root@node-3:/usr/centos/
5、主节点启动spark:./start-all.sh,查看启动是否成功:浏览器中输入:http://node-1:8080/
6、如果启动时报:
node-2: failed to launch: nice -n 0 /usr/centos/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://node-1:7077
node-2: JAVA_HOME is not set
node-2: full log in /usr/centos/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node-2.out
node-3: failed to launch: nice -n 0 /usr/centos/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://node-1:7077
node-3: JAVA_HOME is not set
则需要在node-2、node-3上的/root/.bashrc文件中配置export JAVA_HOME=/usr/centos/jdk1.8.0_201,然后再./stop-all.sh,再重启:./start-all.sh。
7、配置Job History Server
(1)、进入spark的config文件夹,将 spark-defaults.conf.template复制为 spark-defaults.conf,修改 spark-defaults.conf文件,开启log,将下面三个被注释的:
#spark.master spark://master:7077
#spark.eventLog.enabled true
#spark.eventLog.dir hdfs://namenode:8021/directory
改为:
spark.master spark://node-1:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://node-1:9000/directory
(2)、修改spark-env.sh,添加如下配置:
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://node-1:8021/directory"
(3)、将spark-defaults.conf、spark-env.sh复制到另外两台从节点上:
scp spark-defaults.conf node-2:/usr/centos/spark/conf/
scp spark-env.sh node-2:/usr/centos/spark/conf/
scp spark-defaults.conf node-3:/usr/centos/spark/conf/
scp spark-env.sh node-3:/usr/centos/spark/conf/
(4)、启动历史日志:./start-history-server.sh。
(5)、浏览器查看历史日志:http://node-1:4000/。
8、计算spark中求pi案例:
bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node-1:7077 /usr/centos/spark/examples/jars/spark-examples_2.12-2.4.2.jar 100