spark2.3.1集群安装方法hadoop2.7.6和scala2.11.6
spark2.3.1集群安装方法hadoop2.7.6和scala2.11.6
下载安装包：spark-2.3.1-bin-hadoop2.7.tgz
http://spark.apache.org/downloads.html
下载安装包：scala-2.11.6.tgz
https://www.scala-lang.org/download/all.html
集群规划
节点名称 IP Zookeeper Master Worker
master Zookeeper 主Master Worker
slave1 Zookeeper 备Master Worker
slave2 Zookeeper Worker
安装scala
tar zxvf scala-2.11.6.tgz
mv scala
安装spark
tar zxvf spark-2.3.1-bin-hadoop2.7.tgz
mv spark-2.3.1 spark
进入spark/conf/ 目录
复制spark-env.sh.template并重命名为spark-env.sh
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
#=====================================
#指定默认master的ip或主机名
export SPARK_MASTER_HOST=master
#export SPARK_MASTER_IP=slave1
#指定maaster提交任务的默认端口为7077
export SPARK_MASTER_PORT=7077
#指定masster节点的webui端口
export SPARK_MASTER_WEBUI_PORT=18080
#每个worker从节点能够支配的内存数
export SPARK_WORKER_MEMORY=1g
#允许Spark应用程序在计算机上使用的核心总数（默认值：所有可用核心）
export SPARK_WORKER_CORES=1
#每个worker从节点的实例（可选配置）
export SPARK_WORKER_INSTANCES=1
#指向包含Hadoop集群的（客户端）配置文件的目录，运行在Yarn上配置此项
export HADOOP_CONF_DIR=/home/hadoop3/app/hadoop/etc/hadoop
#指定整个集群状态是通过zookeeper来维护的，包括集群恢复
export SPARK_DAEMON_JAVA_OPTS="
-Dspark.deploy.recoveryMode=ZOOKEEPER
-Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181
-Dspark.deploy.zookeeper.dir=/spark"
#=====================================slave1 /app/spark/conf/spark-env.sh
修改slave1 节点上conf/spark-env.sh配置的MasterIP为SPARK_MASTER_IP=slave1
#=====================================
#指定默认master的ip或主机名
#export SPARK_MASTER_HOST=master
export SPARK_MASTER_IP=slave1
#指定maaster提交任务的默认端口为7077
export SPARK_MASTER_PORT=7077
#指定masster节点的webui端口
export SPARK_MASTER_WEBUI_PORT=18080
#每个worker从节点能够支配的内存数
export SPARK_WORKER_MEMORY=1g
#允许Spark应用程序在计算机上使用的核心总数（默认值：所有可用核心）
export SPARK_WORKER_CORES=1
#每个worker从节点的实例（可选配置）
export SPARK_WORKER_INSTANCES=1
#指向包含Hadoop集群的（客户端）配置文件的目录，运行在Yarn上配置此项
export HADOOP_CONF_DIR=/home/hadoop3/app/hadoop/etc/hadoop
#指定整个集群状态是通过zookeeper来维护的，包括集群恢复
export SPARK_DAEMON_JAVA_OPTS="
-Dspark.deploy.recoveryMode=ZOOKEEPER
-Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181
-Dspark.deploy.zookeeper.dir=/spark"
#=====================================复制slaves.template成slaves
cp slaves.template slaves
vi slaves
并修改配置内容
slaves文件内容为：
# A Spark Worker will be started on each of the machines listed below.
master
slave1
slave2配置环境变量
所有节点均要配置
vi ~/.bashrc
export SCALA_HOME=/home/hadoop3/app/scala
export PATH=${SCALA_HOME}/bin:$PATH
export SPARK_HOME=/home/hadoop3/app/spark
export PATH=${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH~/.bashrc文件内容为：
[hadoop3@master conf]$ cat ~/.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
# add to path for /home/hadoop3/app/jdk and /home/hadoop3/tools
JAVA_HOME=/home/hadoop3/app/jdk
HADOOP_HOME=/home/hadoop3/app/hadoop
ZOOKEEPER_HOME=/home/hadoop3/app/zookeeper
HBASE_HOME=/home/hadoop3/app/hbase
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$HBASE_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:/home/hadoop3/tools:$PATH
export JAVA_HOME CLASSPATH HADOOP_HOME ZOOKEEPER_HOME HBASE_HOME PATH
export HBASE_CLASSPATH=/home/hadoop3/app/hbase/conf
export HBASE_MANAGES_ZK=false #不用自带的zookeeper
export SCALA_HOME=/home/hadoop3/app/scala
export PATH=${SCALA_HOME}/bin:$PATH
export SPARK_HOME=/home/hadoop3/app/spark
export PATH=${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH
启动集群
启动zookeeper集群
所有zookeeper节点均要执行
zkServer.sh start
runRemoveCmd.sh “zkServer.sh start” all
启动Hadoop集群
start-dfs.sh
start-yarn.sh
yarn-daemon.sh start resourcemanager
启动Spark集群
启动spark：
启动master节点：sbin/start-master.sh
启动worker节点：sbin/start-slaves.sh
或者：sbin/start-all.sh
备用master节点需要手动启动
slave1 sbin/start-master.sh
查看进程
[hadoop3@master sbin]$ runRemoteCmd.sh "jps" all
*******************master***************************
8800 Master
8593 Worker
7779 ResourceManager
8916 Jps
6311 NameNode
2360 QuorumPeerMain
7881 NodeManager
6589 JournalNode
6717 DFSZKFailoverController
6415 DataNode
*******************slave1***************************
8496 Jps
4289 DataNode
2339 QuorumPeerMain
7110 ResourceManager
4217 NameNode
4366 JournalNode
6926 NodeManager
7790 Worker
4431 DFSZKFailoverController
8047 Master
*******************slave2***************************
2673 QuorumPeerMain
4482 Jps
3443 DataNode
3940 NodeManager
4310 Worker
[hadoop3@master sbin]$
验证集群HA
看Web页面Master状态
slave1是ALIVE状态，master为STANDBY状态，WebUI查看：
http://192.168.10.200:18080/
http://192.168.10.201:18080/
Spark Master at spark://master:7077
URL: spark://master:7077
REST URL: spark://master:6066 (cluster mode)
Alive Workers: 0
Cores in use: 0 Total, 0 Used
Memory in use: 0.0 B Total, 0.0 B Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: STANDBYSpark Master at spark://slave1:7077
URL: spark://slave1:7077
REST URL: spark://slave1:6066 (cluster mode)
Alive Workers: 3
Cores in use: 3 Total, 0 Used
Memory in use: 3.0 GB Total, 0.0 B Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE从节点连接地址：
http://192.168.10.200:8081/
http://192.168.10.201:8081/
http://192.168.10.202:8081/
Spark Worker at 192.168.10.200:43208
ID: worker-20180810202818-192.168.10.200-43208
Master URL: spark://slave1:7077
Cores: 1 (0 Used)
Memory: 1024.0 MB (0.0 B Used)
Back to MasterSpark Worker at 192.168.10.201:35779
ID: worker-20180810202817-192.168.10.201-35779
Master URL: spark://slave1:7077
Cores: 1 (0 Used)
Memory: 1024.0 MB (0.0 B Used)
Back to MasterSpark Worker at 192.168.10.202:39623
ID: worker-20180810202817-192.168.10.202-39623
Master URL: spark://slave1:7077
Cores: 1 (0 Used)
Memory: 1024.0 MB (0.0 B Used)
Back to Master验证HA的高可用
手动干掉slave1上面的Master进程，slave1:8080无法访问，master:18080状态如下，Master状态成功自动进行切换。
主备切换过程中不能提交Application。
主备切换过程中不影响已经在集群中运行的Application。因为Spark是粗粒度资源调度。
—the–end—