Spark+Scala安装
集群环境
| hostname | IP |
|---|---|
| master | 192.168.63.12 |
| slave1 | 192.168.63.13 |
| slave2 | 192.168.63.14 |
Scala安装
master
下载scala包
wget https://downloads.lightbend.com/scala/2.11.4/scala-2.11.4.tgz
解压至指定路径
tar -zxvf scala-2.11.4.tgz -C /usr/local/src
设置scala环境变量
vim ~/.bashrc
export SCALA_HOME=/usr/local/src/scala-2.11.4
export PATH=$PATH:$SCALA_HOME/bin
source ~/.bashrc使文件生效
验证
scala分发给其他节点(slave1、slave2)
scp -r /usr/local/src/scala-2.11.4 root@slave1:/usr/local/src
scp -r /usr/local/src/scala-2.11.4 root@slave2:/usr/local/src
master、slave1、slave2
修改~/.bashrc文件
export SCALA_HOME=/usr/local/src/scala-2.11.4
export PATH=$PATH:$SCALA_HOME/bin
source ~/.bashrc
Spark安装
下载spark源码包
wget https://archive.apache.org/dist/spark/spark-2.0.2/spark-2.0.2-bin-hadoop2.6.tgz
解压至指定路径
tar -zxvf spark-2.0.2-bin-hadoop2.6.tgz -C /usr/local/src/
切换到${SPARK_HOME}/conf
修改spark配置文件
vim spark-env.sh
export SCALA_HOME=/usr/local/src/scala-2.11.4
export JAVA_HOME=/usr/local/src/jdk1.8.0_181
export HADOOP_HOME=/usr/local/src/hadoop-2.6.1
export HADOOP_CONF_DIR=/usr/local/src/hadoop-2.6.1/etc/hadoop
SPARK_MASTER_IP=master
SPARK_LOCAL_DIRS=/usr/local/src/spark-2.0.2-bin-hadoop2.6
SPARK_DRIVER_MEMORY=1G
vim slaves
slave1
slave2
spark分发到slave1和slave2节点上
scp -r /usr/local/src/spark-2.0.2-bin-hadoop2.6 root@slave1:/usr/local/src
scp -r /usr/local/src/spark-2.0.2-bin-hadoop2.6 root@slave2:/usr/local/src
master、slave1、slave2
配置spark环境变量
vim ~/.bashrc
export SPARK_HOME=/usr/local/src/spark-2.0.2-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin
启动集群
cd /usr/local/src/spark-2.0.2-bin-hadoop2.6
./sbin/start-all.sh
查看jps进程
master
slave1
slave2
网页监控面板
master:8080
验证
本地模式
./bin/run-example SparkPi 10 --master local[2]
独立集群Standlone
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://master:7077 examples/jars/spark-examples_2.11-2.0.2.jar 10
集群
spark on yarn
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster examples/jars/spark-examples_2.11-2.0.2.jar 10
