spark standalone work扩展

所有节点配置Java环境以及下载spark安装包

所有节点配置hosts文件

192.168.2.28 master
192.168.2.29 node1
192.168.2.30 node2

1. 创建spark用户

[root@master ~]# useradd spark
[root@node1 ~]# useradd spark

2. 配置免秘钥登录

• 具体操作查看此处: http://blog.csdn.net/zxf_668899/article/details/53726226

测试：

[spark@master ~]$ ssh master    登录自己
Last login: Thu Feb  9 15:12:08 2017 from master
[spark@master ~]$ exit
logout
Connection to master closed.
[spark@master ~]$ ssh node1     登录node1
Last login: Thu Feb  9 14:55:46 2017
[spark@node1 ~]$ exit
logout
Connection to node1 closed.

3. 目录权限设置

master:
[spark@master ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:53 /opt/source/spark-2.0.2-bin-hadoop2.7

node1:
[spark@node1 ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:54 /opt/source/spark-2.0.2-bin-hadoop2.7

4. spark 配置(spark用户操作)

master节点操作：

[spark@master ~]$ cd /opt/spark/
[spark@master spark]$ cp conf/spark-env.sh.template conf/spark-env.sh
[spark@master spark]$ cp conf/slaves.template conf/slaves
[spark@master spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*
[spark@master spark]$ tail -n 2 /opt/spark/conf/slaves
master
node1

node1节点操作：

[spark@node1 spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*

启动spark

[spark@master spark]$ ./sbin/start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.master.Master-1-master.out
node1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-node1.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-master.out

扩展添加 work节点

添加node2 work 节点

• 配置Java环境、hosts文件、spark安装包、创建用户、免密登录(略)

[spark@node2 ~]$ ll -d /opt/source/spark-2.0.2-bin-hadoop2.7
drwxr-xr-x 14 spark spark 4096 Feb  9 10:54 /opt/source/spark-2.0.2-bin-hadoop2.7

[spark@node2 spark]$ vim conf/spark-env.sh
......
export JAVA_HOME=/opt/jdk
export SPARK_WORKER_CORES=1
export SPARK_WORKER_DIR=/home/spark/work
export SPARK_DAEMON_MEMORY=1G
export SHARK_MASTER_MEM=1G
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/lib/*:/opt/spark/lib/*

[spark@node2 spark]$ tail -n 2 conf/slaves
# A Spark Worker will be started on each of the machines listed below.
node2

[spark@node2 spark]$ ./sbin/start-slave.sh spark://master:7077
starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-node2.out