利用CDH进行spark集群部署
1. 规划：
master: brain01
worker: brain02、brain03、brain04
2. 准备工作：
2.1 安装Java 1.7
2.2 设置主机名，编辑/etc/hosts
2.3 关闭iptables:
service iptables stop
chkconfig iptables off
2.4 关闭selinux,修改文件/etc/selinux/config，然后重启操作系统
SELINUX=disabled
3.部署
3.1 brain01~brain04:
yum install spark-core spark-master spark-worker spark-history-server spark-python -y
3.2 brain01上，修改/etc/spark/conf/spark-env.sh
vi /etc/spark/conf/spark-env.sh
export STANDALONE_SPARK_MASTER_HOST=brain01
3.3 brain01上，修改/etc/spark/conf/spark-defaults.conf
spark.master spark://brain01:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://brain01:8020/user/spark/eventlog
spark.yarn.historyServer.address http://brain01:18081
spark.executor.memory 12g
spark.logConf true
spark.yarn.jar hdfs://brain01:8020/user/spark/share/lib/spark-assembly.jar
3.4 brain01上，修改/etc/default/spark
export SPARK_HISTORY_SERVER_LOG_DIR=hdfs://ctdn01:8020/user/spark/eventlog
3.5 scp 上述各文件 brain02/3/4:/上述各文件（含目录）
3.6 hdfs操作：
sudo -u hdfs hadoop fs -mkdir /user/spark
sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
sudo -u hdfs hadoop fs -mkdir /user/spark/eventlog
sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory
sudo -u hdfs hadoop fs -chmod 1777 /user/spark/eventlog
3.7 优化：向HDFS上传spark-assembly.jar文件，从而提高集群加载该依赖文件的速度;上传spark-examples.jar文件是为了提高cluster模式下加载应用程序的速度
vi /etc/spark/conf/spark-defaults.conf
spark.yarn.jar hdfs://brain01:8020/user/spark/share/lib/spark-assembly.jar
执行如下命令：
sudo -u hdfs hadoop fs -mkdir -p /user/spark/share/lib
sudo -u hdfs hadoop fs -put /usr/lib/spark/lib/spark-assembly.jar /user/spark/share/lib/spark-assembly.jar
sudo -u hdfs hadoop fs -put /usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-hadoop2.6.0-cdh5.8.0.jar /user/spark/share/lib/spark-examples.jar
sudo -u hdfs hadoop fs -chown -R root:spark /user/spark/share/lib
4. 启动服务
brain01：
sudo service spark-master start
sudo service spark-history-server start
sudo service spark-worker start
brain02/3/4:
sudo service spark-worker start
5. 测试
5.1 yarn-client执行模式: 连接到YARN集群，driver在client运行，而executor在cluster中运行
spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode client --master yarn --driver-library-path /usr/lib/hadoop/lib/native/ --driver-class-path /usr/lib/hadoop/lib/ /usr/lib/spark/examples/lib/spark-examples-1.6.0-cdh5.8.0-hadoop2.6.0-cdh5.8.0.jar 10
5.2 yarn-cluster执行模式: 连接到YARN集群，driver和executor都在cluster中运行
spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn --driver-library-path /usr/lib/hadoop/lib/native/ --driver-class-path /usr/lib/hadoop/lib/ hdfs://brain01:8020/user/spark/share/lib/spark-examples.jar 10