spark安装与使用（入门）
安装环境： Ubuntu sever版 ，java ，scala，
一：在linux下安装java环境（自行安装jdk）
二：安装Scala2.9.3
$ tar -zxf scala-2.9.3.tgz
$ sudo mv scala-2.9.3 /usr/lib
$ sudo vim /etc/profile
# add the following lines at the end
export SCALA_HOME=/usr/lib/scala-2.9.3
export PATH=$PATH:$SCALA_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
source /etc/profile
# test
$ scala -version三：安装spark
从官网下载最新版本的spark，截止目前最新版的是1.5.1.下载地址：http://spark.apache.org/downloads.html
记住选择预编译好的文件下载，选择Pre-build for Hadoop 2.6 and later,下载的文件为spark-1.5.1-bin-hadoop2.6.tgz
解压
$ tar -zxf spark-1.5.1-bin-hadoop2.6.tgz设置SPARK_EXAMPLES_JAR 环境变量
$ vim ~/.bash_profile
# add the following lines at the end
export SPARK_EXAMPLES_JAR=$HOME/spark-0.7.2/examples/target/scala-2.9.3/spark-examples_2.9.3-0.7.2.jar
# save and exit vim
#make the bash profile take effect immediately
$ source /etc/profile
这一步其实最关键，很不幸的是，官方文档和网上的博客，都没有提及这一点。我是偶然看到了这两篇帖子，Running SparkPi, Null pointer exception when running ./run spark.examples.SparkPi local，才补上了这一步，之前死活都无法运行SparkPi。
（可选）设置 SPARK_HOME环境变量，并将SPARK_HOME/bin加入PATH
$ vim ~/.bash_profile
# add the following lines at the end
export SPARK_HOME=$HOME/spark-0.7.2
export PATH=$PATH:$SPARK_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
$ source /etc/profile单机运行spark
四：Spark配置
配置Spark环境变量
cd $SPARK_HOME/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh 添加以下内容：
export JAVA_HOME=/usr/local/java-1.7.0
export HADOOP_HOME=/opt/hadoop-2.3.0-cdh5.0.0
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SCALA_HOME=/usr/local/scala-2.11.4
export SPARK_HOME=/home/lxw1234/spark-1.3.1-bin-hadoop2.3
export SPARK_MASTER_IP=127.0.0.1
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8099
export SPARK_WORKER_CORES=3 //每个Worker使用的CPU核数
export SPARK_WORKER_INSTANCES=1 //每个Slave中启动几个Worker实例
export SPARK_WORKER_MEMORY=10G //每个Worker使用多大的内存
export SPARK_WORKER_WEBUI_PORT=8081 //Worker的WebUI端口号
export SPARK_EXECUTOR_CORES=1 //每个Executor使用使用的核数
export SPARK_EXECUTOR_MEMORY=1G //每个Executor使用的内存
export SPARK_CLASSPATH=/opt/hadoop-lzo/current/hadoop-lzo.jar //由于要用到lzo，因此需要配置
export SPARK_CLASSPATH=$SPARK_CLASSPATH:$CLASSPATH
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$HADOOP_HOME/lib/native- 配置Slave
cp slaves.template slaves
vi slaves 添加以下内容：
localhost
五、配置免密码ssh登陆
因为Master和Slave处于一台机器，因此配置本机到本机的免密码ssh登陆，如有其他Slave，都需要配置Master到Slave的无密码ssh登陆。
cd ~/
ssh-keygen （一路回车）
cd .ssh/
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys六、启动Spark Master
cd $SPARK_HOME/sbin/
./start-master.sh
启动日志位于 $SPARK_HOME/logs/目录下，正常启动的日志如下：
15/06/05 14:54:16 INFO server.AbstractConnector: Started SelectChannelConnector@localhost:6066
15/06/05 14:54:16 INFO util.Utils: Successfully started service on port 6066.
15/06/05 14:54:16 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
15/06/05 14:54:16 INFO master.Master: Starting Spark master at spark://127.0.0.1:7077
15/06/05 14:54:16 INFO master.Master: Running Spark version 1.3.1
15/06/05 14:54:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/06/05 14:54:16 INFO server.AbstractConnector: Started [email protected]:8099
15/06/05 14:54:16 INFO util.Utils: Successfully started service ‘MasterUI’ on port 8099.
15/06/05 14:54:16 INFO ui.MasterWebUI: Started MasterWebUI at http://127.1.1.1:8099
15/06/05 14:54:16 INFO master.Master: I have been elected leader! New state: ALIVE
七、启动Spark Slave
cd $SPARK_HOME/sbin/
./start-slaves.sh
会根据$SPARK_HOME/conf/slaves文件中配置的主机，逐个ssh过去，启动Spark Worker
成功启动后，在WebUI界面上可以看到，已经有Worker注册上来了，如图：
在浏览器输入：http://192.168.1.84:8080/ （前面为master的ip地址）
八、简单小实例（统计文件中出现最多的50个单词）
在bin目录下直接运行./spark-shell
hadoop@Master:/usr/local/spark-1.5.1-bin-hadoop2.6/bin$ ./spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.1
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/10/13 19:12:16 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
Spark context available as sc.
15/10/13 19:12:18 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/10/13 19:12:19 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/10/13 19:12:35 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
15/10/13 19:12:35 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
15/10/13 19:12:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/13 19:12:39 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/10/13 19:12:39 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
SQL context available as sqlContext.var srcFile = sc.textFile("/usr/local/kern.log")
var a = srcFile.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)
a.map(word=>(word._2,word._1)).sortByKey(false).map(word=>(word._2,word._1)).take(50).foreach(println)
结果打印在终端：
在4040端口可查看job的情况 http://192.168.1.84:4040/jobs/
八、Spark Java programming (Spark and Spark Streaming)
1：spark批处理：统计一个文件中出现a和出现b的单词数：SimpleApp.java
package org.apache.eagle.spark_streaming_kafka;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
public class SimpleApp {
public static void main(String[] args) {
String logFile = "/var/log/boot.log"; // Should be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple Application");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD logData = sc.textFile(logFile).cache();
long numAs = logData.filter(new Function() {
/**
*
*/
private static final long serialVersionUID = 1L;
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function() {
public Boolean call(String s) { return s.contains("b"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
}
}
2：Spark Streaming， 读取kafka数据做单词统计。
package org.apache.eagle.spark_streaming_kafka;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Pattern;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.streaming.Duration;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import com.google.common.collect.Lists;
import scala.Tuple2;
/**
* spark-streaming-kafka
*
*/
public class JavaKafkaWordCount
{
private static final Pattern SPACE = Pattern.compile(" ");
private JavaKafkaWordCount() {
}
public static void main( String[] args )
{
String zkQuorum = "10.64.255.161";
String group = "test-consumer-group";
SparkConf sparkConf = new SparkConf().setAppName("JavaKafkaWordCount");
// Create the context with 2 seconds batch size
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));
Map topicMap = new HashMap();
topicMap.put("noise",1);
JavaPairReceiverInputDStream messages =
KafkaUtils.createStream(jssc, zkQuorum, group, topicMap);;
JavaDStream lines = messages.map(new Function, String>() {
public String call(Tuple2 tuple2) {
return tuple2._2();
}
});
JavaDStream words = lines.flatMap(new FlatMapFunction() {
public Iterable call(String x) {
return Lists.newArrayList(SPACE.split(x));
}
});
JavaPairDStream wordCounts = words.mapToPair(
new PairFunction() {
public Tuple2 call(String s) {
return new Tuple2(s, 1);
}
}).reduceByKey(new Function2() {
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
wordCounts.print();
jssc.start();
jssc.awaitTermination();
}
}
注意几点：
1：环境：要确保spark在本机中正确安装，安装步骤如上所述。zookeeper集群和kafka集群要安装好，kafka的topic要新建好。
2：之前运行遇到找不到jar的情况(kafkaUtil)，原因没有把所有依赖的jar包都打包到最终的jar包里去。应在pom.xml中添加一下：
src/main/java
src/test/java
maven-assembly-plugin
jar-with-dependencies
org.apache.eagle.spark_streaming_kafka.JavaKafkaWordCount
make-assembly
package
single
3：如何提交任务？spark和spark streaming提交的方式都一样，用$SPARK_HOME/bin/soark-submit脚本提交，进入bin目录下，
以下是spark streaming任务提交，具体如下：
./spark-submit --master local[8] /home/zqin/workspace/spark-streaming-kafka/target/spark-streaming-kafka-0.0.1-SNAPSHOT-jar-with-dependencies.jar
由于在pom.xml中指明了入口类，因此不用加--class，如果没有指明，在命令中要用--class 指明入口。
以下是spark任务提交：
./spark-submit --class org.apache.eagle.spark_streaming_kafka.SimpleApp --master local[8] /home/zqin/workspace/spark-streaming-kafka/target/spark-streaming-kafka-0.0.1-SNAPSHOT-jar-with-dependencies.jar
需要指明程序main入口。
4：在运行spark streaming时，控制台满屏日志，不好查看结果，
在Spark的conf目录下，把log4j.properties.template修改为log4j.properties，把log4j.rootCategory=INFO, console改为log4j.rootCategory=WARN, console即可抑制Spark把INFO级别的日志打到控制台上。如果要显示全面的信息，则把INFO改为DEBUG
。
九、关闭spark
在spark目录下输入：sbin/stop-all.sh