Spark学习笔记-安装部署与运行实例

先解压scala，本次选用版本scala-2.11.1

[hadoop@centos software]$ tar -xzvf scala-2.11.1.tgz

[hadoop@centos software]$ su -

[root@centos ~]# vi /etc/profile

添加如下内容：

SCALA_HOME=/home/hadoop/software/scala-2.11.1

PATH=$SCALA_HOME/bin

EXPORT SCALA_HOME

[root@centos ~]# source /etc/profile

[root@centos ~]# scala -version

Scala code runner version 2.11.1 -- Copyright 2002-2013, LAMP/EPFL

然后解压spark，本次选用版本spark-1.0.0-bin-hadoop1.tgz，这次用的是hadoop 1.0.4

[hadoop@centos software]$ tar -xzvf spark-1.0.0-bin-hadoop1.tgz

进入到spark的conf目录下

[hadoop@centos conf]$ cp spark-env.sh.template spark-env.sh

[hadoop@centos conf]$ vi spark-env.sh

添加如下内容：

export SCALA_HOME=/home/hadoop/software/scala-2.11.1

export SPARK_MASTER_IP=centos.host1

export SPARK_WORKER_MEMORY=2G

export JAVA_HOME=/usr/software/jdk

如果是要集群安装部署的话，需要修改文件conf/slaves，添加要作为worker的主机
然后将spark-1.0.0-bin-hadoop1目录拷贝到相应主机上，注意目录要一致。

启动

[hadoop@centos spark-1.0.0-bin-hadoop1]$ sbin/start-master.sh

可以通过 http://centos.host1:8080/ 看到对应界面

[hadoop@centos spark-1.0.0-bin-hadoop1]$ sbin/start-slaves.sh spark://centos.host1:7077

可以通过 http://centos.host1:8081/ 看到对应界面

下面在spark上运行第一个例子：与Hadoop交互的WordCount

首先将word.txt文件上传到HDFS上，这里路径是 hdfs://centos.host1:9000/user/hadoop/data/wordcount/001/word.txt

进入交互模式

[hadoop@centos spark-1.0.0-bin-hadoop1]$ master=spark://centos.host1:7077 ./bin/spark-shell

scala>val file=sc.textFile("hdfs://centos.host1:9000/user/hadoop/data/wordcount/001/word.txt")  

scala>val count=file.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_) 

scala>count.collect()

可以看到控制台有如下结果：

res0: Array[(String, Int)] = Array((hive,2), (zookeeper,1), (pig,1), (spark,1), (hadoop,4), (hbase,2))

同时也可以将结果保存到HDFS上

scala>count.saveAsTextFile("hdfs://centos.host1:9000/user/hadoop/data/wordcount/001/result.txt")

接下来再来看下如何运行Java版本的WordCount

这里需要用到一个jar文件：spark-assembly-1.0.0-hadoop1.0.4.jar

WordCount代码如下：

 public class WordCount {

     private static final Pattern SPACE = Pattern.compile(" ");

     @SuppressWarnings("serial")
     public static void main(String[] args) throws Exception {
         if (args.length < 1) {
             System.err.println("Usage: JavaWordCount <file>");
             System.exit(1);
         }

         SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
         JavaSparkContext ctx = new JavaSparkContext(sparkConf);
         JavaRDD<String> lines = ctx.textFile(args[0], 1);

         JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
                     @Override
                     public Iterable<String> call(String s) {
                         return Arrays.asList(SPACE.split(s));
                     }
                 });

         JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>() {
                     @Override
                     public Tuple2<String, Integer> call(String s) {
                         return new Tuple2<String, Integer>(s, 1);
                     }
                 });

         JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() {
                     @Override
                     public Integer call(Integer i1, Integer i2) {
                         return i1 + i2;
                     }
                 });

         List<Tuple2<String, Integer>> output = counts.collect();
         for (Tuple2<?, ?> tuple : output) {
             System.out.println(tuple._1() + " : " + tuple._2());
         }

         ctx.stop();
     }
 }

导出类文件生成jar包，这里生成为mining.jar。然后执行下面命令，其中--class 指定主类，--master 指定spark master地址，后面是执行的jar和需要的参数。

[hadoop@centos spark-1.0.0-bin-hadoop1]$ bin/spark-submit --class org.project.modules.spark.java.WordCount --master spark://centos.host1:7077 /home/hadoop/project/mining.jar hdfs://centos.host1:9000/user/hadoop/data/wordcount/001/word.txt

可以看到控制台有如下结果：

spark : 1

hive : 2

hadoop : 4

zookeeper : 1

pig : 1

hbase : 2

最后再来看下如何运行Python版本的WordCount

WordCount代码如下：

 import sys
 from operator import add

 from pyspark import SparkContext

 if __name__ == "__main__":
     if len(sys.argv) != 2:
         print >> sys.stderr, "Usage: wordcount <file>"
         exit(-1)
     sc = SparkContext(appName="PythonWordCount")
     lines = sc.textFile(sys.argv[1], 1)
     counts = lines.flatMap(lambda x: x.split(' ')) \
                   .map(lambda x: (x, 1)) \
                   .reduceByKey(add)
     output = counts.collect()
     for (word, count) in output:
         print "%s: %i" % (word, count)

输入文件路径可以是本地也可以是HDFS上文件，命令如下：

[hadoop@centos spark-1.0.0-bin-hadoop1]$ bin/spark-submit --master spark://centos.host1:7077 /home/hadoop/project/WordCount.py /home/hadoop/temp/word.txt 

[hadoop@centos spark-1.0.0-bin-hadoop1]$ bin/spark-submit --master spark://centos.host1:7077 /home/hadoop/project/WordCount.py hdfs://centos.host1:9000/user/hadoop/data/wordcount/001/word.txt 

可以看到控制台有如下结果：

spark: 1

hbase: 2

hive: 2

zookeeper: 1

hadoop: 4

pig: 1

版权声明：本文为博主原创文章，未经博主允许不得转载。