【菜鸟系列】spark常用算子总结(scala、java)--map,flatMap,flatMapToPair

map,flatMap,flatMapToPair是最常用的算子,map算子的输入和输出是一对一的,也就是子RDD的分区与父RDD的分区时一对一的关系;flatMap是压平,输入和输出是一对多的关系;需要注意的是:scala版本的map可以将RDD转成PairRDD,但是在java版本中,这个功能是通过mapToPair函数实现的,需要实现PairFunction函数;scala版本没有flatMapToPair函数,是通过先flatMap,在map来实现的;java版本的flatMapToPair,需要实现PairFlatMapFunction函数;java版本的flatMap是实现FlatMapFunction

scala
map样例

val conf = new SparkConf().setAppName("jiangtao_demo").setMaster("local")
  val sc = new SparkContext(conf)
  val data = sc.makeRDD(List("pandas pip","numpy","pip","pip","pip"))
  //map函数是一对一的,scala中map函数可以将rdd转为pairrdd,java是两个函数,map和mapToPair
  val rdd1 = data.map((_,1))

flatMap样例

  val conf = new SparkConf().setAppName("jiangtao_demo").setMaster("local")
  val sc = new SparkContext(conf)
  val data = sc.makeRDD(List("pandas pip","numpy","pip","pip","pip"))
  //flatMap是将数据压平,输出的是一对多的,在此例中输出的是6个元素
  val rdd2 = data.flatMap(_.split(" "))

wordcount

sc.textFile("hdfs://bjdx_clusters/test").flatMap(_.split(" ")).map((_+_)).reduceByKey(_+_)

================================================================
java
map样例
输入与输出是一对一的关系!!!!!

SparkConf conf = new SparkConf().setAppName("jiangtao_demo").setMaster("local");
        JavaSparkContext jsc = new JavaSparkContext(conf);
        //并行集合生成JavaRDD
        JavaRDD lines = jsc.parallelize(Arrays.asList("pandas pip","numpy","pip","pip","pip"));
        //map 输出包含newTail的元素
        JavaRDD<String> mapResult = lines.map(new Function<String,String>() {
            @Override
            public String call(String o) throws Exception {
                return o.concat("newTail");
            }
        });

        //map算子输入与输出一对一,输出结果为
        JavaRDDString,Integer>>> maprdd = lines.map(new Function<String,IterableString,Integer>>>(){
            public IterableString,Integer>> call(String line) throws Exception{
                String[] fields = line.split(" ");
                ArrayListString,Integer>> al = new ArrayListString,Integer>>();
                for(int i=0; inew Tuple2(fields[i],1));
                }
                return al;
            }
        });
//map算子输入与输出一对一
        // 结果:[[(pandas,1), (pip,1)], [(numpy,1)], [(pip,1)], [(pip,1)], [(pip,1)]]
        JavaRDDString,Integer>>> maprdd = lines.map(new Function<String,IterableString,Integer>>>(){
            public IterableString,Integer>> call(String line) throws Exception{
                String[] fields = line.split(" ");
                ArrayListString,Integer>> al = new ArrayListString,Integer>>();
                for(int i=0; inew Tuple2(fields[i],1));
                }
                return al;
            }
        });

flatMap
与输入是一对多的关系,返回的是可迭代的list

JavaRDD<String> flatMapResult = lines.flatMap(new FlatMapFunction<String,String>() {
            @Override
            public Iterable<String> call(String line) throws Exception {
                return Arrays.asList(line.toString().split(" "));
            }
        });

flatMapToPair
注意返回值,返回的是list,里面封装了tuple2

JavaPairRDD<String,Integer> result1 = lines.flatMapToPair(new PairFlatMapFunction<String,String,Integer>(){
            @Override
            public IterableString,Integer>> call(String line) throws Exception {
                String[] fields = line.split(" ");
                ArrayListString,Integer>> al = new ArrayListString,Integer>>();
                for(int i=0; inew Tuple2<String,Integer>(fields[i],1));
                }
                return al;
            }
        });

后续会补全更多的常用算子,敬请关注,此文原创,转载请注明出处

你可能感兴趣的:(spark)