filter a RDD

How to filter a RDD according to a function based another RDD in Spark?
当输入变得很大时,只有有限的内存,那么应该增加number of splits:

val input = List("a,b,c,d","b,c,e","a,c,d","e,g")
val splitSize = 10000
val numSplits = (inout.size / splitSize) +1 //必须 > 0
val groups = sc.parallelize(input, numSplits) //分片
val weights = Array(("a", 3), ("b", 2), ("c", 5), ("d", 1), ("e", 9), ("f", 4), ("g", 6)).toMap
def isHeavy(inp:String) = inp.split(",").map(weights(_)).sum > 12
val result = groups.filter(isHeavy)

也可以考虑使用spark.executor.memory增加executor memory size

你可能感兴趣的:(scala,spark,filter)