一、updateStateByKey算子应用示例
object SparkStreamingApp {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[2]").setAppName("SparkStreamingApp")
val ssc = new StreamingContext(conf,Seconds(5))
val lines = ssc.socketTextStream("hadoop000",9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(x=>(x,1))
val wordcounts = pairs.reduceByKey(_+_)
wordcounts.print()
ssc.start()
ssc.awaitTermination()
}
}
上面这段简单的wordcount代码，输入一批数据就处理一批数据，输出结果都是当前批次数据的统计结果，那么怎么能够统计出所有输入数据的一个总的统计结果呢？再此介绍一下updateStateByKey方法
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
object UpdateStatebyKeyApp {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[2]").setAppName("UpdateStatebyKeyApp")
val ssc = new StreamingContext(conf,Seconds(5))
ssc.checkpoint("hdfs://192.168.137.251:9000/spark/data")
val lines = ssc.socketTextStream("hadoop000",9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(x=>(x,1))
val wordcounts = pairs.updateStateByKey(updateFunction)
wordcounts.print()
ssc.start()
ssc.awaitTermination()
}
def updateFunction(newValues: Seq[Int], oldValues: Option[Int]): Option[Int] = {
val newCount = newValues.sum
val oldCount = oldValues.getOrElse(0)
Some(newCount + oldCount)
}
}
输入：
[hadoop@hadoop000 ~]$ nc -lk 9999
huluwa huluwa dawa erwa
spark计算结果：
18/11/13 14:19:00 INFO CheckpointWriter: Submitted checkpoint of time 1542089940000 ms to writer queue
18/11/13 14:19:00 INFO CheckpointWriter: Saving checkpoint for time 1542089940000 ms to file 'hdfs://192.168.137.251:9000/spark/data/checkpoint-1542089940000'
-------------------------------------------
Time: 1542089940000 ms
-------------------------------------------
(huluwa,2)
(dawa,1)
(erwa,1)
...
...
...
-------------------------------------------
Time: 1542089945000 ms
-------------------------------------------
(huluwa,2)
(dawa,1)
(erwa,1)
...
...
...
-------------------------------------------
Time: 1542089950000 ms
-------------------------------------------
(huluwa,2)
(dawa,1)
(erwa,1)
如果不输入新的数据，会一直展示之前的结果
再输入一条新的数据：
[hadoop@hadoop000 ~]$ nc -lk 9999
huluwa huluwa dawa erwa
huluwa huluwa dae erwa
输出结果：
-------------------------------------------
Time: 1542089970000 ms
-------------------------------------------
(dae,1)
(huluwa,4)
(dawa,1)
(erwa,2)
继续输入数据：
[hadoop@hadoop000 ~]$ nc -lk 9999
huluwa huluwa dawa erwa
huluwa huluwa dae erwa
huluwa huluwa dawa erwa
huluwa huluwa dawa erwa
huluwa huluwa dawa erwa
huluwa huluwa dae erwa
huluwa huluwa dae erwa
结果：
-------------------------------------------
Time: 1542090080000 ms
-------------------------------------------
(dae,3)
(huluwa,14)
(dawa,4)
(erwa,7)
查看checkpoint文件夹下，发现有很多类似于checkpoint-1542090065000的状态文件
[hadoop@hadoop000 data]$ hadoop fs -ls /spark/data
Found 15 items
-rw-r--r-- 1 ÃÎcandybear supergroup 3514 2018-11-13 22:21 /spark/data/checkpoint-1542090065000
-rw-r--r-- 1 ÃÎcandybear supergroup 3518 2018-11-13 22:21 /spark/data/checkpoint-1542090065000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3511 2018-11-13 22:21 /spark/data/checkpoint-1542090070000
-rw-r--r-- 1 ÃÎcandybear supergroup 3518 2018-11-13 22:21 /spark/data/checkpoint-1542090070000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3514 2018-11-13 22:21 /spark/data/checkpoint-1542090075000
-rw-r--r-- 1 ÃÎcandybear supergroup 3518 2018-11-13 22:21 /spark/data/checkpoint-1542090075000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3511 2018-11-13 22:21 /spark/data/checkpoint-1542090080000
-rw-r--r-- 1 ÃÎcandybear supergroup 3518 2018-11-13 22:21 /spark/data/checkpoint-1542090080000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3512 2018-11-13 22:21 /spark/data/checkpoint-1542090085000
-rw-r--r-- 1 ÃÎcandybear supergroup 3516 2018-11-13 22:21 /spark/data/checkpoint-1542090085000.bk
drwxr-xr-x - ÃÎcandybear supergroup 0 2018-11-13 22:21 /spark/data/e02daf3a-0805-4612-b612-67ca34d32ff8
drwxr-xrwx - ÃÎcandybear supergroup 0 2018-11-13 22:21 /spark/data/receivedBlockMetadata
这些checkpoint文件都是小文件，对hdfs的压力很大，怎么解决呢？下文会讲
附：updateStateByKey源码（1.6版本之前用这个）
/**
* Return a new "state" DStream where the state for each key is updated by applying
* the given function on the previous state of the key and the new values of each key.
* In every batch the updateFunc will be called for each state even if there are no new values.
* Hash partitioning is used to generate the RDDs with Spark's default number of partitions.
* @param updateFunc State update function. If `this` function returns None, then
* corresponding state key-value pair will be eliminated.
* @tparam S State type
*/
def updateStateByKey[S: ClassTag](
updateFunc: (Seq[V], Option[S]) => Option[S]
): DStream[(K, S)] = ssc.withScope {
updateStateByKey(updateFunc, defaultPartitioner())
}
1.6版本之后用mapWithState
/**
* :: Experimental ::
* Return a [[MapWithStateDStream]] by applying a function to every key-value element of
* `this` stream, while maintaining some state data for each unique key. The mapping function
* and other specification (e.g. partitioners, timeouts, initial state data, etc.) of this
* transformation can be specified using `StateSpec` class. The state data is accessible in
* as a parameter of type `State` in the mapping function.
*
* Example of using `mapWithState`:
* {{{
* // A mapping function that maintains an integer state and return a String
* def mappingFunction(key: String, value: Option[Int], state: State[Int]): Option[String] = {
* // Use state.exists(), state.get(), state.update() and state.remove()
* // to manage state, and return the necessary string
* }
*
* val spec = StateSpec.function(mappingFunction).numPartitions(10)
*
* val mapWithStateDStream = keyValueDStream.mapWithState[StateType, MappedType](spec)
* }}}
*
* @param spec Specification of this transformation
* @tparam StateType Class type of the state data
* @tparam MappedType Class type of the mapped data
*/
@Experimental
def mapWithState[StateType: ClassTag, MappedType: ClassTag](
spec: StateSpec[K, V, StateType, MappedType]
): MapWithStateDStream[K, V, StateType, MappedType] = {
new MapWithStateDStreamImpl[K, V, StateType, MappedType](
self,
spec.asInstanceOf[StateSpecImpl[K, V, StateType, MappedType]]
)
}
二、mapWithState算子应用示例
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, State, StateSpec, StreamingContext}
object MapWithStateApp {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[2]").setAppName("MapWithStateApp")
val ssc = new StreamingContext(conf,Seconds(5))
ssc.checkpoint("hdfs://192.168.137.251:9000/spark/data")
val lines = ssc.socketTextStream("hadoop000",9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(x=>(x,1)).reduceByKey(_+_)
// Update the cumulative count using mapWithState
// This will give a DStream made of state (which is the cumulative count of the words)
val mappingFunc = (word: String, one: Option[Int], state: State[Int]) => {
val sum = one.getOrElse(0) + state.getOption.getOrElse(0)
val output = (word, sum)
state.update(sum)
output
}
val wordcounts = pairs.mapWithState(StateSpec.function(mappingFunc))
wordcounts.print()
ssc.start()
ssc.awaitTermination()
}
}
在控制台输入一条数据：
[hadoop@hadoop000 ~]$ nc -lk 9999
hadoop spark spark hive
spark输出结果：
-------------------------------------------
Time: 1542098100000 ms
-------------------------------------------
(hive,1)
(spark,2)
(hadoop,1)
再输入两条数据：
[hadoop@hadoop000 ~]$ nc -lk 9999
hadoop spark spark hive
hadoop spark spark hive
hadoop spark spark hive
输出：
-------------------------------------------
Time: 1542098115000 ms
-------------------------------------------
(hive,3)
(spark,6)
(hadoop,3)
下面输入一条新的数据：
[hadoop@hadoop000 ~]$ nc -lk 9999
hadoop spark spark hive
hadoop spark spark hive
hadoop spark spark hive
huluwa huluwa dawa erwa
发现spark计算结果只展示与输入数据相匹配的结果
-------------------------------------------
Time: 1542098120000 ms
-------------------------------------------
(huluwa,2)
(dawa,1)
(erwa,1)
[hadoop@hadoop000 ~]$ nc -lk 9999
hadoop spark spark hive
hadoop spark spark hive
hadoop spark spark hive
huluwa huluwa dawa erwa
huluwa huluwa dawa erwa
-------------------------------------------
Time: 1542098125000 ms
-------------------------------------------
(huluwa,4)
(dawa,2)
(erwa,2)
那之前的计算结果是否还存在？验证一下：
[hadoop@hadoop000 ~]$ nc -lk 9999
hadoop spark spark hive
hadoop spark spark hive
hadoop spark spark hive
huluwa huluwa dawa erwa
huluwa huluwa dawa erwa
hadoop spark spark hive
从输出结果可以看到，之前的统计结果还存在，只是选择性的展示出来
-------------------------------------------
Time: 1542098135000 ms
-------------------------------------------
(hive,4)
(spark,8)
(hadoop,4)
打开checkpoint目录，和updateStateByKey一样，有很多checkpoint-时间戳的小文件存在
[hadoop@hadoop000 data]$ hadoop fs -ls /spark/data
Found 12 items
-rw-r--r-- 1 ÃÎcandybear supergroup 3974 2018-11-14 00:35 /spark/data/checkpoint-1542098120000
-rw-r--r-- 1 ÃÎcandybear supergroup 3978 2018-11-14 00:35 /spark/data/checkpoint-1542098120000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3975 2018-11-14 00:35 /spark/data/checkpoint-1542098125000
-rw-r--r-- 1 ÃÎcandybear supergroup 3979 2018-11-14 00:35 /spark/data/checkpoint-1542098125000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 3975 2018-11-14 00:35 /spark/data/checkpoint-1542098130000
-rw-r--r-- 1 ÃÎcandybear supergroup 3979 2018-11-14 00:35 /spark/data/checkpoint-1542098130000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 4037 2018-11-14 00:35 /spark/data/checkpoint-1542098135000
-rw-r--r-- 1 ÃÎcandybear supergroup 3979 2018-11-14 00:35 /spark/data/checkpoint-1542098135000.bk
-rw-r--r-- 1 ÃÎcandybear supergroup 4043 2018-11-14 00:35 /spark/data/checkpoint-1542098140000
-rw-r--r-- 1 ÃÎcandybear supergroup 4047 2018-11-14 00:35 /spark/data/checkpoint-1542098140000.bk
drwxr-xr-x - ÃÎcandybear supergroup 0 2018-11-14 00:35 /spark/data/d55f3470-753c-4735-9b18-1b2c75f3a300
drwxr-xrwx - ÃÎcandybear supergroup 0 2018-11-14 00:34 /spark/data/receivedBlockMetadata
那么updateStateByKey和mapWithState产生这些小文件应该怎么处理或者怎么规避产生这么多的小文件呢？
其实解决办法很简单，想要统计从某个时间段内的数据，可以不使用这两个算子，每个批次的数据处理之后在后面附上一个处理时间，然后保存到数据库比如MySQL中，等需要的时候，再取出历史数据进行统计，这样就从源头上避免了小文件的产生，数据库保存格式如下：
+---------+-------+---------------+
| word | count | timestamp |
+---------+-------+---------------+
| hive | 1 | 1542098135000 |
| spark | 2 | 1542098135000 |
| hadoop | 1 | 1542098135000 |
| hive | 1 | 1542098140000 |
| spark | 2 | 1542098140000 |
| hadoop | 1 | 1542098140000 |
| hive | 1 | 1542098145000 |
| spark | 2 | 1542098145000 |
| hadoop | 1 | 1542098145000 |
+---------+-------+---------------+