BlockManager架构原理、运行流程图和源码解密(DT大数据梦工厂)
内容：
1、BlockManager运行实例；
2、BlockManager原理流程图；
3、BlockManager源码解析；
BlockManager是管理管理Spark运行时读写，包括数据存储本身。由于Spark是分布式的，所以BlockManager也是分布式，BlockManager本身是一个比较庞大的模块。本身源码量非常大。
==========从Application启动的角度来观察Block Manager============
1、在Application启动的时候，会在SparkEnv中注册BlockManagerMaster以及MapOutputTracker，其中BlockManagerMaster：
1）BlockManagerMaster，在对整个集群的block数据进行管理的；
2）MapOutputTracker，跟踪所有的Mapper的输出的；
2、BlockManageEndpoint是一个消息体，通过远程消息通信的方式去管理所有节点的BlockManager；
3、每启动一个ExecutorBackend都会实例化BlockManage并通过远程通信的方式注册给BlockManageMaster，实质上是Executor中的BlockManager在启动的时候注册给了Driver上的BlockManagerMasterEndpoint；
4、MemoryStore是BlockManager中专门负责内存数据存储和读写的类；
5、DiskStore是BlockManager中专门负责基于磁盘数据存储和读写的类；
6、DiskBlockManager：管理LogicBlock与Disk上的Physic Block之间的映射关系并负责磁盘文件的创建、读写等；
==========从Job运行的角度来观察Block Manager============
用36讲的程序:
root@Master:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/bin# ./spark-shell --master spark://Master:7077,Worker1:7077,Worker2:7077
scala> sc.textFile("/historyserverforSpark/README.md", 3).flatMap(_.split(" ")).map(word=>(word,1)).reduceByKey(_+_,1).saveAsTextFile("/historyserverforSpark /output/onclick3")
16/02/20 15:32:47 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 212.8 KB, free 539.1 KB)
16/02/20 15:32:47 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 19.7 KB, free 558.8 KB)
16/02/20 15:32:47 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.145.131:53080 (size: 19.7 KB, free: 1247.2 MB)
16/02/20 15:32:47 INFO spark.SparkContext: Created broadcast 3 from textFile at <console>:28
16/02/20 15:32:49 INFO spark.SparkContext: Starting job: saveAsTextFile at <console>:28
16/02/20 15:32:49 INFO mapred.FileInputFormat: Total input paths to process : 1
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Registering RDD 9 (map at <console>:28)
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Got job 1 (saveAsTextFile at <console>:28) with 1 output partitions
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (saveAsTextFile at <console>:28)
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 2)
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[9] at map at <console>:28), which has no missing parents
16/02/20 15:32:49 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 4.1 KB, free 562.9 KB)
16/02/20 15:32:49 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.3 KB, free 565.2 KB)
16/02/20 15:32:49 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.145.131:53080 (size: 2.3 KB, free: 1247.2 MB)
16/02/20 15:32:49 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
16/02/20 15:32:49 INFO scheduler.DAGScheduler: Submitting 3 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[9] at map at <console>:28)
16/02/20 15:32:49 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 3 tasks
16/02/20 15:32:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, Master, partition 0,NODE_LOCAL, 2141 bytes)
16/02/20 15:32:49 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 3, Worker1, partition 1,NODE_LOCAL, 2141 bytes)
16/02/20 15:32:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 4, Master, partition 2,NODE_LOCAL, 2141 bytes)
16/02/20 15:32:49 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on Master:44388 (size: 2.3 KB, free: 511.1 MB)
16/02/20 15:32:49 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on Master:44388 (size: 19.7 KB, free: 511.1 MB)
16/02/20 15:32:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 4) in 369 ms on Master (1/3)
16/02/20 15:32:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 394 ms on Master (2/3)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.145.131:53080 in memory (size: 19.7 KB, free: 1247.2 MB)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on Master:44388 in memory (size: 19.7 KB, free: 511.1 MB)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.145.131:53080 in memory (size: 22.6 KB, free: 1247.2 MB)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on Master:44388 in memory (size: 22.6 KB, free: 511.1 MB)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on Worker1:35104 (size: 2.3 KB, free: 511.1 MB)
16/02/20 15:32:50 INFO spark.ContextCleaner: Cleaned accumulator 2
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.145.131:53080 in memory (size: 2.3 KB, free: 1247.2 MB)
16/02/20 15:32:50 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on Master:44388 in memory (size: 2.3 KB, free: 511.1 MB)
16/02/20 15:32:50 INFO spark.ContextCleaner: Cleaned accumulator 1
16/02/20 15:32:50 INFO spark.ContextCleaner: Cleaned shuffle 0
16/02/20 15:32:52 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on Worker1:35104 (size: 19.7 KB, free: 511.1 MB)
16/02/20 15:32:58 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 3) in 9579 ms on Worker1 (3/3)
16/02/20 15:32:58 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
16/02/20 15:32:58 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (map at <console>:28) finished in 9.569 s
16/02/20 15:32:58 INFO scheduler.DAGScheduler: looking for newly runnable stages
16/02/20 15:32:58 INFO scheduler.DAGScheduler: running: Set()
16/02/20 15:32:58 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 3)
16/02/20 15:32:58 INFO scheduler.DAGScheduler: failed: Set()
16/02/20 15:32:58 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[11] at saveAsTextFile at <console>:28), which has no missing parents
16/02/20 15:32:59 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 64.9 KB, free 303.8 KB)
16/02/20 15:32:59 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 22.6 KB, free 326.4 KB)
16/02/20 15:32:59 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.145.131:53080 (size: 22.6 KB, free: 1247.2 MB)
16/02/20 15:32:59 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006
16/02/20 15:32:59 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[11] at saveAsTextFile at <console>:28)
16/02/20 15:32:59 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/02/20 15:32:59 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, Master, partition 0,NODE_LOCAL, 1894 bytes)
16/02/20 15:32:59 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on Master:44388 (size: 22.6 KB, free: 511.1 MB)
16/02/20 15:32:59 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to Master:47683
16/02/20 15:32:59 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 166 bytes
16/02/20 15:33:02 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 3598 ms on Master (1/1)
16/02/20 15:33:02 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
16/02/20 15:33:02 INFO scheduler.DAGScheduler: ResultStage 3 (saveAsTextFile at <console>:28) finished in 3.601 s
16/02/20 15:33:02 INFO scheduler.DAGScheduler: Job 1 finished: saveAsTextFile at <console>:28, took 13.339011 s
1、首先通过MemoryStore来存储广播变量；
2、在Driver中是通过BlockManagerInfo来管理集群中每个ExecutorBackend中的BlockManager中的元数据信息；
3、当改变了具体的ExecutorBackend上的信息后，就必须发消息给Driver中的BlockManagerMaster来更新相应的BlockManagerInfo的信息；
4、当执行第二个Stage的时候，第二个Stage会向Driver中的MapOutputTrackerEnpoint发送消息请求上一个Stage中相应的输出，此时MapOutputTrackerMaster会把上一个STage的输出数据的元数据信息发送给当前请求的Stage；
王家林老师名片：
中国Spark第一人
新浪微博：http://weibo.com/ilovepains
微信公众号：DT_Spark
博客：http://blog.sina.com.cn/ilovepains
手机：18610086859
QQ：1740415547
本文出自 “一枝花傲寒” 博客，谢绝转载！