这样的spark你喜欢吗?

BlockManager定义 

BlockManager是Spark的分布式存储系统,与我们平常说的分布式存储系统是有区别的,区别就是这个分布式存储系统只会管理Block块数据,它运行在所有节点上。BlockManager的结构是Maser-Slave架构,Master就是Driver上的BlockManagerMaster,Slave就是每个Executor上的BlockManager。BlockManagerMaster负责接受Executor上的BlockManager的注册以及管理BlockManager的元数据信息

BlockManager原理

从上边的定义我们已经得知,BlockManager是分布式的,运行在各个节点上的。从BlockManager的创建过程来看,其实Block是运行在Driver和每个Executor的。因为在创建SparkContext的时候,会调用SparkEnv.blockManager.initialize方法实例化BlockManager对象,在创建Executor对象的时候也会创建BlockManager。

在初始化BlockManager的时候,第一步会初始化BlockTransferService的init方法(子类NettyBlockTransferService实现了init方法),这个方法的作用就是初始化Netty服务,为拉取block数据提供服务。第二步是调用shuffleClient的init方法,shuffleClient这个引用有可能是BlockTransferService有可能是ExternalShuffleClient,取决于我们的配置文件是否配置了externalShuffleServiceEnabled未开启状态,其实无论是哪种,都是为了对外提供服务,能够使block数据再节点之间流动起来。

BlockManagerMaster调用registerBlockManager方法,向BlockManagerMaster(其实BlockManagerMasterEndpoint)发送BlockManager的注册请求。

BlockManagerMaster(其实BlockManagerMasterEndpoint)接受到BlockManager的注册请求后。会调用register方法,开始注册Executor上的BlockManager,注册完成以后将BlockManagerId返回给对应Executor上的BlockManager。 

BlockManager源码

当我们的程序启动的时候,首先会创建SparkContext对象,在创建SparkContext对象的时候就会调用_env.blockManager.initialize(_applicationId)创建BlockManager对象,这个BlockManager就是Driver上的BlockManager,它负责管理集群中Executor上的BlockManager

SparkContext里创建BlockManager代码片段

//为Driver创建BlockManager_env.blockManager.initialize(_applicationId)

1

2

创建Executor的时候,Executor内部会调用_env.blockManager.initialize(conf.getAppId)方法创建BlockManager

if (!isLocal) {    env.metricsSystem.registerSource(executorSource)    env.blockManager.initialize(conf.getAppId)  }

1

2

3

4

BlockManager类里的initialize方法,该方法作用是创建BlockManager,并且向BlockManagerMaster进行注册

definitialize(appId: String): Unit = {//初始化BlockTransferService,其实是它的子类NettyBlockTransferService是下了init方法,//该方法的作用就是初始化传输服务,通过传输服务可以从不同的节点上拉取Block数据blockTransferService.init(this)    shuffleClient.init(appId)//设置block的复制分片策略,由spark.storage.replication.policy指定blockReplicationPolicy = {valpriorityClass = conf.get("spark.storage.replication.policy", classOf[RandomBlockReplicationPolicy].getName)valclazz = Utils.classForName(priorityClass)valret = clazz.newInstance.asInstanceOf[BlockReplicationPolicy]      logInfo(s"Using $priorityClass for block replication policy")      ret    }//根据给定参数为对对应的Executor封装一个BlockManagerId对象(块存储的唯一标识)//executorID:executor的Id,blockTransferService.hostName:传输Block数据的服务的主机名//blockTransferService.port:传输Block数据的服务的主机名valid = BlockManagerId(executorId, blockTransferService.hostName, blockTransferService.port, None)//调用BlockManagerMaster的registerBlockManager方法向Driver上的BlockManagerMaster注册validFromMaster = master.registerBlockManager(      id,      maxMemory,      slaveEndpoint)//更新BlockManagerIdblockManagerId =if(idFromMaster !=null) idFromMasterelseid//判断是否开了外部shuffle服务shuffleServerId =if(externalShuffleServiceEnabled) {      logInfo(s"external shuffle service port = $externalShuffleServicePort")      BlockManagerId(executorId, blockTransferService.hostName, externalShuffleServicePort)    }else{      blockManagerId    }// 如果开启了外部shuffle服务,并且该节点是Driver的话就调用registerWithExternalShuffleServer方法//将BlockManager注册在本地if(externalShuffleServiceEnabled && !blockManagerId.isDriver) {      registerWithExternalShuffleServer()    }    logInfo(s"Initialized BlockManager: $blockManagerId")  }

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

BlockManagerMaster类里的registerBlockManager方法,向Driver发送RegisterBlockManager消息进行注册

defregisterBlockManager(blockManagerId: BlockManagerId,maxMemSize: Long,slaveEndpoint: RpcEndpointRef):BlockManagerId = {    logInfo(s"Registering BlockManager $blockManagerId")    //向Driver发送注册BlockManager请求    //blockManagerId:块存储的唯一标识,里边封装了该BlockManager所在的executorId,提供Netty服务的主机名和端口    //maxMemSize最大的内存    val updatedId = driverEndpoint.askWithRetry[BlockManagerId](      RegisterBlockManager(blockManagerId, maxMemSize, slaveEndpoint))    logInfo(s"Registered BlockManager $updatedId")    updatedId  }

1

2

3

4

5

6

7

8

9

10

BlockManagerMasterEndpoint类里的receiveAndReply方法,这个方法就是接受请求的消息,然后并处理请求

defreceiveAndReply(context: RpcCallContext):PartialFunction[Any, Unit] = {    //BlocManagerMasterEndPoint接收到来自Executor上的BlockManager注册请求的时候,    //调用register方法开始注册BlockManager,    case RegisterBlockManager(blockManagerId, maxMemSize, slaveEndpoint) =>      context.reply(register(blockManagerId, maxMemSize, slaveEndpoint))    //....其余代码省略  }

1

2

3

4

5

6

7

BlockManagerMasterEndpoint类里的register方法,该方法的作用就是开始注册executor上的BlockManager

// BlockManager到BlockManaInfo的映射private val blockManagerInfo = newmutable.HashMap[BlockManagerId, BlockManagerInfo]//executorId到BlockManagerId的映射private val blockManagerIdByExecutor = newmutable.HashMap[String, BlockManagerId]    private defregister(      idWithoutTopologyInfo: BlockManagerId,      maxMemSize: Long,      slaveEndpoint: RpcEndpointRef): BlockManagerId = {//利用从Executor上传过来的BlockManagerId信息重新封装BlockManagerId,并且//之前传过来没有拓扑信息,这次直接将拓扑信息也封装进去,得到一个更完整的BlockManagerIdvalid= BlockManagerId(      idWithoutTopologyInfo.executorId,      idWithoutTopologyInfo.host,      idWithoutTopologyInfo.port,      topologyMapper.getTopologyForHost(idWithoutTopologyInfo.host))    val time = System.currentTimeMillis()//判断当前这个BlockManagerId是否注册过,注册结构为:HashMap[BlockManagerId, BlockManagerInfo]//如果没注册过就向下执行开始注册if(!blockManagerInfo.contains(id)) {//首先会根据executorId查找内存缓存结构中是否有对应的BlockManagerId,如果为存在那么就将调用removeExecutor方法,//将executor从BlockManagerMaster中移除,首先会移除executorId对应的BlockManagerId,然后在移除该旧的BlockManager//其实就是移除以前的注册过的旧数据blockManagerIdByExecutor.get(id.executorId) match {caseSome(oldId) =>// A block manager of the same executor already exists, so remove it (assumed dead)logError("Got two different block manager registrations on same executor - "+ s" will replace old one $oldId with new one $id")          removeExecutor(id.executorId)caseNone =>      }      logInfo("Registering block manager %s with %s RAM, %s".format(id.hostPort, Utils.bytesToString(maxMemSize),id))//将executorId与BlockManagerId映射起来,放到内存缓存中blockManagerIdByExecutor(id.executorId) =id//将BlockManagerId与BlockManagerInfo映射起来,放入内存缓存中//BlockManagerInfo封住了BlockMangerId,当前注册的事件,最大的内存blockManagerInfo(id) = new BlockManagerInfo(id, System.currentTimeMillis(), maxMemSize, slaveEndpoint)    }    listenerBus.post(SparkListenerBlockManagerAdded(time,id, maxMemSize))id}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

BlockManagerMasterEndpoint类里的removeExecutor方法,该方法的作用就是移除掉之前注册过的旧数据

你可能感兴趣的:(这样的spark你喜欢吗?)