Flink源码之Checkpoint执行流程
Checkpoint完整流程如上图所示：
- JobMaster的CheckpointCoordinator向所有SourceTask发送RPC触发一次CheckPoint
- SourceTask向下游广播CheckpointBarrier
- SouceTask完成状态快照后向JobMaster发送快照结果
- 非SouceTask在Barrier对齐后完成状态快照向JobMaster发送快照结果
- JobMaster保存SubTask快照结果
- JobMaster收到所有SubTask快照结果后保存快照信息，想SubTask通知Checkpoint完成
以下对整个流程具体说明。
CheckpointCoordinator
JobMaster将JobGraph转换为ExecutionGraph时，如果开启Checkpoint，会为ExecutionGraph生成一个CheckpointCoordinator
DefaultExecutionGraphBuilder.buildGraph//在此会将JobGraph转换为ExecutionGraph
DefaultExecutionGraph::new
DefaultExecutionGraph::attachJobGraph //创建ExecutionJobVertex
DefaultExecutionTopology.fromExecutionGraph //创建ExecutionTopology
DefaultExecutionGraph::enableCheckpointing //创建CheckpointCoordinator
DefaultExecutionGraph::createCheckpointPlanCalculator//创建DefaultCheckpointPlanCalculator
CheckpointCoordinator::new
CheckpointCoordinator封装了StateBackend和CheckpointStorage
StateBackend负责管理状态：
- HashMapStateBackend //内存
- EmbeddedRocksDBStateBackend //内存+磁盘
CheckpointStorage则是负责存储StateBackend管理的状态：
- JobManagerCheckpointStorage //checkpoint state放入JobManager内存
- FileSystemCheckpointStorage //配置state.checkpoints.dir时
在为StreamTask构造SubtaskCheckpointCoordinatorImpl时会调用：
CheckpointStorage::createCheckpointStorage
创建CheckpointStorageAccess用于执行Checkpoint时解析状态存储位置
- MemoryBackendCheckpointStorageAccess //对应JobManagerCheckpointStorage
- FsCheckpointStorageAccess //对应FileSystemCheckpointStorage
CheckpointCoordinator在执行状态快照时会调用
CheckpointStorageAccess::resolveCheckpointStorageLocation
生成CheckpointStreamFactory用于生成读写状态数据流
- MemCheckpointStreamFactory //对应JobManagerCheckpointStorage
- FsCheckpointStreamFactory //对应FileSystemCheckpointStorage
Checkpoint触发流程
JobMaster状态转换为running后，通过CheckpointCoordinator向SourceTask发送TriggerCheckpoint
JobMaster端触发流程
JobMaster::start //RPCServer启动
JobMaster::onStart
JobMaster::startJobExecution
JobMaster::startJobMasterServices //获取RM地址后与RM建立连接
JobMaster::startScheduling
SchedulerBase::startScheduling
DefaultScheduler::startSchedulingInternal
SchedulerBase::transitionToRunning
DefaultExecutionGraph::transitionToRunning //调用ExecutionGraph监听器通知状态变化
CheckpointCoordinatorDeActivator::jobStatusChanges//触发checkpoint
CheckpointCoordinator::startCheckpointScheduler
CheckpointCoordinator::scheduleTriggerWithDelay //定时不断触发Checkpoint
CheckpointCoordinator::triggerCheckpoint
CheckpointCoordinator::startTriggeringCheckpoint
DefaultCheckpointPlanCalculator::calculateCheckpointPlan//Plan中会隔离出SourceTask作为作为Trigger Checkpoint的入口
CheckpointCoordinator::createPendingCheckpoint
CheckpointCoordinator::triggerCheckpointRequest
CheckpointCoordinator::triggerTasks
Execution::triggerCheckpoint //向每个SourceTask发送TriggerCheckpoint请求
Execution::triggerCheckpointHelper
TaskManagerGateway::triggerCheckpoint//向TaskExecutor发RPC
StreamTask端执行流程
SourceTask
SourceTask由JobMaster RPC直接触发，执行时先广播CheckpointBarrier，然后对状态执行异步快照
TaskExecutor::triggerCheckpoint
Task::triggerCheckpointBarrier
AbstractInvokable::triggerCheckpointAsync
SourceStreamTask::triggerCheckpointAsync
StreamTask::triggerCheckpointAsync
StreamTask::triggerCheckpointAsyncInMailbox
StreamTask::performCheckpoint
SubtaskCheckpointCoordinatorImpl::checkpointState
OperatorChain.broadcastEvent //广播CheckpointBarrier
CheckpointStorage::createCheckpointStorage//为JobId创建CheckpointStorageAccess
SubtaskCheckpointCoordinatorImpl::takeSnapshotSync
CheckpointStorageWorkerView::resolveCheckpointStorageLocation//CheckpointStorageAccess创建 CheckpointStreamFactory
OperatorChain::snapshotState //对每个Operator
RegularOperatorChain::buildOperatorSnapshotFutures
RegularOperatorChain::checkpointStreamOperator
AbstractStreamOperator::snapshotState
StreamOperatorStateHandler::snapshotState//调用Operator/Keyed Backend的snapshot
StateSnapshotContextSynchronousImpl::new
AbstractUdfStreamOperator::snapshotState //调用UDF中snapshotState方法，一般用于更新OperatorState
DefaultOperatorStateBackend::snapshot
SnapshotStrategyRunner::snapshot
DefaultOperatorStateBackendSnapshotStrategy::syncPrepareResources//深copy operator state,便于后续进行异步快照
DefaultOperatorStateBackendSnapshotStrategy::asyncSnapshot//异步快照 CheckpointStateOutputStream::closeAndGetHandle
OperatorStreamStateHandle::new //包装元信息及数据StreamStateHandle
HeapKeyedStateBackend::snapshot
SnapshotStrategyRunner::snapshot
HeapSnapshotStrategy::syncPrepareResources
HeapSnapshotStrategy::asyncSnapshot //采用COWSateTable异步快照
CheckpointStateOutputStream::closeAndGetHandle
KeyGroupsStateHandle::new //包装KeyGroup及数据StreamStateHandle
SubtaskCheckpointCoordinatorImpl::finishAndReportAsync //向JobMaster发送checkpoint的结果
AsyncCheckpointRunnable::new
AsyncCheckpointRunnable::run
AsyncCheckpointRunnable::finalizeNonFinishedSnapshots
OperatorSnapshotFinalizer::new //等待TaskSnapshot状态信息序列化完成
AsyncCheckpointRunnable::reportCompletedSnapshotStates
TaskStateManagerImpl::reportTaskStateSnapshots
RpcCheckpointResponder::acknowledgeCheckpoint//向JobMaster发送Ack，带上State信息
非SourceTask
在StreamTask启动后调用StreamTask::processInput不断读取数据进行处理， 非SourceTask在收到上游的CheckpointBarrier对齐后触发Checkpoint,
StreamTask::processInput
StreamOneInputProcessor::processInput
StreamTaskNetworkInput::emitNext(StreamTaskNetworkOutput)
AbstractStreamTaskNetworkInput::emitNext //循环不断从buffer中读取StreamElement
处理
CheckpointedInputGate::pollNext
CheckpointedInputGate::handleEvent
SingleCheckpointBarrierHandler::processBarrier
SingleCheckpointBarrierHandler::markCheckpointAlignedAndTransformState
WaitingForFirstBarrier::barrierReceived
AbstractAlignedBarrierHandlerState::barrierReceived
SingleCheckpointBarrierHandler.ControllerImpl::allBarriersReceived//判断对齐
AbstractAlignedBarrierHandlerState::triggerGlobalCheckpoint
SingleCheckpointBarrierHandler.ControllerImpl::triggerGlobalCheckpoint
SingleCheckpointBarrierHandler::triggerCheckpoint
CheckpointBarrierHandler::notifyCheckpoint //触发StreamTask Checkpoint
StreamTask::triggerCheckpointOnBarrier
StreamTask::performCheckpoint //后续调用过程与SourceTask一样
SubtaskCheckpointCoordinatorImpl::checkpointState
根据调用栈看出，非SourceStreamTask执行Checkpoint只是触发时机不同，SourceTask由JobMaster RPC定时不断触发，非SourceTask则是在上游的CheckpointBarrier对齐后触发Checkpoint，最终执行逻辑都是将当前算子的信息写入CheckpointStorage后向JobMaster发送确认信息。
StreamTask向JobMaster ACK信息中包含状态元信息及StreamStateHandle，根据状态存储位置分为：
- ByteStreamStateHandle //对应JobManagerCheckpointStorage,将状态序列化为byte[]发送给JobMaster
- FileStateHandle //对应FileSystemCheckpointStorage,将状态写入文件系统后将文件路径发送给JobMaster
JobMaster端完成流程
JobMaster收到StreamTask的acknowledgeCheckpoint后：
JobMaster::acknowledgeCheckpoint
SchedulerBase::acknowledgeCheckpoint
ExecutionGraphHandler::acknowledgeCheckpoint
CheckpointCoordinator::receiveAcknowledgeMessage
PendingCheckpoint::acknowledgeTask //某一个Task的确认
PendingCheckpoint::updateOperatorState//更新SubTask状态信息
CheckpointCoordinator::completePendingCheckpoint//所有Task Ack后
PendingCheckpoint::finalizeCheckpoint
Checkpoints.storeCheckpointMetadata//保存CheckpointMetadata
CompletedCheckpoint::new
CheckpointCoordinator::sendAcknowledgeMessages//向Task通知Checkpoint完成消息
ExecutionVertex::notifyCheckpointComplete
TaskManagerGateway.notifyCheckpointComplete
JobMaster收到所有StreamTask的Checkpoint状态信息后，标志一次Checkpoint完成，这时会通知StreamTask CheckPoint完成消息，便于SubTask监听Checkpoint完成后做后续动作。