Spark任务调度

1.SparkSubmit 提交应用

2.调用Client来RequestSubmitDriver

3.Master launchDriver

4.DriverRunner 在work上启动

5.DriverRunner运行DriverWrapper

6.DriverWrapper 在work上启动

7.DriverWrapper运行提交的应用的main方法

8.SparkContext 创建

9.TaskSchedulerImpl启动,  SparkDeploySchedulerBackend CoarseGrainedSchedulerBackend启动

10.SparkDeploySchedulerBackend 调用AppClient, 传递了封装了CoarseGrainedExecutorBackend的ApplicationDescription

11.AppClient向Master注册应用

12.Master从获得的ApplicationDescription创建出 ApplicationInfo, 调度到Work上运行

13.Work启动ExecutorRunner

14.ExecutorRunner运行CoarseGrainedExecutorBackend

15.CoarseGrainedExecutorBackend调用RegisterExecutor, 向CoarseGrainedSchedulerBackend注册

16.SparkContext runjob 

17.DAGScheduler 调runjob  submitJob  handleJobSubmitted  submitStage  submitMissingTasks

18.TaskSchedulerImpl submitTasks

19.CoarseGrainedSchedulerBackend ---> reviveOffers --->  makeOffers  ---> launchTasks

20.CoarseGrainedExecutorBackend  LaunchTask执行

21.Executor LaunchTask

22.TaskRunner  run

广告:
欢迎加入Spark中文社区