spark源码----Spark任务划分、调度、执行
从RDD的创建开始讲起
把它当做入口,然后点进去
主要关注hadoopFile,进去会发现new了一个HadoopRDD
以上其实就是一个RDD的构建过程
又比如我们看flatMap,它一样会去构建一个新的RDD,把之前的RDD给传进去了
又比如我们看map,它一样会去构建一个新的RDD,把之前的RDD给传进去了
在换一个算子reduceByKey,点进去,包含一个默认的分区器
然后再点进去,combineByKeyWithClassTag是一个预聚合概念,所以reduceByKey会先在本地做一次聚合,所以它的效率比groupByKey的效率高
再进去,你会看到最终还是一个RDD
Shuffled会有一个依赖的,所以我们可以进去看看
而map这种,最后进去,它是一对一的依赖
所以reduceByKey这种算子,就叫做宽依赖,map这种就叫窄依赖
宽窄看完后,在看看我们的active算子,以foreach为例(主要是看其任务是怎么一步步的提交的)
点击runJob进去
这里面会不断的去调用runJob,所以我们需要一直点进去,直到没有runJob
如上图由dagScheduler来进行我们的作业
再点runJob进去
def runJob[T, U](
rdd: RDD[T],
func: (TaskContext, Iterator[T]) => U,
partitions: Seq[Int],
callSite: CallSite,
resultHandler: (Int, U) => Unit,
properties: Properties): Unit = {
val start = System.nanoTime
// 提交我们的作业
val waiter = submitJob(rdd, func, partitions, callSite, resultHandler, properties)
// 后面都是在等待我们的执行
ThreadUtils.awaitReady(waiter.completionFuture, Duration.Inf)
waiter.completionFuture.value.get match {
case scala.util.Success(_) =>
logInfo("Job %d finished: %s, took %f s".format
(waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
case scala.util.Failure(exception) =>
logInfo("Job %d failed: %s, took %f s".format
(waiter.jobId, callSite.shortForm, (System.nanoTime - start) / 1e9))
// SPARK-8644: Include user stack trace in exceptions coming from DAGScheduler.
val callerStackTrace = Thread.currentThread().getStackTrace.tail
exception.setStackTrace(exception.getStackTrace ++ callerStackTrace)
throw exception
}
}
在从submitJob进去
看到这儿会看到是一直往eventProcessLoop里面放东西,那么再进去看看
再进去
再点,这个BlockingQueue是一个阻塞式队列(放满了阻塞,取没有数据也阻塞)
private[spark] abstract class EventLoop[E](name: String) extends Logging {
private val eventQueue: BlockingQueue[E] = new LinkedBlockingDeque[E]()
private val stopped = new AtomicBoolean(false)
private val eventThread = new Thread(name) {
setDaemon(true)
override def run(): Unit = {
try {
while (!stopped.get) {
val event = eventQueue.take()
try {
// 发了消息,接收
onReceive(event)
} catch {
case NonFatal(e) =>
try {
onError(e)
} catch {
case NonFatal(e) => logError("Unexpected error in " + name, e)
}
}
}
}
.....
看一下onReceive是怎么处理的,会发现它是一个抽象的方法,那么需要去找具体的实现
JobSubmitted 这个就是刚刚看到的提交的东西,进去看看handleJobSubmitted,下面这一行代码的意思就是创建一个结果的阶段
再进去,里面一定有一个完成阶段的结果阶段,而val parents = getOrCreateParentStages(rdd, jobId)表示上一级的阶段,有还是没有
进去看看,红框,判断有没有shuffle的依赖
在点进去看看
// 其实就是在从后往前查找shuffle再哪儿
private[scheduler] def getShuffleDependencies(
rdd: RDD[_]): HashSet[ShuffleDependency[_, _, _]] = {
// 这个就是最终的返回
val parents = new HashSet[ShuffleDependency[_, _, _]]
val visited = new HashSet[RDD[_]]
val waitingForVisit = new Stack[RDD[_]]
waitingForVisit.push(rdd) // 往栈里放值
while (waitingForVisit.nonEmpty) {
val toVisit = waitingForVisit.pop() // 判断之后取出 toVisit 就是当前的这个RDD
if (!visited(toVisit)) {
visited += toVisit
toVisit.dependencies.foreach { // 看一下当前访问的RDD的依赖里面有没有shuffle的依赖
// 在这里赋值的
case shuffleDep: ShuffleDependency[_, _, _] =>
parents += shuffleDep
case dependency => // 没有的情况下是把当前RDD依赖的那个RDD给放进去
waitingForVisit.push(dependency.rdd)
}
}
}
parents // 比如wordCount只有一个shuffle,所以就只会返回一个
}
再回去
private def getOrCreateParentStages(rdd: RDD[_], firstJobId: Int): List[Stage] = {
// shuffle依赖拿到之后,再根据shuffle来拿到和转化stage
getShuffleDependencies(rdd).map { shuffleDep =>
getOrCreateShuffleMapStage(shuffleDep, firstJobId)
}.toList
}
到这儿的话val parents = getOrCreateParentStages(rdd, jobId)就已经完成了
然后把这个放到了val stage = new ResultStage(id, rdd, func, partitions, parents, jobId, callSite)这里面去
所以,我们再去看handleJobSubmitted这个方法
private[scheduler] def handleJobSubmitted(jobId: Int,
finalRDD: RDD[_],
func: (TaskContext, Iterator[_]) => _,
partitions: Array[Int],
callSite: CallSite,
listener: JobListener,
properties: Properties) {
var finalStage: ResultStage = null
try {
// New stage creation may throw an exception if, for example, jobs are run on a
// HadoopRDD whose underlying HDFS files have been deleted.
finalStage = createResultStage(finalRDD, func, partitions, jobId, callSite)
} catch {
case e: Exception =>
logWarning("Creating new stage failed due to exception - job: " + jobId, e)
listener.jobFailed(e)
return
}
// 从这里就可以看出,我们的job里面包含了stage,虽然只有一个,但是finalStage里面又包含了parents
val job = new ActiveJob(jobId, finalStage, callSite, listener, properties)
clearCacheLocs()
logInfo("Got job %s (%s) with %d output partitions".format(
job.jobId, callSite.shortForm, partitions.length))
logInfo("Final stage: " + finalStage + " (" + finalStage.name + ")")
logInfo("Parents of final stage: " + finalStage.parents)
logInfo("Missing parents: " + getMissingParentStages(finalStage))
val jobSubmissionTime = clock.getTimeMillis()
jobIdToActiveJob(jobId) = job
activeJobs += job
finalStage.setActiveJob(job)
val stageIds = jobIdToStageIds(jobId).toArray
val stageInfos = stageIds.flatMap(id => stageIdToStage.get(id).map(_.latestInfo))
listenerBus.post(
SparkListenerJobStart(job.jobId, jobSubmissionTime, stageInfos, properties))
// 阶段划分完了之后,提交 finalStage这个是最终的那个阶段
submitStage(finalStage)
}
到submitStage里面去看看
// stage 这个就是我们最终的那个阶段
private def submitStage(stage: Stage) {
val jobId = activeJobForStage(stage)
if (jobId.isDefined) {
logDebug("submitStage(" + stage + ")")
if (!waitingStages(stage) && !runningStages(stage) && !failedStages(stage)) {
// 获取丢失的ParentStages
val missing = getMissingParentStages(stage).sortBy(_.id)
logDebug("missing: " + missing)
// 如果没有上级
if (missing.isEmpty) {
logInfo("Submitting " + stage + " (" + stage.rdd + "), which has no missing parents")
// 这里一定是先提交第一个阶段或者前面的阶段先提交
submitMissingTasks(stage, jobId.get) //这是最终跳出去的地方 也是提交任务的地方
} else {
// 然后又把上级挨个遍历 然后做了个递归操作
for (parent <- missing) {
submitStage(parent)
}
waitingStages += stage
}
}
} else {
abortStage(stage, "No active job for stage " + stage.id, None)
}
}
到getMissingParentStages这里面去看看, 把当前阶段的最终的RDD放进去,当不为空的时候,放到visit里面去
其实就是在判断有没有上一级的阶段
到submitMissingTasks去看看,发现其中只有两种stage
看到这儿,你会发现它会判断你的阶段到底是符合哪种类型的,也只有两种类型的阶段,然后每一种类型的阶段,都会变成对应的task,task的个数又取决于partitionsToCompute,顾名思义就是我们的计算分区
如果task不是空的,就说明我们有任务,那么就提交我们的任务 先转化成TaskSet,然后提交
到submitTasks去看,发现是一个抽象的,那么我们再去搜索一下,在TaskSchedulerImpl这里面
override def submitTasks(taskSet: TaskSet) {
val tasks = taskSet.tasks
logInfo("Adding task set " + taskSet.id + " with " + tasks.length + " tasks")
this.synchronized {
// 创建一个TaskSetManager来管理和调度这个taskSet中的task
val manager = createTaskSetManager(taskSet, maxTaskFailures)
val stage = taskSet.stageId
val stageTaskSets =
taskSetsByStageIdAndAttempt.getOrElseUpdate(stage, new HashMap[Int, TaskSetManager])
stageTaskSets(taskSet.stageAttemptId) = manager
val conflictingTaskSet = stageTaskSets.exists { case (_, ts) =>
ts.taskSet != taskSet && !ts.isZombie
}
if (conflictingTaskSet) {
throw new IllegalStateException(s"more than one active taskSet for stage $stage:" +
s" ${stageTaskSets.toSeq.map{_._2.taskSet.id}.mkString(",")}")
}
// 然后放到TaskSetManagerPoll中去(这句话的最终意思可以到addTaskSetManager中去看看,又是一个抽象的方法,再去找找,最后会看到rootPool.addSchedulable(manager),之所以要放到池子里面去是因为,不能保证我们的任务好了之后execute已经准备好了,等有资源之后在发送任务,否则来一个发一个,资源都没有,根本没法发)
schedulableBuilder.addTaskSetManager(manager, manager.taskSet.properties)
if (!isLocal && !hasReceivedTask) {
starvationTimer.scheduleAtFixedRate(new TimerTask() {
override def run() {
if (!hasLaunchedTask) {
logWarning("Initial job has not accepted any resources; " +
"check your cluster UI to ensure that workers are registered " +
"and have sufficient resources")
} else {
this.cancel()
}
}
}, STARVATION_TIMEOUT_MS, STARVATION_TIMEOUT_MS)
}
hasReceivedTask = true
}
backend.reviveOffers()
}
然后到reviveOffers去看看,又是抽象,去找下具体实现,在这里会作为一条信息接收
再点进去看看,到这里就会从我们的任务池中去取我们的任务,如果去到就往下执行
private def launchTasks(tasks: Seq[Seq[TaskDescription]]) {
// 取出我们的任务
for (task <- tasks.flatten) {
// 序列化 因为是从driver发到execute中的,所以需要序列化
val serializedTask = TaskDescription.encode(task)
if (serializedTask.limit >= maxRpcMessageSize) {
scheduler.taskIdToTaskSetManager.get(task.taskId).foreach { taskSetMgr =>
try {
var msg = "Serialized task %s:%d was %d bytes, which exceeds max allowed: " +
"spark.rpc.message.maxSize (%d bytes). Consider increasing " +
"spark.rpc.message.maxSize or using broadcast variables for large values."
msg = msg.format(task.taskId, task.index, serializedTask.limit, maxRpcMessageSize)
taskSetMgr.abort(msg)
} catch {
case e: Exception => logError("Exception in error callback", e)
}
}
}
else {
val executorData = executorDataMap(task.executorId)
executorData.freeCores -= scheduler.CPUS_PER_TASK
logDebug(s"Launching task ${task.taskId} on executor id: ${task.executorId} hostname: " +
s"${executorData.executorHost}.")
// 这里就是driver的终端和execute的终端需要发送消息了 发送的消息是启动的任务
executorData.executorEndpoint.send(LaunchTask(new SerializableBuffer(serializedTask)))
}
}
}
再去看看我们是怎么接收上面的消息的,名字叫CoarseGrainedExecutorBackend
override def receive: PartialFunction[Any, Unit] = {
case RegisteredExecutor =>
logInfo("Successfully registered with driver")
try {
executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
} catch {
case NonFatal(e) =>
exitExecutor(1, "Unable to create executor due to " + e.getMessage, e)
}
case RegisterExecutorFailed(message) =>
exitExecutor(1, "Slave registration failed: " + message)
// 这就是刚刚发送的任务
case LaunchTask(data) =>
if (executor == null) {
exitExecutor(1, "Received LaunchTask command but executor was null")
} else {
// 反序列化
val taskDesc = TaskDescription.decode(data.value)
logInfo("Got assigned task " + taskDesc.taskId)
// 然后发送到计算对象,开始准备计算了
executor.launchTask(this, taskDesc)
}
case KillTask(taskId, _, interruptThread, reason) =>
if (executor == null) {
exitExecutor(1, "Received KillTask command but executor was null")
} else {
executor.killTask(taskId, interruptThread, reason)
}
case StopExecutor =>
stopping.set(true)
logInfo("Driver commanded a shutdown")
// Cannot shutdown here because an ack may need to be sent back to the caller. So send
// a message to self to actually do the shutdown.
self.send(Shutdown)
case Shutdown =>
stopping.set(true)
new Thread("CoarseGrainedExecutorBackend-stop-executor") {
override def run(): Unit = {
// executor.stop() will call `SparkEnv.stop()` which waits until RpcEnv stops totally.
// However, if `executor.stop()` runs in some thread of RpcEnv, RpcEnv won't be able to
// stop until `executor.stop()` returns, which becomes a dead-lock (See SPARK-14180).
// Therefore, we put this line in a new thread.
executor.stop()
}
}.start()
}
以上就是源码层次的任务划分、调度、执行