Flink JAR包上传和运行逻辑

文章目录

    • 说明
    • 启动ResetServer
    • 注册Handler
    • Upload JAR
    • Run Jar
      • 生成JobGraph的过程
      • 调用用户程序main方法
      • 执行用户程序main方法
      • 执行execute (和接触过一个概念很类似-打桩测试)
    • 提交JobGraph
    • ExectionGraph Deploy的过程

说明

  1. 目标:走读Flink Clint中Upload jar、Run jar相关代码
  2. 源码版本:1.6.1
  3. 部属模式:Standalone
  4. 相关知识点:Netty、 CompletedFuture

启动ResetServer

RestServerEndpoint.start

注册Handler

代码From DispatcherRestEndpoint.java

protected List<Tuple2<RestHandlerSpecification, ChannelInboundHandler>> initializeHandlers(CompletableFuture<String> restAddressFuture) {
		List<Tuple2<RestHandlerSpecification, ChannelInboundHandler>> handlers = super.initializeHandlers(restAddressFuture);
    ...

	JobSubmitHandler jobSubmitHandler = new JobSubmitHandler(
		restAddressFuture,
		leaderRetriever,
		timeout,
		responseHeaders,
		executor,
		clusterConfiguration);

	if (clusterConfiguration.getBoolean(WebOptions.SUBMIT_ENABLE)) {
		try {
		    // 此处注册了JAR Upload和Run的处理方法
			webSubmissionExtension = WebMonitorUtils.loadWebSubmissionExtension(
				leaderRetriever,
				restAddressFuture,
				timeout,
				responseHeaders,
				uploadDir,
				executor,
				clusterConfiguration);

			// register extension handlers
			handlers.addAll(webSubmissionExtension.getHandlers());
		} catch (FlinkException e) {
		...
		}
	} else {
		log.info("Web-based job submission is not enabled.");
	}

    ...

	return handlers;
}

在WebSubmissionExtension中,可以看到定义了Upload、Run、List、Delete、Plan的Handler

Upload JAR

处理代码在JarUploadHandler的handleRequest方法中。

Jar包存放路径:

jarDir.resolve(UUID.randomUUID() + "_" + fileUpload.getFileName());

方法本身逻辑简单,比较隐蔽的是jarDir的值。通过倒推寻找该值的赋值过程。

  1. JarUploadHandler 构造时赋值属性jarDir;
  2. JarUploadHandler由WebSubmissionExtension通过WebMonitorUtils.loadWebSubmissionExtension构造,jarDir源自父类RestServerEndpoint中的变量uploadDir;
  3. RestServerEndpoint中uploadDir通过configuration.getUploadDir()初始化
  4. 在RestServerEndpointConfiguration中找到了源头:
    final Path uploadDir = Paths.get(
    	config.getString(WebOptions.UPLOAD_DIR,	config.getString(WebOptions.TMP_DIR)),
    	"flink-web-upload");
    

一般情况下,大家都不会改写配置项WebOption.UPLOAD_DIR(对应配置项“web.upload.dir”),所以JAR包存放到了"$WebOptions.TMP_DIR/flink-web-upload"

WebOptions.TMP_DIR的赋值比较隐蔽,只从配置文件看,是在/tmp目录。但是在ClusterEntrypoint的generateClusterConfiguration中,其实对该值进行了改写:

final String webTmpDir = configuration.getString(WebOptions.TMP_DIR);
final File uniqueWebTmpDir = new File(webTmpDir, "flink-web-" + UUID.randomUUID());

resultConfiguration.setString(WebOptions.TMP_DIR, uniqueWebTmpDir.getAbsolutePath());

最终的效果JAR包存放目录是"/tmp/flink-web-UUID/flink-web-upload"

存放在tmp目录里面是有风险的,过期后会被删除。

Run Jar

同上,重点关注JarRunHandler的handleRequest

@Override
protected CompletableFuture<JarRunResponseBody> handleRequest(
		@Nonnull final HandlerRequest<JarRunRequestBody, JarRunMessageParameters> request,
		@Nonnull final DispatcherGateway gateway) throws RestHandlerException {
    ...

    # 产生JobGraph
	final CompletableFuture<JobGraph> jobGraphFuture = getJobGraphAsync(
		jarFile,
		entryClass,
		programArgs,
		savepointRestoreSettings,
		parallelism);

	CompletableFuture<Integer> blobServerPortFuture = gateway.getBlobServerPort(timeout);

    # Jar上传JobGraph,UserJar和UserArtifact
	CompletableFuture<JobGraph> jarUploadFuture = jobGraphFuture.thenCombine(blobServerPortFuture, (jobGraph, blobServerPort) -> {
		final InetSocketAddress address = new InetSocketAddress(gateway.getHostname(), blobServerPort);
		try {
			ClientUtils.extractAndUploadJobGraphFiles(jobGraph, () -> new BlobClient(address, configuration));
		} catch (FlinkException e) {
			throw new CompletionException(e);
		}

		return jobGraph;
	});

	CompletableFuture<Acknowledge> jobSubmissionFuture = jarUploadFuture.thenCompose(jobGraph -> {
		// we have to enable queued scheduling because slots will be allocated lazily
		jobGraph.setAllowQueuedScheduling(true);
		# 提交Job
		return gateway.submitJob(jobGraph, timeout);
	});

	return jobSubmissionFuture
		.thenCombine(jarUploadFuture, (ack, jobGraph) -> new JarRunResponseBody(jobGraph.getJobID()))
		.exceptionally(throwable -> {
			throw new CompletionException(new RestHandlerException(
				throwable.getMessage(),
				HttpResponseStatus.INTERNAL_SERVER_ERROR,
				throwable));
		});
}

生成JobGraph的过程

/* 在JarRunHandler的getJobGraphAsync中构造了PackagedProgram */
final PackagedProgram packagedProgram = new PackagedProgram(
		jarFile.toFile(),
		entryClass,
		programArgs.toArray(new String[programArgs.size()]));
		jobGraph = PackagedProgramUtils.createJobGraph(packagedProgram, configuration, parallelism);
/* From PackagedProgramUtils.java */
public static JobGraph createJobGraph(
	PackagedProgram packagedProgram,
	Configuration configuration,
	int defaultParallelism) throws ProgramInvocationException {
    ....

	if (packagedProgram.isUsingProgramEntryPoint()) {
		...
	} else if (packagedProgram.isUsingInteractiveMode()) {
	    /* 一般提交的流程序会走这个分支,判断原则是用户程序的main Class是否isAssignableFrom ProgramDescription */
		final OptimizerPlanEnvironment optimizerPlanEnvironment = new OptimizerPlanEnvironment(optimizer);

		optimizerPlanEnvironment.setParallelism(defaultParallelism);

        // 会触发main函数调用
		flinkPlan = optimizerPlanEnvironment.getOptimizedPlan(packagedProgram);
	} else {
		throw new ProgramInvocationException("PackagedProgram does not have a valid invocation mode.");
	}

	if (flinkPlan instanceof StreamingPlan) {
	    // 获取JobGraph
		jobGraph = ((StreamingPlan) flinkPlan).getJobGraph();
		jobGraph.setSavepointRestoreSettings(packagedProgram.getSavepointSettings());
	} else {
		...
	}

    ...

	return jobGraph;
}

调用用户程序main方法

/* From OptimizerPlanEnvironment.java */
public FlinkPlan getOptimizedPlan(PackagedProgram prog) throws ProgramInvocationException {
    ...
    
    /* 设置ContextEnviormentFacoty对应的env为OptimizerPlanEnvironment */
	setAsContext();
	try {
	    /* 调用用户程序main方法 */
		prog.invokeInteractiveModeForExecution();
	}
	...
}

执行用户程序main方法

// 一个常见的main 结构
public static void main(String[] args) throws Exception {
    /* 此处获取的是上一步setAsContext中设置的OptimizerPlanEnvironment */
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    ...
    /* 对应的是执行OptimizerPlanEnvironment的execute */
    env.execute();
}

执行execute (和接触过一个概念很类似-打桩测试)

public JobExecutionResult execute(String jobName) throws Exception {
    /* 反馈Compile后的FlinkPlan */
	Plan plan = createProgramPlan(jobName);
	this.optimizerPlan = compiler.compile(plan);

    // execute后不要带其他的用户程序
	// do not go on with anything now!
	throw new ProgramAbortException();
}

提交JobGraph

OK,已经得到了JobGraph,再细看提交JobGraph的过程

/* From Dispatcher.java */
public CompletableFuture<Acknowledge> submitJob(JobGraph jobGraph, Time timeout) {

	...

	if (jobSchedulingStatus == RunningJobsRegistry.JobSchedulingStatus.DONE || jobManagerRunnerFutures.containsKey(jobId)) {
		return FutureUtils.completedExceptionally(
			new JobSubmissionException(jobId, String.format("Job has already been submitted and is in state %s.", jobSchedulingStatus)));
	} else {
	    //重点关注persistAndRunJob
		final CompletableFuture<Acknowledge> persistAndRunFuture = waitForTerminatingJobManager(jobId, jobGraph, this::persistAndRunJob)
			.thenApply(ignored -> Acknowledge.get());

		return persistAndRunFuture.exceptionally(
			(Throwable throwable) -> {
				final Throwable strippedThrowable = ExceptionUtils.stripCompletionException(throwable);
				log.error("Failed to submit job {}.", jobId, strippedThrowable);
				throw new CompletionException(
					new JobSubmissionException(jobId, "Failed to submit job.", strippedThrowable));
			});
	}
}

省略一些方法间调用,调用顺序如下:

  1. Dispatch.persistAndRunJob
  2. Dispatch.runJob
  3. Dispatch.createJobManagerRunner,创建JobMaster
  4. JobMaster.createAndRestoreExecutionGraph
    终于看到了ExecutionGraph

ExectionGraph Deploy的过程

方法间调用关系:

  1. 上接Dispatcher.createJobManagerRunner
  2. Dispatcher.startJobManagerRunner
  3. JobManagerRunner.start
  4. StandaloneLeaderElectionService.start
  5. JobManagerRunner.grantLeadership
  6. JobManagerRunner.verifyJobSchedulingStatusAndStartJobManager
  7. JobMaster.start
  8. JobMaster.startJobExecution
  9. JobMaster.resetAndScheduleExecutionGraph
  10. JobMaster.scheduleExecutionGraph
  11. ExecutionGraph.scheduleForExecution
  12. ExecutionGraph.scheduleEager
  13. Execution.deploy

你可能感兴趣的:(Flink,flink,execution,graph,job,graph,client)