Flink源码学习二 用户代码逻辑计划生成 1.17分支
1、Streaming环境获取
接上节flink调用用户代码的main方法后，用户代码中一般会有如下获取flink环境的代码
val env = StreamExecutionEnvironment.getExecutionEnvironment这里获取的环境再命令行客户的执行这个方法是就已经初始化好了，这个env里面封装了一些用户环境配置、streaming执行配置等。
//org/apache/flink/client/ClientUtils.java:66
public static void executeProgram(
PipelineExecutorServiceLoader executorServiceLoader,
Configuration configuration,
PackagedProgram program,
boolean enforceSingleJobExecution,
boolean suppressSysout)
throws ProgramInvocationException {
checkNotNull(executorServiceLoader);
final ClassLoader userCodeClassLoader = program.getUserCodeClassLoader();
final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
try {
Thread.currentThread().setContextClassLoader(userCodeClassLoader);
LOG.info(
"Starting program (detached: {})",
!configuration.getBoolean(DeploymentOptions.ATTACHED));
ContextEnvironment.setAsContext(
executorServiceLoader,
configuration,
userCodeClassLoader,
enforceSingleJobExecution,
suppressSysout);
StreamContextEnvironment.setAsContext(
executorServiceLoader,
configuration,
userCodeClassLoader,
enforceSingleJobExecution,
suppressSysout);
try {
program.invokeInteractiveModeForExecution();
} finally {
ContextEnvironment.unsetAsContext();
StreamContextEnvironment.unsetAsContext();
}
} finally {
Thread.currentThread().setContextClassLoader(contextClassLoader);
}
}2、DataStreamSource生成
DataStream是flink关于streaming的最核心抽象，它是从StreamExecutionEnvironment的addSource方法和fromSource两个方法生成。其中fromSource是新的api，下面分别进行介绍。
addSource方法会触发以下这段逻辑：
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:1933
private DataStreamSource addSource(
final SourceFunction function,
final String sourceName,
@Nullable final TypeInformation typeInfo,
final Boundedness boundedness) {
checkNotNull(function);
checkNotNull(sourceName);
checkNotNull(boundedness);
TypeInformation resolvedTypeInfo =
getTypeInfo(function, sourceName, SourceFunction.class, typeInfo);
boolean isParallel = function instanceof ParallelSourceFunction;
clean(function);
final StreamSource sourceOperator = new StreamSource<>(function);
return new DataStreamSource<>(
this, resolvedTypeInfo, sourceOperator, isParallel, sourceName, boundedness);
} 这里输入的SourceFunction就是用户代码自定义实现或调用数据源函数，如env.addSource(new FlinkKafkaConsumer11）
总结上述方法逻辑：
- 根据用户传入的SourceFunction获取source的输出类型的TypeInformation封装。
- flink会将所有streaming的输入输出类型封装成TypeInformation，其中基础类型数组类型的实现是BasicTypeInfo、BasicArrayTypeInfo，用户自定义的pojo类型会封装到GenericTypeInfo。
- 把SourceFunction封装到sourceOperator中，这里是把SourceFunction封装成StreamSource。StreamSource是一个StreamOperator算子抽象，在flink中一个DataSream封装了一次数据流转换，一个StreamOperator封装了一个函数接口，如map、filter、sourcefunction等。可以看到StreamSource走的是如下一套接口体系，这命名很随意。。。
其中用户的sourcefunction是封装在了AbstractUdfStreamOperator中
//org/apache/flink/streaming/api/operators/AbstractUdfStreamOperator.java:54
/** The user function. */
protected final F userFunction;
public AbstractUdfStreamOperator(F userFunction) {
this.userFunction = requireNonNull(userFunction);
checkUdfCheckpointingPreconditions();
}- 生成DataStreamSource数据流。DataStreamSource的具体继承关系如下：
//org/apache/flink/streaming/api/datastream/DataStreamSource.java:57
/** The constructor used to create legacy sources. */
public DataStreamSource(
StreamExecutionEnvironment environment,
TypeInformation outTypeInfo,
StreamSource operator,
boolean isParallel,
String sourceName,
Boundedness boundedness) {
super(
environment,
new LegacySourceTransformation<>(
sourceName,
operator,
outTypeInfo,
environment.getParallelism(),
boundedness));
this.isParallel = isParallel;
if (!isParallel) {
setParallelism(1);
}
} - 这里调用DataStreamSource的构造方法，注意这里的LegacySourceTransformation，它就是一次数据流转换的抽象，它封装了前面的StreamSource算子操作和流的并行度，它的继承体系如下：
- 最后调用了父类DataStream的构造函数，把一次流转换操作封装到DataStream中。其具体代码如下：
//org/apache/flink/streaming/api/datastream/DataStream.java:129
protected final StreamExecutionEnvironment environment;
protected final Transformation transformation;
/**
* Create a new {@link DataStream} in the given execution environment with partitioning set to
* forward by default.
*
* @param environment The StreamExecutionEnvironment
*/
public DataStream(StreamExecutionEnvironment environment, Transformation transformation) {
this.environment =
Preconditions.checkNotNull(environment, "Execution Environment must not be null.");
this.transformation =
Preconditions.checkNotNull(
transformation, "Stream Transformation must not be null.");
} - 至此DataStream生成，后续的数据流转换操作都在DataStream上面完成了。
freomSource方法会触发以下这段逻辑：
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:1999
public DataStreamSource fromSource(
Source source,
WatermarkStrategy timestampsAndWatermarks,
String sourceName,
TypeInformation typeInfo) {
final TypeInformation resolvedTypeInfo =
getTypeInfo(source, sourceName, Source.class, typeInfo);
return new DataStreamSource<>(
this,
checkNotNull(source, "source"),
checkNotNull(timestampsAndWatermarks, "timestampsAndWatermarks"),
checkNotNull(resolvedTypeInfo),
checkNotNull(sourceName));
} fromSource是flink新的数据源生成方法，这里用户生成Souece和原来的SourceFunction不一样，走的是另外一套继承接口。
总结上述逻辑：
- 获取source的输出类型的TypeInformation封装
- 构建DataStreamSource流
- 把source和水位线等标记封装到SourceTransformation中
- 构建DataStream，把SourceTransformation赋值给这个流的Transformation
3、DataStream的转换操作
上一步从数据源中获取DataStram的抽象DataStreamSource后，后面所有的转换都会在DataStram上进行。
注意到StreamExecutionEnvironment在初始化的时候会构建一个List
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:190
protected final List> transformations = new ArrayList<>(); 每一次的DataStream流转换操作都会把当前的Transformation添加到算子列表 transformations 中（只有 转换 transform 转换操作才会添加算子，其它都只是暂时做了 transformation 的叠加封装）
下面以map函数 为例来介绍一次流转换操作。
//org/apache/flink/streaming/api/datastream/DataStream.java:592
public SingleOutputStreamOperator map(
MapFunction mapper, TypeInformation outputType) {
return transform("Map", outputType, new StreamMap<>(clean(mapper)));
} - 可以看到一次map操作会触发transform函数
//org/apache/flink/streaming/api/datastream/DataStream.java:1180
public SingleOutputStreamOperator transform(
String operatorName,
TypeInformation outTypeInfo,
OneInputStreamOperatorFactory operatorFactory) {
return doTransform(operatorName, outTypeInfo, operatorFactory);
}
//org/apache/flink/streaming/api/datastream/DataStream.java:1188
protected SingleOutputStreamOperator doTransform(
String operatorName,
TypeInformation outTypeInfo,
StreamOperatorFactory operatorFactory) {
// read the output type of the input Transform to coax out errors about MissingTypeInfo
transformation.getOutputType();
OneInputTransformation resultTransform =
new OneInputTransformation<>(
this.transformation,
operatorName,
operatorFactory,
outTypeInfo,
environment.getParallelism());
@SuppressWarnings({"unchecked", "rawtypes"})
SingleOutputStreamOperator returnStream =
new SingleOutputStreamOperator(environment, resultTransform);
getExecutionEnvironment().addOperator(resultTransform);
return returnStream;
} - 可以看到transform会调用doTransform函数，总结其作用
- 调用当前DataStram封装的transformation，获取它的输出类型，用来触发缺少输出类型的错误。
- 将当前DataSteam的transformation和需要进行操作的算子转换成新的一次输入转换操作。这里的OneInputTransformation也是Transformation的抽象。它的两个泛型分别表示当前转换的输入和转换后的输出类型。
- 最后将当前环境对象和新的transformation对象最为成员变量封装成另外一个新的DataStream对象并返回。这里注意只有需要transform的流才会生成新的DataStream算子。
- 注意到这里的getExecutionEnvironment().addOperator(resultTransform) 操作，flink会将所有的transformation操作维护起来。
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:2325
public void addOperator(Transformation transformation) {
Preconditions.checkNotNull(transformation, "transformation must not be null.");
this.transformations.add(transformation);
}- 由此可见，用户在DataStream上面执行的一系列操作，如map、filter等，实际上是在DataStream上做的转换，由flink将这些transformation操作维护了起来，一直到最后执行env.execute()方法StramGraph的构建才真正开始。
3、逻辑计划StreamGraph的生成
用户执行env.execute()后会触发以下逻辑：
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:2041
public JobExecutionResult execute(String jobName) throws Exception {
final List> originalTransformations = new ArrayList<>(transformations);
StreamGraph streamGraph = getStreamGraph();
if (jobName != null) {
streamGraph.setJobName(jobName);
}
try {
return execute(streamGraph);
} catch (Throwable t) {
Optional clusterDatasetCorruptedException =
ExceptionUtils.findThrowable(t, ClusterDatasetCorruptedException.class);
if (!clusterDatasetCorruptedException.isPresent()) {
throw t;
}
// Retry without cache if it is caused by corrupted cluster dataset.
invalidateCacheTransformations(originalTransformations);
streamGraph = getStreamGraph(originalTransformations);
return execute(streamGraph);
}
} 这段代码主要调用getStreamGraph生成StreamGraph。
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:2237
private StreamGraph getStreamGraph(List> transformations) {
synchronizeClusterDatasetStatus();
return getStreamGraphGenerator(transformations).generate();
} 这里可以看到先生成了一个StreamGraphGenerator，再调用它的generate方法生成StreamGraph。主要是把transformations、执行配置、检查点配置和一些缓存信息作为成员变量封装了进去。具体的生成StreamGraph逻辑是下面这段代码：
//org/apache/flink/streaming/api/graph/StreamGraphGenerator.java:308
public StreamGraph generate() {
streamGraph = new StreamGraph(executionConfig, checkpointConfig, savepointRestoreSettings);
streamGraph.setEnableCheckpointsAfterTasksFinish(
configuration.get(
ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH));
shouldExecuteInBatchMode = shouldExecuteInBatchMode();
configureStreamGraph(streamGraph);
alreadyTransformed = new IdentityHashMap<>();
for (Transformation transformation : transformations) {
transform(transformation);
}
streamGraph.setSlotSharingGroupResource(slotSharingGroupResources);
setFineGrainedGlobalStreamExchangeMode(streamGraph);
for (StreamNode node : streamGraph.getStreamNodes()) {
if (node.getInEdges().stream().anyMatch(this::shouldDisableUnalignedCheckpointing)) {
for (StreamEdge edge : node.getInEdges()) {
edge.setSupportsUnalignedCheckpoints(false);
}
}
}
final StreamGraph builtStreamGraph = streamGraph;
alreadyTransformed.clear();
alreadyTransformed = null;
streamGraph = null;
return builtStreamGraph;
}总结上述代码逻辑：
- 构建一个StreamGraph对象并进行设置，主要是一些配置的设置和判断是流任务还是批任务，流批任务的生成逻辑不一样。
- 在 for (Transformation transformation : transformations) {
transform(transformation);
}
这段代码中对StreamExecutionEnvironment中管理的transformation转换成运行时的TransformationTranslator，然后根据这些转换信息生成StreamGraph的节点StreamNode，并创建节点连接。//org/apache/flink/streaming/api/graph/StreamGraphGenerator.java:187 static { @SuppressWarnings("rawtypes") Map- Transformation转换成TransformationTranslator，是一一对应的关系，所有的对应关系如上。
- 流任务具体的StreamNode是调用TransformationTranslator的translateForStreaming方法来生成，并添加到StreamGraph中的。这里以一次OneInputTransformation转换为例，它首先会转换成OneInputTransformationTranslator，继承关系如下。
- 这里再调用抽象父类的translateInternal方法进行实际的添加StramNode操作。
//org/apache/flink/streaming/runtime/translators/OneInputTransformationTranslator.java:63 @Override public CollectiontranslateForStreamingInternal( final OneInputTransformation - 最后会调用StreamGraph的addNode方法,这里StreamNode会保存算子的信息
//org/apache/flink/streaming/api/graph/StreamGraph.java:511 protected StreamNode addNode( Integer vertexID, @Nullable String slotSharingGroup, @Nullable String coLocationGroup, Class vertexClass, StreamOperatorFactory operatorFactory, String operatorName) { if (streamNodes.containsKey(vertexID)) { throw new RuntimeException("Duplicate vertexID " + vertexID); } StreamNode vertex = new StreamNode( vertexID, slotSharingGroup, coLocationGroup, operatorFactory, operatorName, vertexClass); streamNodes.put(vertexID, vertex); return vertex; } - 经过遍历操作处理后，由StreamNode构建的DAG图StreamGraph就生成了。
- 最后返回构建好的逻辑计划图StreamGraph。
4、优化的逻辑计划JobGraph的生成
生成好的StreamGraph会再次根据算子链做优化
//org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:2183
public JobClient executeAsync(StreamGraph streamGraph) throws Exception {
checkNotNull(streamGraph, "StreamGraph cannot be null.");
final PipelineExecutor executor = getPipelineExecutor();
CompletableFuture jobClientFuture =
executor.execute(streamGraph, configuration, userClassloader);
try {
JobClient jobClient = jobClientFuture.get();
jobListeners.forEach(jobListener -> jobListener.onJobSubmitted(jobClient, null));
collectIterators.forEach(iterator -> iterator.setJobClient(jobClient));
collectIterators.clear();
return jobClient;
} catch (ExecutionException executionException) {
final Throwable strippedException =
ExceptionUtils.stripExecutionException(executionException);
jobListeners.forEach(
jobListener -> jobListener.onJobSubmitted(null, strippedException));
throw new FlinkException(
String.format("Failed to execute job '%s'.", streamGraph.getJobName()),
strippedException);
}
} yarn job调用的是下面这个方法
//org/apache/flink/client/deployment/executors/AbstractJobClusterExecutor.java:66
@Override
public CompletableFuture execute(
@Nonnull final Pipeline pipeline,
@Nonnull final Configuration configuration,
@Nonnull final ClassLoader userCodeClassloader)
throws Exception {
final JobGraph jobGraph = PipelineExecutorUtils.getJobGraph(pipeline, configuration);
try (final ClusterDescriptor clusterDescriptor =
clusterClientFactory.createClusterDescriptor(configuration)) {
final ExecutionConfigAccessor configAccessor =
ExecutionConfigAccessor.fromConfiguration(configuration);
final ClusterSpecification clusterSpecification =
clusterClientFactory.getClusterSpecification(configuration);
final ClusterClientProvider clusterClientProvider =
clusterDescriptor.deployJobCluster(
clusterSpecification, jobGraph, configAccessor.getDetachedMode());
LOG.info("Job has been submitted with JobID " + jobGraph.getJobID());
return CompletableFuture.completedFuture(
new ClusterClientJobClientAdapter<>(
clusterClientProvider, jobGraph.getJobID(), userCodeClassloader));
}
} StreamGraph优化生成JobGraph
//org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:204
private JobGraph createJobGraph() {
preValidate();
jobGraph.setJobType(streamGraph.getJobType());
jobGraph.enableApproximateLocalRecovery(
streamGraph.getCheckpointConfig().isApproximateLocalRecoveryEnabled());
// Generate deterministic hashes for the nodes in order to identify them across
// submission iff they didn't change.
Map hashes =
defaultStreamGraphHasher.traverseStreamGraphAndGenerateHashes(streamGraph);
// Generate legacy version hashes for backwards compatibility
List> legacyHashes = new ArrayList<>(legacyStreamGraphHashers.size());
for (StreamGraphHasher hasher : legacyStreamGraphHashers) {
legacyHashes.add(hasher.traverseStreamGraphAndGenerateHashes(streamGraph));
}
setChaining(hashes, legacyHashes);
setPhysicalEdges();
markContainsSourcesOrSinks();
setSlotSharingAndCoLocation();
setManagedMemoryFraction(
Collections.unmodifiableMap(jobVertices),
Collections.unmodifiableMap(vertexConfigs),
Collections.unmodifiableMap(chainedConfigs),
id -> streamGraph.getStreamNode(id).getManagedMemoryOperatorScopeUseCaseWeights(),
id -> streamGraph.getStreamNode(id).getManagedMemorySlotScopeUseCases());
configureCheckpointing();
jobGraph.setSavepointRestoreSettings(streamGraph.getSavepointRestoreSettings());
final Map distributedCacheEntries =
JobGraphUtils.prepareUserArtifactEntries(
streamGraph.getUserArtifacts().stream()
.collect(Collectors.toMap(e -> e.f0, e -> e.f1)),
jobGraph.getJobID());
for (Map.Entry entry :
distributedCacheEntries.entrySet()) {
jobGraph.addUserArtifact(entry.getKey(), entry.getValue());
}
// set the ExecutionConfig last when it has been finalized
try {
jobGraph.setExecutionConfig(streamGraph.getExecutionConfig());
} catch (IOException e) {
throw new IllegalConfigurationException(
"Could not serialize the ExecutionConfig."
+ "This indicates that non-serializable types (like custom serializers) were registered");
}
jobGraph.setChangelogStateBackendEnabled(streamGraph.isChangelogStateBackendEnabled());
addVertexIndexPrefixInVertexName();
setVertexDescription();
// Wait for the serialization of operator coordinators and stream config.
try {
FutureUtils.combineAll(
vertexConfigs.values().stream()
.map(
config ->
config.triggerSerializationAndReturnFuture(
serializationExecutor))
.collect(Collectors.toList()))
.get();
FutureUtils.combineAll(coordinatorSerializationFutures).get();
} catch (Exception e) {
throw new FlinkRuntimeException("Error in serialization.", e);
}
if (!streamGraph.getJobStatusHooks().isEmpty()) {
jobGraph.setJobStatusHooks(streamGraph.getJobStatusHooks());
}
return jobGraph;
} 核心生成逻辑是line222 setChaining中调用createChain，具体核心逻辑如下：
//org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:596
private List createChain(
final Integer currentNodeId,
final int chainIndex,
final OperatorChainInfo chainInfo,
final Map chainEntryPoints) {
Integer startNodeId = chainInfo.getStartNodeId();
if (!builtVertices.contains(startNodeId)) {
List transitiveOutEdges = new ArrayList();
List chainableOutputs = new ArrayList();
List nonChainableOutputs = new ArrayList();
StreamNode currentNode = streamGraph.getStreamNode(currentNodeId);
for (StreamEdge outEdge : currentNode.getOutEdges()) {
if (isChainable(outEdge, streamGraph)) {
chainableOutputs.add(outEdge);
} else {
nonChainableOutputs.add(outEdge);
}
}
for (StreamEdge chainable : chainableOutputs) {
transitiveOutEdges.addAll(
createChain(
chainable.getTargetId(),
chainIndex + 1,
chainInfo,
chainEntryPoints));
}
for (StreamEdge nonChainable : nonChainableOutputs) {
transitiveOutEdges.add(nonChainable);
createChain(
nonChainable.getTargetId(),
1, // operators start at position 1 because 0 is for chained source inputs
chainEntryPoints.computeIfAbsent(
nonChainable.getTargetId(),
(k) -> chainInfo.newChain(nonChainable.getTargetId())),
chainEntryPoints);
}
chainedNames.put(
currentNodeId,
createChainedName(
currentNodeId,
chainableOutputs,
Optional.ofNullable(chainEntryPoints.get(currentNodeId))));
chainedMinResources.put(
currentNodeId, createChainedMinResources(currentNodeId, chainableOutputs));
chainedPreferredResources.put(
currentNodeId,
createChainedPreferredResources(currentNodeId, chainableOutputs));
OperatorID currentOperatorId =
chainInfo.addNodeToChain(
currentNodeId,
streamGraph.getStreamNode(currentNodeId).getOperatorName());
if (currentNode.getInputFormat() != null) {
getOrCreateFormatContainer(startNodeId)
.addInputFormat(currentOperatorId, currentNode.getInputFormat());
}
if (currentNode.getOutputFormat() != null) {
getOrCreateFormatContainer(startNodeId)
.addOutputFormat(currentOperatorId, currentNode.getOutputFormat());
}
StreamConfig config =
currentNodeId.equals(startNodeId)
? createJobVertex(startNodeId, chainInfo)
: new StreamConfig(new Configuration());
setVertexConfig(
currentNodeId,
config,
chainableOutputs,
nonChainableOutputs,
chainInfo.getChainedSources());
if (currentNodeId.equals(startNodeId)) {
config.setChainStart();
config.setChainIndex(chainIndex);
config.setOperatorName(streamGraph.getStreamNode(currentNodeId).getOperatorName());
LinkedHashSet transitiveOutputs = new LinkedHashSet<>();
for (StreamEdge edge : transitiveOutEdges) {
NonChainedOutput output =
opIntermediateOutputs.get(edge.getSourceId()).get(edge);
transitiveOutputs.add(output);
connect(startNodeId, edge, output);
}
config.setVertexNonChainedOutputs(new ArrayList<>(transitiveOutputs));
config.setTransitiveChainedTaskConfigs(chainedConfigs.get(startNodeId));
} else {
chainedConfigs.computeIfAbsent(
startNodeId, k -> new HashMap());
config.setChainIndex(chainIndex);
StreamNode node = streamGraph.getStreamNode(currentNodeId);
config.setOperatorName(node.getOperatorName());
chainedConfigs.get(startNodeId).put(currentNodeId, config);
}
config.setOperatorID(currentOperatorId);
if (chainableOutputs.isEmpty()) {
config.setChainEnd();
}
return transitiveOutEdges;
} else {
return new ArrayList<>();
}
} 总结上述算子链接优化逻辑：
- 如果从当前算子链的startNode开始没有生成过 JobVertex，则执行 chain逻辑，line622会进行深度遍历，将从源节点开始到第一个不可 chain 的 StreamNode 之间的算子做 chain 操作【先算叶子节点的 chain，依次往根节点计算】
- line 629 遇到不可 chain 的边，开始深度遍历生成 JobVertex
- line 672 会将 StreamNode 的输入输出配置，包括序列化配置等设置到上面的 StreamingConfig 中，并在 vertexConfigs 中保存起来，如果是新生成的 JobVertex，对应的 StreamingConfig 会以startNodeId做为 key 进行保存
- transitiveOutEdges会保存的该节点下游所有的nonChainableOutputs的边，最终的方法会返回这个nonChainableOutputs的数据结构
- 连接startNode和所有的transitiveOutEdges【在输入 JobVertex 创建 IntermediateDataSet，partition类型为 pipeline，生成 JobEdge】
- 如果是新生成JobVertex，继续设置config，包括setChainStart，所有物理输出，及直接逻辑输出、chainedConfigs等
- 如果不是新生成 JobVertex，直接chainedConfigs
简述JobGraph生成的流程
- 从DataStream上操作生成的transformations List
- 根据transformations生成运行时的TransformationTranslator并生成StreamNode和StreamEdge
- 做算子的chain操作合并生成JobVertex然后生成新的StramEdge
一个JobVertex代表一个逻辑计划的节点，也就是DAG图上的顶点。注意在代码里面，JobVertex其实被作为配置封装到了StreamConfig
那具体这个StreamConfig（JobVertex）怎么生成的：
//org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java:770
private StreamConfig createJobVertex(Integer streamNodeId, OperatorChainInfo chainInfo) {
JobVertex jobVertex;
StreamNode streamNode = streamGraph.getStreamNode(streamNodeId);
byte[] hash = chainInfo.getHash(streamNodeId);
if (hash == null) {
throw new IllegalStateException(
"Cannot find node hash. "
+ "Did you generate them before calling this method?");
}
JobVertexID jobVertexId = new JobVertexID(hash);
List> chainedOperators =
chainInfo.getChainedOperatorHashes(streamNodeId);
List operatorIDPairs = new ArrayList<>();
if (chainedOperators != null) {
for (Tuple2 chainedOperator : chainedOperators) {
OperatorID userDefinedOperatorID =
chainedOperator.f1 == null ? null : new OperatorID(chainedOperator.f1);
operatorIDPairs.add(
OperatorIDPair.of(
new OperatorID(chainedOperator.f0), userDefinedOperatorID));
}
}
if (chainedInputOutputFormats.containsKey(streamNodeId)) {
jobVertex =
new InputOutputFormatVertex(
chainedNames.get(streamNodeId), jobVertexId, operatorIDPairs);
chainedInputOutputFormats
.get(streamNodeId)
.write(new TaskConfig(jobVertex.getConfiguration()));
} else {
jobVertex = new JobVertex(chainedNames.get(streamNodeId), jobVertexId, operatorIDPairs);
}
if (streamNode.getConsumeClusterDatasetId() != null) {
jobVertex.addIntermediateDataSetIdToConsume(streamNode.getConsumeClusterDatasetId());
}
for (OperatorCoordinator.Provider coordinatorProvider :
chainInfo.getCoordinatorProviders()) {
coordinatorSerializationFutures.add(
CompletableFuture.runAsync(
() -> {
try {
jobVertex.addOperatorCoordinator(
new SerializedValue<>(coordinatorProvider));
} catch (IOException e) {
throw new FlinkRuntimeException(
String.format(
"Coordinator Provider for node %s is not serializable.",
chainedNames.get(streamNodeId)),
e);
}
},
serializationExecutor));
}
jobVertex.setResources(
chainedMinResources.get(streamNodeId), chainedPreferredResources.get(streamNodeId));
jobVertex.setInvokableClass(streamNode.getJobVertexClass());
int parallelism = streamNode.getParallelism();
if (parallelism > 0) {
jobVertex.setParallelism(parallelism);
} else {
parallelism = jobVertex.getParallelism();
}
jobVertex.setMaxParallelism(streamNode.getMaxParallelism());
if (LOG.isDebugEnabled()) {
LOG.debug("Parallelism set: {} for {}", parallelism, streamNodeId);
}
jobVertices.put(streamNodeId, jobVertex);
builtVertices.add(streamNodeId);
jobGraph.addVertex(jobVertex);
return new StreamConfig(jobVertex.getConfiguration());
} 这里重点注意
- line798 chainedInputOutputFormats.containsKey(streamNodeId) ，判断当前节点是不是数据源节点，如果是会将用户代码设置到中jobVertex的配置中，方便在jobmanager提交作业的时候做初始化。
- line836 jobVertex.setInvokableClass(streamNode.getJobVertexClass()); 这是设置任务实际运行时的执行类，通过这个类调用用户自定义的操作算子函数，是flink任务中真正被执行的类。
最终JobGraoh生成完成，由客户端提交给yarn创建的集群，在jobmanager中再生成物理计划。下一节介绍flink如何向yarn申请资源，并启动jobmanager和taskmanager，部署整个任务。