flink分析使用之二入口代码分析

flink分析使用之二入口代码分析

一、简介

在Flink中，启动的方式有三种，即local,standalone和Yarn,同时其还可以配置为高可用（HA）集群。它们的实现虽然有所异同，但是总体的原理是一致的。这里的源码分析从standalone的源码开始分析，即：org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint此类的入口点开始分析。

二、启动代码

先看一下代码：

1、脚本代码

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`

. "$bin"/config.sh

# Start the JobManager instance(s)
shopt -s nocasematch
if [[ $HIGH_AVAILABILITY == "zookeeper" ]]; then
    # HA Mode
    readMasters

    echo "Starting HA cluster with ${#MASTERS[@]} masters."

    for ((i=0;i<${#MASTERS[@]};++i)); do
        master=${MASTERS[i]}
        webuiport=${WEBUIPORTS[i]}

        if [ ${MASTERS_ALL_LOCALHOST} = true ] ; then
            "${FLINK_BIN_DIR}"/jobmanager.sh start "${master}" "${webuiport}"
        else
            ssh -n $FLINK_SSH_OPTS $master -- "nohup /bin/bash -l \"${FLINK_BIN_DIR}/jobmanager.sh\" start ${master} ${webuiport} &"
        fi
    done

else
    echo "Starting cluster."

    # Start single JobManager on this machine
    "$FLINK_BIN_DIR"/jobmanager.sh start
fi
shopt -u nocasematch

# Start TaskManager instance(s)
TMSlaves start

上面的代码可以看出，首先要启动一个JobManager的实例，而这个实例又会处理两种情况，即HA模式和单JobManager模式，这两种模式都会调用obmanager.sh 这个脚本，不过HA模式下会传入两个相关的参数"$master"和"${webuiport}"，所以看一下这个脚本：

USAGE="Usage: jobmanager.sh ((start|start-foreground) [host] [webui-port])|stop|stop-all"

STARTSTOP=$1
HOST=$2 # optional when starting multiple instances
WEBUIPORT=$3 # optional when starting multiple instances

#首先判断命令的启动条件
if [[ $STARTSTOP != "start" ]] && [[ $STARTSTOP != "start-foreground" ]] && [[ $STARTSTOP != "stop" ]] && [[ $STARTSTOP != "stop-all" ]]; then
  echo $USAGE
  exit 1
fi

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`

. "$bin"/config.sh

#启动的入口
ENTRYPOINT=standalonesession

#根据启动条件设置相关参数（包含相关启动应用的传入参数）
if [[ $STARTSTOP == "start" ]] || [[ $STARTSTOP == "start-foreground" ]]; then
    if [ ! -z "${FLINK_JM_HEAP_MB}" ] && [ "${FLINK_JM_HEAP}" == 0 ]; then
            echo "used deprecated key \`${KEY_JOBM_MEM_MB}\`, please replace with key \`${KEY_JOBM_MEM_SIZE}\`"
    else
            flink_jm_heap_bytes=$(parseBytes ${FLINK_JM_HEAP})
            FLINK_JM_HEAP_MB=$(getMebiBytes ${flink_jm_heap_bytes})
    fi

    if [[ ! ${FLINK_JM_HEAP_MB} =~ $IS_NUMBER ]] || [[ "${FLINK_JM_HEAP_MB}" -lt "0" ]]; then
        echo "[ERROR] Configured JobManager memory size is not a valid value. Please set '${KEY_JOBM_MEM_SIZE}' in ${FLINK_CONF_FILE}."
        exit 1
    fi

    if [ "${FLINK_JM_HEAP_MB}" -gt "0" ]; then
        export JVM_ARGS="$JVM_ARGS -Xms"$FLINK_JM_HEAP_MB"m -Xmx"$FLINK_JM_HEAP_MB"m"
    fi

    # Add JobManager-specific JVM options
    export FLINK_ENV_JAVA_OPTS="${FLINK_ENV_JAVA_OPTS} ${FLINK_ENV_JAVA_OPTS_JM}"

    # Startup parameters
    args=("--configDir" "${FLINK_CONF_DIR}" "--executionMode" "cluster")
    if [ ! -z $HOST ]; then
        args+=("--host")
        args+=("${HOST}")
    fi

    if [ ! -z $WEBUIPORT ]; then
        args+=("--webui-port")
        args+=("${WEBUIPORT}")
    fi
fi

#根据启动模式启动相关脚本，前端一个，守护进程一个
if [[ $STARTSTOP == "start-foreground" ]]; then
    exec "${FLINK_BIN_DIR}"/flink-console.sh $ENTRYPOINT "${args[@]}"
else
    "${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP $ENTRYPOINT "${args[@]}"
fi

在这个脚本里，设置了启动的模式，用来在flink-daemon.sh 或flink-console.sh中进行相关启动的判定，并启动相应的参数设置环境及相关的程序入口。这两个脚本参数没有本质的区别，分析一下常用的守护进程版本：

#根据前面的设置模式选择启动的JAVA程序入口点
case $DAEMON in
    (taskexecutor)
        CLASS_TO_RUN=org.apache.flink.runtime.taskexecutor.TaskManagerRunner
    ;;

    (zookeeper)
        CLASS_TO_RUN=org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer
    ;;

    (historyserver)
        CLASS_TO_RUN=org.apache.flink.runtime.webmonitor.history.HistoryServer
    ;;

    #默认在上面设置的为此模式，所以启动的代码在Flink的这个包下
    (standalonesession)
        CLASS_TO_RUN=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint
    ;;

    (standalonejob)
        CLASS_TO_RUN=org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint
    ;;

    (*)
        echo "Unknown daemon '${DAEMON}'. $USAGE."
        exit 1
    ;;
esac

#同样根据上面的设定参数来进行Flink的启停相关设置
case $STARTSTOP in

    (start)
        # Rotate log files
        rotateLogFilesWithPrefix "$FLINK_LOG_DIR" "$FLINK_LOG_PREFIX"

        # Print a warning if daemons are already running on host
        if [ -f "$pid" ]; then
          active=()
          while IFS='' read -r p || [[ -n "$p" ]]; do
            kill -0 $p >/dev/null 2>&1
            if [ $? -eq 0 ]; then
              active+=($p)
            fi
          done < "${pid}"

          count="${#active[@]}"

          if [ ${count} -gt 0 ]; then
            echo "[INFO] $count instance(s) of $DAEMON are already running on $HOSTNAME."
          fi
        fi

        # Evaluate user options for local variable expansion
        FLINK_ENV_JAVA_OPTS=$(eval echo ${FLINK_ENV_JAVA_OPTS})

        echo "Starting $DAEMON daemon on host $HOSTNAME."
        #注意这里进行了启动的控制，特别注意最后的守护进程设置和相关的显示设置
        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 200<&- 2>&1 < /dev/null &

        mypid=$!

        # Add to pid file if successful start
        if [[ ${mypid} =~ ${IS_NUMBER} ]] && kill -0 $mypid > /dev/null 2>&1 ; then
            echo $mypid >> "$pid"
        else
            echo "Error starting $DAEMON daemon."
            exit 1
        fi
    ;;

(stop)
    if [ -f "$pid" ]; then
        # Remove last in pid file
        to_stop=$(tail -n 1 "$pid")

        if [ -z $to_stop ]; then
            rm "$pid" # If all stopped, clean up pid file
            echo "No $DAEMON daemon to stop on host $HOSTNAME."
        else
            sed \$d "$pid" > "$pid.tmp" # all but last line

            # If all stopped, clean up pid file
            [ $(wc -l < "$pid.tmp") -eq 0 ] && rm "$pid" "$pid.tmp" || mv "$pid.tmp" "$pid"

            if kill -0 $to_stop > /dev/null 2>&1; then
                echo "Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME."
                kill $to_stop
            else
                echo "No $DAEMON daemon (pid: $to_stop) is running anymore on $HOSTNAME."
            fi
        fi
    else
        echo "No $DAEMON daemon to stop on host $HOSTNAME."
    fi
;;

(stop-all)
    if [ -f "$pid" ]; then
        mv "$pid" "${pid}.tmp"

        while read to_stop; do
            if kill -0 $to_stop > /dev/null 2>&1; then
                echo "Stopping $DAEMON daemon (pid: $to_stop) on host $HOSTNAME."
                kill $to_stop
            else
                echo "Skipping $DAEMON daemon (pid: $to_stop), because it is not running anymore on $HOSTNAME."
            fi
        done < "${pid}.tmp"
        rm "${pid}.tmp"
    fi
;;

(*)
    echo "Unexpected argument '$STARTSTOP'. $USAGE."
    exit 1
;;

esac

通过上面的脚本可以看出，Flink找到了组织，可以干活了。这和在嵌入式开发中Boot起到的作用完全一致。下面就进入到框架代码中去分析。

2、Flink框架代码启动

看一下Flink源码中相关的包下面的代码如下：

public class StandaloneSessionClusterEntrypoint extends SessionClusterEntrypoint {

	public StandaloneSessionClusterEntrypoint(Configuration configuration) {
		super(configuration);
	}

	@Override
	protected DispatcherResourceManagerComponentFactory<?> createDispatcherResourceManagerComponentFactory(Configuration configuration) {
		return new SessionDispatcherResourceManagerComponentFactory(StandaloneResourceManagerFactory.INSTANCE);
	}

	public static void main(String[] args) {
		// startup checks and logging
		EnvironmentInformation.logEnvironmentInfo(LOG, StandaloneSessionClusterEntrypoint.class.getSimpleName(), args);
		SignalHandler.register(LOG);
		JvmShutdownSafeguard.installAsShutdownHook(LOG);

		EntrypointClusterConfiguration entrypointClusterConfiguration = null;
		final CommandLineParser<EntrypointClusterConfiguration> commandLineParser = new CommandLineParser<>(new EntrypointClusterConfigurationParserFactory());

		try {
      //处理脚本中传入的相关参数
			entrypointClusterConfiguration = commandLineParser.parse(args);
		} catch (FlinkParseException e) {
			LOG.error("Could not parse command line arguments {}.", args, e);
			commandLineParser.printHelp(StandaloneSessionClusterEntrypoint.class.getSimpleName());
			System.exit(1);
		}

    //加载相关配置
		Configuration configuration = loadConfiguration(entrypointClusterConfiguration);
    //创建并启动集群
		StandaloneSessionClusterEntrypoint entrypoint = new StandaloneSessionClusterEntrypoint(configuration);

		ClusterEntrypoint.runClusterEntrypoint(entrypoint);
	}
}

上面的代码，主要分成几个步骤，可参看代码中的注释。下面是调用的启动代码：

	public static void runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint) {

		final String clusterEntrypointName = clusterEntrypoint.getClass().getSimpleName();
		try {
			clusterEntrypoint.startCluster();//从本行以后，为善后相关处理
		} catch (ClusterEntrypointException e) {
			LOG.error(String.format("Could not start cluster entrypoint %s.", clusterEntrypointName), e);
			System.exit(STARTUP_FAILURE_RETURN_CODE);
		}

		clusterEntrypoint.getTerminationFuture().whenComplete((applicationStatus, throwable) -> {
			final int returnCode;

			if (throwable != null) {
				returnCode = RUNTIME_FAILURE_RETURN_CODE;
			} else {
				returnCode = applicationStatus.processExitCode();
			}

			LOG.info("Terminating cluster entrypoint process {} with exit code {}.", clusterEntrypointName, returnCode, throwable);
			System.exit(returnCode);
		});
	}

通过反向启动startCluster:

public void startCluster() throws ClusterEntrypointException {
  LOG.info("Starting {}.", getClass().getSimpleName());

  try {
    configureFileSystems(configuration);

    SecurityContext securityContext = installSecurityContext(configuration);

    //此处处理异步线程调用
    securityContext.runSecured((Callable<Void>) () -> {
      runCluster(configuration);//注意此处启动集群

      return null;
    });//其后为异常处理
  } catch (Throwable t) {
    final Throwable strippedThrowable = ExceptionUtils.stripException(t, UndeclaredThrowableException.class);
......
}

它又会调用runCluster:

private void runCluster(Configuration configuration) throws Exception {
  synchronized (lock) {
    initializeServices(configuration);  //服务被初始化

    // write host information into configuration
    configuration.setString(JobManagerOptions.ADDRESS, commonRpcService.getAddress());
    configuration.setInteger(JobManagerOptions.PORT, commonRpcService.getPort());

    //创建一个分发器
    final DispatcherResourceManagerComponentFactory<?> dispatcherResourceManagerComponentFactory = createDispatcherResourceManagerComponentFactory(configuration);

    //在分发器中处理了大理的需要使用的相关服务，如AKKA，包括服务器的图形化页面都要使用
    clusterComponent = dispatcherResourceManagerComponentFactory.create(
      configuration,
      commonRpcService,
      haServices,
      blobServer,
      heartbeatServices,
      metricRegistry,
      archivedExecutionGraphStore,
      new AkkaQueryServiceRetriever(
        metricQueryServiceActorSystem,
        Time.milliseconds(configuration.getLong(WebOptions.TIMEOUT))),
      this);

    clusterComponent.getShutDownFuture().whenComplete(
      (ApplicationStatus applicationStatus, Throwable throwable) -> {
        if (throwable != null) {
          shutDownAsync(
            ApplicationStatus.UNKNOWN,
            ExceptionUtils.stringifyException(throwable),
            false);
        } else {
          // This is the general shutdown path. If a separate more specific shutdown was
          // already triggered, this will do nothing
          shutDownAsync(
            applicationStatus,
            null,
            true);
        }
      });
  }
}

这时候儿就快到准备完毕了，因为后面需要处理各种任务，所以相关的线程池等开始准备了：

protected void initializeServices(Configuration configuration) throws Exception {

  LOG.info("Initializing cluster services.");

  synchronized (lock) {
    final String bindAddress = configuration.getString(JobManagerOptions.ADDRESS);
    final String portRange = getRPCPortRange(configuration);

    //创建RPC服务
    commonRpcService = createRpcService(configuration, bindAddress, portRange);

    // update the configuration used to create the high availability services
    configuration.setString(JobManagerOptions.ADDRESS, commonRpcService.getAddress());
    configuration.setInteger(JobManagerOptions.PORT, commonRpcService.getPort());

    //线程池准备
    ioExecutor = Executors.newFixedThreadPool(
      Hardware.getNumberCPUCores(),
      new ExecutorThreadFactory("cluster-io"));

    //创建服务并启动
    haServices = createHaServices(configuration, ioExecutor);
    blobServer = new BlobServer(configuration, haServices.createBlobStore());
    blobServer.start();
    //心跳处理
    heartbeatServices = createHeartbeatServices(configuration);
    metricRegistry = createMetricRegistry(configuration);

    // TODO: This is a temporary hack until we have ported the MetricQueryService to the new RpcEndpoint
    // Start actor system for metric query service on any available port
    metricQueryServiceActorSystem = MetricUtils.startMetricsActorSystem(configuration, bindAddress, LOG);
    metricRegistry.startQueryService(metricQueryServiceActorSystem, null);

    //提交Graph存储的创建
    archivedExecutionGraphStore = createSerializableExecutionGraphStore(configuration, commonRpcService.getScheduledExecutor());

    //大型二进制的缓存，这个属于用户自行管理，不存储
    transientBlobCache = new TransientBlobCache(
      configuration,
      new InetSocketAddress(
        commonRpcService.getAddress(),
        blobServer.getPort()));
  }
}

通过上述的代码流程，可以清晰的看到Flink的启动过程，当然，其它情况下的启动与此相通，有机会再详细分析。

三、总结

Flink的源码的启动还需要处理很多的相关工作的，这里只是从一个standalone模式下的展开的，为了让整个流程更清晰，以后会在相关的代码分析中，加入Example目录下的源码调用入流程分析，将二者结合起来，会更加容易理解Flink的运行机制。