weixin_34383618
weixin_34383618

Flink1.7.2 并行计算源码分析

Flink1.7.2 并行计算源码分析

源码

  • 源码:https://github.com/opensourceteams/fink-maven-scala-2
  • Flink1.7.2 Source、Window数据交互源码分析: https://github.com/opensourceteams/fink-maven-scala-2/blob/master/md/miniCluster/flink-source-window-data-exchange.md

概述

  • Flink Window如何进行并行计算
  • Flink source如何按key,hash分区，并发射到对应分区的下游Window

WordCount 程序

package com.opensourceteams.module.bigdata.flink.example.stream.worldcount.nc.parallelism

import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.windowing.time.Time

/**
  * nc -lk 1234  输入数据
  */
object SocketWindowWordCountLocal {



  def main(args: Array[String]): Unit = {


    val port = 1234
    // get the execution environment
   // val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment


    val configuration : Configuration = getConfiguration(true)

    val env:StreamExecutionEnvironment = StreamExecutionEnvironment.createLocalEnvironment(1,configuration)





    // get input data by connecting to the socket
    val dataStream = env.socketTextStream("localhost", port, '\n')



    import org.apache.flink.streaming.api.scala._
    val textResult = dataStream.flatMap( w => w.split("\\s") ).map( w => WordWithCount(w,1))
      .keyBy("word")
      /**
        * 每20秒刷新一次，相当于重新开始计数，
        * 好处，不需要一直拿所有的数据统计
        * 只需要在指定时间间隔内的增量数据，减少了数据规模
        */
      .timeWindow(Time.seconds(5))
      //.countWindow(3)
      //.countWindow(3,1)
      //.countWindowAll(3)


      .sum("count" )

    textResult
      .setParallelism(3)
      .print()




    if(args == null || args.size ==0){


      println("==================================以下为执行计划==================================")
      println("执行地址(firefox效果更好):https://flink.apache.org/visualizer")
      //执行计划
      //println(env.getExecutionPlan)
     // println("==================================以上为执行计划 JSON串==================================\n")
      //StreamGraph
     println(env.getStreamGraph.getStreamingPlanAsJSON)



      //JsonPlanGenerator.generatePlan(jobGraph)

      env.execute("默认作业")

    }else{
      env.execute(args(0))
    }

    println("结束")

  }


  // Data type for words with count
  case class WordWithCount(word: String, count: Long){
    //override def toString: String = Thread.currentThread().getName + word + " : " + count
  }


  def getConfiguration(isDebug:Boolean = false):Configuration = {

    val configuration : Configuration = new Configuration()

    if(isDebug){
      val timeout = "100000 s"
      val timeoutHeartbeatPause = "1000000 s"
      configuration.setString("akka.ask.timeout",timeout)
      configuration.setString("akka.lookup.timeout",timeout)
      configuration.setString("akka.tcp.timeout",timeout)
      configuration.setString("akka.transport.heartbeat.interval",timeout)
      configuration.setString("akka.transport.heartbeat.pause",timeoutHeartbeatPause)
      configuration.setString("akka.watch.heartbeat.pause",timeout)
      configuration.setInteger("heartbeat.interval",10000000)
      configuration.setInteger("heartbeat.timeout",50000000)
    }


    configuration
  }


}

输入数据

1 2 3 4 5 6 7 8 9 10

源码分析

ExecutionGraph.scheduleEager

  • ExecutionGraph 调度

  • executionsToDeploy包括所有的(Source,Window,Sink),在这里设置的setParallelism()并行度为多少，就有多少个Window,本案例设置的为3，所以executionsToDeploy对象的数据如下

    • (Source: Socket Stream -> Flat Map -> Map (1/1))
    • (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (3/3))
    • (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (2/3))
    • (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (1/3))
    • (Sink: Print to Std. Out (1/1))
    • 详细executionsToDeploy对象
  1. = {Execution@5324} "Attempt #0 (Source: Socket Stream -> Flat Map -> Map (1/1)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@22dc33b2 - [SCHEDULED]"
  2. = {Execution@5506} "Attempt #0 (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (3/3)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@8f216e4 - [SCHEDULED]"
  3. = {Execution@5507} "Attempt #0 (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (2/3)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@50ccca83 - [SCHEDULED]"
  4. = {Execution@5508} "Attempt #0 (Window(TumblingProcessingTimeWindows(5000), ProcessingTimeTrigger, SumAggregator, PassThroughWindowFunction) (1/3)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@243b4f41 - [SCHEDULED]"

  5. = {Execution@5509} "Attempt #0 (Sink: Print to Std. Out (1/1)) @ org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@67b9a9d7 - [SCHEDULED]"

源码

调用Execution.deploy()进行部署

/**
    *
    *
    * @param slotProvider  The resource provider from which the slots are allocated
    * @param timeout       The maximum time that the deployment may take, before a
    *                      TimeoutException is thrown.
    * @returns Future which is completed once the {@link ExecutionGraph} has been scheduled.
    * The future can also be completed exceptionally if an error happened.
    */
   private CompletableFuture scheduleEager(SlotProvider slotProvider, final Time timeout) {
       checkState(state == JobStatus.RUNNING, "job is not running currently");

       // Important: reserve all the space we need up front.
       // that way we do not have any operation that can fail between allocating the slots
       // and adding them to the list. If we had a failure in between there, that would
       // cause the slots to get lost
       final boolean queued = allowQueuedScheduling;

       // collecting all the slots may resize and fail in that operation without slots getting lost
       final ArrayList> allAllocationFutures = new ArrayList<>(getNumberOfExecutionJobVertices());

       final Set allPreviousAllocationIds =
           Collections.unmodifiableSet(computeAllPriorAllocationIdsIfRequiredByScheduling());

       // allocate the slots (obtain all their futures
       for (ExecutionJobVertex ejv : getVerticesTopologically()) {
           // these calls are not blocking, they only return futures
           Collection> allocationFutures = ejv.allocateResourcesForAll(
               slotProvider,
               queued,
               LocationPreferenceConstraint.ALL,
               allPreviousAllocationIds,
               timeout);

           allAllocationFutures.addAll(allocationFutures);
       }

       // this future is complete once all slot futures are complete.
       // the future fails once one slot future fails.
       final ConjunctFuture> allAllocationsFuture = FutureUtils.combineAll(allAllocationFutures);

       final CompletableFuture currentSchedulingFuture = allAllocationsFuture
           .thenAccept(
               (Collection executionsToDeploy) -> {
                   for (Execution execution : executionsToDeploy) {
                       try {
                           execution.deploy();
                       } catch (Throwable t) {
                           throw new CompletionException(
                               new FlinkException(
                                   String.format("Could not deploy execution %s.", execution),
                                   t));
                       }
                   }
               })
           // Generate a more specific failure message for the eager scheduling
           .exceptionally(
               (Throwable throwable) -> {
                   final Throwable strippedThrowable = ExceptionUtils.stripCompletionException(throwable);
                   final Throwable resultThrowable;

                   if (strippedThrowable instanceof TimeoutException) {
                       int numTotal = allAllocationsFuture.getNumFuturesTotal();
                       int numComplete = allAllocationsFuture.getNumFuturesCompleted();
                       String message = "Could not allocate all requires slots within timeout of " +
                           timeout + ". Slots required: " + numTotal + ", slots allocated: " + numComplete;

                       resultThrowable = new NoResourceAvailableException(message);
                   } else {
                       resultThrowable = strippedThrowable;
                   }

                   throw new CompletionException(resultThrowable);
               });

       return currentSchedulingFuture;
   }

ExecutionState

  • 由于(Source、Window、Sink)都是做为Execution对象来运行，先来了解一下Execution有哪些状态，即状态的流转，方便理解流程
  • Execution状态的流转为: CREATED(已创建) -> SCHEDULED(已调度) -> DEPLOYING(部署中) -> RUNNING(运行中) -> FINISHED(已完成) 等，以下ExecutionState中有详细说明
package org.apache.flink.runtime.execution;

/**
 * An enumeration of all states that a task can be in during its execution.
 * Tasks usually start in the state {@code CREATED} and switch states according to
 * this diagram:
 * 
{@code
 *
 *     CREATED  -> SCHEDULED -> DEPLOYING -> RUNNING -> FINISHED
 *        |            |            |          |
 *        |            |            |   +------+
 *        |            |            V   V
 *        |            |         CANCELLING -----+----> CANCELED
 *        |            |                         |
 *        |            +-------------------------+
 *        |
 *        |                                   ... -> FAILED
 *        V
 *    RECONCILING  -> RUNNING | FINISHED | CANCELED | FAILED
 *
 * }

 *
 * 
It is possible to enter the {@code RECONCILING} state from {@code CREATED}
 * state if job manager fail over, and the {@code RECONCILING} state can switch into
 * any existing task state.
 *
 * 
It is possible to enter the {@code FAILED} state from any other state.
 *
 * 
The states {@code FINISHED}, {@code CANCELED}, and {@code FAILED} are
 * considered terminal states.
 */
public enum ExecutionState {

    CREATED,
    
    SCHEDULED,
    
    DEPLOYING,
    
    RUNNING,

    /**
     * This state marks "successfully completed". It can only be reached when a
     * program reaches the "end of its input". The "end of input" can be reached
     * when consuming a bounded input (fix set of files, bounded query, etc) or
     * when stopping a program (not cancelling!) which make the input look like
     * it reached its end at a specific point.
     */
    FINISHED,
    
    CANCELING,
    
    CANCELED,
    
    FAILED,

    RECONCILING;

    public boolean isTerminal() {
        return this == FINISHED || this == CANCELED || this == FAILED;
    }
}

Execution.deploy()

  • 对Execution进行部署

  • 更新Execution状态,将当前Execution的状态由SCHEDULED更新为DEPLOYING，即由已调度状态更新为部署中

    transitionState(previous, DEPLOYING)

  • INFO日志输出：部署哪一个Execution到哪一台机器上

    LOG.info(String.format("Deploying %s (attempt #%d) to %s", 
    13:11:55,910 INFO  [flink-akka.actor.default-dispatcher-3] org.apache.flink.runtime.executiongraph.Execution.deploy(Execution.java:599)      - Deploying Source: Socket Stream -> Flat Map -> Map (1/1) (attempt #0) to localhost

  • 构建TaskDeploymentDescriptor对象，该对象引用Task实例Execution的id,slot(槽位),就可以确定Execution在哪个slot上运行

    final TaskDeploymentDescriptor deployment = vertex.createDeploymentDescriptor(
            attemptId,
            slot,
            taskRestore,
            attemptNumber);

  • slot得到TaskManager

    final TaskManagerGateway taskManagerGateway = slot.getTaskManagerGateway();

  • TaskManager.submitTask 提交任务，参数为TaskDeploymentDescriptor

    final CompletableFuture submitResultFuture = taskManagerGateway.submitTask(deployment, rpcTimeout);
  • 接下来就交给TaskManager去处理了
  • 源码
/**
     * Deploys the execution to the previously assigned resource.
     *
     * @throws JobException if the execution cannot be deployed to the assigned resource
     */
    public void deploy() throws JobException {
        final LogicalSlot slot  = assignedResource;

        checkNotNull(slot, "In order to deploy the execution we first have to assign a resource via tryAssignResource.");

        // Check if the TaskManager died in the meantime
        // This only speeds up the response to TaskManagers failing concurrently to deployments.
        // The more general check is the rpcTimeout of the deployment call
        if (!slot.isAlive()) {
            throw new JobException("Target slot (TaskManager) for deployment is no longer alive.");
        }

        // make sure exactly one deployment call happens from the correct state
        // note: the transition from CREATED to DEPLOYING is for testing purposes only
        ExecutionState previous = this.state;
        if (previous == SCHEDULED || previous == CREATED) {
            if (!transitionState(previous, DEPLOYING)) {
                // race condition, someone else beat us to the deploying call.
                // this should actually not happen and indicates a race somewhere else
                throw new IllegalStateException("Cannot deploy task: Concurrent deployment call race.");
            }
        }
        else {
            // vertex may have been cancelled, or it was already scheduled
            throw new IllegalStateException("The vertex must be in CREATED or SCHEDULED state to be deployed. Found state " + previous);
        }

        if (this != slot.getPayload()) {
            throw new IllegalStateException(
                String.format("The execution %s has not been assigned to the assigned slot.", this));
        }

        try {

            // race double check, did we fail/cancel and do we need to release the slot?
            if (this.state != DEPLOYING) {
                slot.releaseSlot(new FlinkException("Actual state of execution " + this + " (" + state + ") does not match expected state DEPLOYING."));
                return;
            }

            if (LOG.isInfoEnabled()) {
                LOG.info(String.format("Deploying %s (attempt #%d) to %s", vertex.getTaskNameWithSubtaskIndex(),
                        attemptNumber, getAssignedResourceLocation().getHostname()));
            }

            final TaskDeploymentDescriptor deployment = vertex.createDeploymentDescriptor(
                attemptId,
                slot,
                taskRestore,
                attemptNumber);

            // null taskRestore to let it be GC'ed
            taskRestore = null;

            final TaskManagerGateway taskManagerGateway = slot.getTaskManagerGateway();

            final CompletableFuture submitResultFuture = taskManagerGateway.submitTask(deployment, rpcTimeout);

            submitResultFuture.whenCompleteAsync(
                (ack, failure) -> {
                    // only respond to the failure case
                    if (failure != null) {
                        if (failure instanceof TimeoutException) {
                            String taskname = vertex.getTaskNameWithSubtaskIndex() + " (" + attemptId + ')';

                            markFailed(new Exception(
                                "Cannot deploy task " + taskname + " - TaskManager (" + getAssignedResourceLocation()
                                    + ") not responding after a rpcTimeout of " + rpcTimeout, failure));
                        } else {
                            markFailed(failure);
                        }
                    }
                },
                executor);
        }
        catch (Throwable t) {
            markFailed(t);
            ExceptionUtils.rethrow(t);
        }
    }

TaskExecutor.submitTask

  • TaskManager中是由TaskExecutor来提交任务

  • 将外部化数据从BLOB存储加载回对象

    // re-integrate offloaded data:
        try {
            tdd.loadBigData(blobCacheService.getPermanentBlobService());
        } catch (IOException | ClassNotFoundException e) {
            throw new TaskSubmissionException("Could not re-integrate offloaded TaskDeploymentDescriptor data.", e);
        }

  • 从序列化的对象中反序列化(通过类加载)，JobInformation，TaskInformation，用来构建TaskInformation,Task

        // deserialize the pre-serialized information
        final JobInformation jobInformation;
        final TaskInformation taskInformation;
        try {
            jobInformation = tdd.getSerializedJobInformation().deserializeValue(getClass().getClassLoader());
            taskInformation = tdd.getSerializedTaskInformation().deserializeValue(getClass().getClassLoader());
        } catch (IOException | ClassNotFoundException e) {
            throw new TaskSubmissionException("Could not deserialize the job or task information.", e);
        }

  • 指定Source中的拆分器，就是将不断产生数据的Source拆分给不同的Window做并行任务(RpcInputSplitProvider是其中的一种分配方式)

    InputSplitProvider inputSplitProvider = new RpcInputSplitProvider(
            jobManagerConnection.getJobManagerGateway(),
            taskInformation.getJobVertexId(),
            tdd.getExecutionAttemptId(),
            taskManagerConfiguration.getTimeout());

  • 构建任务状态管理器TaskStateManager

    final TaskStateManager taskStateManager = new TaskStateManagerImpl(
            jobId,
            tdd.getExecutionAttemptId(),
            localStateStore,
            taskRestore,
            checkpointResponder);

  • 构建任务Task

    Task task = new Task(
            jobInformation,
            taskInformation,
            tdd.getExecutionAttemptId(),
            tdd.getAllocationId(),
            tdd.getSubtaskIndex(),
            tdd.getAttemptNumber(),
            tdd.getProducedPartitions(),
            tdd.getInputGates(),
            tdd.getTargetSlotNumber(),
            taskExecutorServices.getMemoryManager(),
            taskExecutorServices.getIOManager(),
            taskExecutorServices.getNetworkEnvironment(),
            taskExecutorServices.getBroadcastVariableManager(),
            taskStateManager,
            taskManagerActions,
            inputSplitProvider,
            checkpointResponder,
            blobCacheService,
            libraryCache,
            fileCache,
            taskManagerConfiguration,
            taskMetricGroup,
            resultPartitionConsumableNotifier,
            partitionStateChecker,
            getRpcService().getExecutor());

  • 将任务增加到任务槽位中

                try {
            taskAdded = taskSlotTable.addTask(task);
        } catch (SlotNotFoundException | SlotNotActiveException e) {
            throw new TaskSubmissionException("Could not submit task.", e);
        }

  • 调用任务的启动线程，该方法会触发调用Task.run()函数，

            if (taskAdded) {
            task.startTaskThread();

            return CompletableFuture.completedFuture(Acknowledge.get());
        } else {
            final String message = "TaskManager already contains a task for id " +
                task.getExecutionId() + '.';

            log.debug(message);
            throw new TaskSubmissionException(message);
        }
  • 源码
@Override
    public CompletableFuture submitTask(
            TaskDeploymentDescriptor tdd,
            JobMasterId jobMasterId,
            Time timeout) {

        try {
            final JobID jobId = tdd.getJobId();
            final JobManagerConnection jobManagerConnection = jobManagerTable.get(jobId);

            if (jobManagerConnection == null) {
                final String message = "Could not submit task because there is no JobManager " +
                    "associated for the job " + jobId + '.';

                log.debug(message);
                throw new TaskSubmissionException(message);
            }

            if (!Objects.equals(jobManagerConnection.getJobMasterId(), jobMasterId)) {
                final String message = "Rejecting the task submission because the job manager leader id " +
                    jobMasterId + " does not match the expected job manager leader id " +
                    jobManagerConnection.getJobMasterId() + '.';

                log.debug(message);
                throw new TaskSubmissionException(message);
            }

            if (!taskSlotTable.tryMarkSlotActive(jobId, tdd.getAllocationId())) {
                final String message = "No task slot allocated for job ID " + jobId +
                    " and allocation ID " + tdd.getAllocationId() + '.';
                log.debug(message);
                throw new TaskSubmissionException(message);
            }

            // re-integrate offloaded data:
            try {
                tdd.loadBigData(blobCacheService.getPermanentBlobService());
            } catch (IOException | ClassNotFoundException e) {
                throw new TaskSubmissionException("Could not re-integrate offloaded TaskDeploymentDescriptor data.", e);
            }

            // deserialize the pre-serialized information
            final JobInformation jobInformation;
            final TaskInformation taskInformation;
            try {
                jobInformation = tdd.getSerializedJobInformation().deserializeValue(getClass().getClassLoader());
                taskInformation = tdd.getSerializedTaskInformation().deserializeValue(getClass().getClassLoader());
            } catch (IOException | ClassNotFoundException e) {
                throw new TaskSubmissionException("Could not deserialize the job or task information.", e);
            }

            if (!jobId.equals(jobInformation.getJobId())) {
                throw new TaskSubmissionException(
                    "Inconsistent job ID information inside TaskDeploymentDescriptor (" +
                        tdd.getJobId() + " vs. " + jobInformation.getJobId() + ")");
            }

            TaskMetricGroup taskMetricGroup = taskManagerMetricGroup.addTaskForJob(
                jobInformation.getJobId(),
                jobInformation.getJobName(),
                taskInformation.getJobVertexId(),
                tdd.getExecutionAttemptId(),
                taskInformation.getTaskName(),
                tdd.getSubtaskIndex(),
                tdd.getAttemptNumber());

            InputSplitProvider inputSplitProvider = new RpcInputSplitProvider(
                jobManagerConnection.getJobManagerGateway(),
                taskInformation.getJobVertexId(),
                tdd.getExecutionAttemptId(),
                taskManagerConfiguration.getTimeout());

            TaskManagerActions taskManagerActions = jobManagerConnection.getTaskManagerActions();
            CheckpointResponder checkpointResponder = jobManagerConnection.getCheckpointResponder();

            LibraryCacheManager libraryCache = jobManagerConnection.getLibraryCacheManager();
            ResultPartitionConsumableNotifier resultPartitionConsumableNotifier = jobManagerConnection.getResultPartitionConsumableNotifier();
            PartitionProducerStateChecker partitionStateChecker = jobManagerConnection.getPartitionStateChecker();

            final TaskLocalStateStore localStateStore = localStateStoresManager.localStateStoreForSubtask(
                jobId,
                tdd.getAllocationId(),
                taskInformation.getJobVertexId(),
                tdd.getSubtaskIndex());

            final JobManagerTaskRestore taskRestore = tdd.getTaskRestore();

            final TaskStateManager taskStateManager = new TaskStateManagerImpl(
                jobId,
                tdd.getExecutionAttemptId(),
                localStateStore,
                taskRestore,
                checkpointResponder);

            Task task = new Task(
                jobInformation,
                taskInformation,
                tdd.getExecutionAttemptId(),
                tdd.getAllocationId(),
                tdd.getSubtaskIndex(),
                tdd.getAttemptNumber(),
                tdd.getProducedPartitions(),
                tdd.getInputGates(),
                tdd.getTargetSlotNumber(),
                taskExecutorServices.getMemoryManager(),
                taskExecutorServices.getIOManager(),
                taskExecutorServices.getNetworkEnvironment(),
                taskExecutorServices.getBroadcastVariableManager(),
                taskStateManager,
                taskManagerActions,
                inputSplitProvider,
                checkpointResponder,
                blobCacheService,
                libraryCache,
                fileCache,
                taskManagerConfiguration,
                taskMetricGroup,
                resultPartitionConsumableNotifier,
                partitionStateChecker,
                getRpcService().getExecutor());

            log.info("Received task {}.", task.getTaskInfo().getTaskNameWithSubtasks());

            boolean taskAdded;

            try {
                taskAdded = taskSlotTable.addTask(task);
            } catch (SlotNotFoundException | SlotNotActiveException e) {
                throw new TaskSubmissionException("Could not submit task.", e);
            }

            if (taskAdded) {
                task.startTaskThread();

                return CompletableFuture.completedFuture(Acknowledge.get());
            } else {
                final String message = "TaskManager already contains a task for id " +
                    task.getExecutionId() + '.';

                log.debug(message);
                throw new TaskSubmissionException(message);
            }
        } catch (TaskSubmissionException e) {
            return FutureUtils.completedExceptionally(e);
        }
    }

Task.run()

  • 先来了解一下任务的概念,Task表示在TaskManager上执行并行子任务。 Task包装Flink操作符（可以是用户函数）并运行它，提供所有必需的服务，例如使用输入数据，生成结果（中间结果分区）并与JobManager通信。
    Flink运算符（作为AbstractInvokable的子类实现，只有数据读取器，写入程序和某些事件回调。该任务将这些操作连接到网络堆栈和actor消息，并跟踪执行状态并处理异常。

任务不知道它们与其他任务的关系，或者它们是第一次执行任务还是重复尝试。 所有这些只有JobManager知道。 所有任务都知道它自己的可运行代码，任务的配置以及要使用和生成的中间结果的ID（如果有的话）。
每个任务由一个专用线程运行。

  • run()是引导任务并执行其代码的核心工作方法

  • 这里是Task的执行状态，前面是Executition的执行状态，需要区分开来,更新任务状态，由CREATED(已创建)到DEPLOYING(部署中)

    // ----------------------------
//  Initial State transition
// ----------------------------
while (true) {
ExecutionState current = this.executionState;
if (current == ExecutionState.CREATED) {
    if (transitionState(ExecutionState.CREATED, ExecutionState.DEPLOYING)) {
        // success, we can start our work
        break;
    }
}

  • 创建文件系统流为这个任务

    // activate safety net for task thread
LOG.info("Creating FileSystem stream leak safety net for task {}", this);
FileSystemSafetyNet.initializeSafetyNetForThread();

  • 加载用户程序jar文件，给当前Task使用

    // first of all, get a user-code classloader
// this may involve downloading the job's JAR files and/or classes
LOG.info("Loading JAR files for task {}.", this);

userCodeClassLoader = createUserCodeClassloader();
final ExecutionConfig executionConfig = serializedExecutionConfig.deserializeValue(userCodeClassLoader);

  • 注册网络追踪给这当前任务

    // ----------------------------------------------------------------
// register the task with the network stack
// this operation may fail if the system does not have enough
// memory to run the necessary data exchanges
// the registration must also strictly be undone
// ----------------------------------------------------------------

LOG.info("Registering task at network: {}.", this);

network.registerTask(this);

  • 给当前任务构建运行环境

    Environment env = new RuntimeEnvironment(
    jobId,
    vertexId,
    executionId,
    executionConfig,
    taskInfo,
    jobConfiguration,
    taskConfiguration,
    userCodeClassLoader,
    memoryManager,
    ioManager,
    broadcastVariableManager,
    taskStateManager,
    accumulatorRegistry,
    kvStateRegistry,
    inputSplitProvider,
    distributedCacheEntries,
    producedPartitions,
    inputGates,
    network.getTaskEventDispatcher(),
    checkpointResponder,
    taskManagerConfig,
    metrics,
    this);

  • 加载并实例化任务的可调用代码(用户代码)

    // now load and instantiate the task's invokable code
invokable = loadAndInstantiateInvokable(userCodeClassLoader, nameOfInvokableClass, env);

  • 更新当前任务状态,从DEPLOYING(部署中)更新为RUNNING(运行中)

        // switch to the RUNNING state, if that fails, we have been canceled/failed in the meantime
if (!transitionState(ExecutionState.DEPLOYING, ExecutionState.RUNNING)) {
    throw new CancelTaskException();
}

  • StreamTask.invoke()

    // run the invokable
invokable.invoke();
  • 源码
/**
     * The core work method that bootstraps the task and executes its code.
     */
    @Override
    public void run() {

        // ----------------------------
        //  Initial State transition
        // ----------------------------
        while (true) {
            ExecutionState current = this.executionState;
            if (current == ExecutionState.CREATED) {
                if (transitionState(ExecutionState.CREATED, ExecutionState.DEPLOYING)) {
                    // success, we can start our work
                    break;
                }
            }
            else if (current == ExecutionState.FAILED) {
                // we were immediately failed. tell the TaskManager that we reached our final state
                notifyFinalState();
                if (metrics != null) {
                    metrics.close();
                }
                return;
            }
            else if (current == ExecutionState.CANCELING) {
                if (transitionState(ExecutionState.CANCELING, ExecutionState.CANCELED)) {
                    // we were immediately canceled. tell the TaskManager that we reached our final state
                    notifyFinalState();
                    if (metrics != null) {
                        metrics.close();
                    }
                    return;
                }
            }
            else {
                if (metrics != null) {
                    metrics.close();
                }
                throw new IllegalStateException("Invalid state for beginning of operation of task " + this + '.');
            }
        }

        // all resource acquisitions and registrations from here on
        // need to be undone in the end
        Map> distributedCacheEntries = new HashMap<>();
        AbstractInvokable invokable = null;

        try {
            // ----------------------------
            //  Task Bootstrap - We periodically
            //  check for canceling as a shortcut
            // ----------------------------

            // activate safety net for task thread
            LOG.info("Creating FileSystem stream leak safety net for task {}", this);
            FileSystemSafetyNet.initializeSafetyNetForThread();

            blobService.getPermanentBlobService().registerJob(jobId);

            // first of all, get a user-code classloader
            // this may involve downloading the job's JAR files and/or classes
            LOG.info("Loading JAR files for task {}.", this);

            userCodeClassLoader = createUserCodeClassloader();
            final ExecutionConfig executionConfig = serializedExecutionConfig.deserializeValue(userCodeClassLoader);

            if (executionConfig.getTaskCancellationInterval() >= 0) {
                // override task cancellation interval from Flink config if set in ExecutionConfig
                taskCancellationInterval = executionConfig.getTaskCancellationInterval();
            }

            if (executionConfig.getTaskCancellationTimeout() >= 0) {
                // override task cancellation timeout from Flink config if set in ExecutionConfig
                taskCancellationTimeout = executionConfig.getTaskCancellationTimeout();
            }

            if (isCanceledOrFailed()) {
                throw new CancelTaskException();
            }

            // ----------------------------------------------------------------
            // register the task with the network stack
            // this operation may fail if the system does not have enough
            // memory to run the necessary data exchanges
            // the registration must also strictly be undone
            // ----------------------------------------------------------------

            LOG.info("Registering task at network: {}.", this);

            network.registerTask(this);

            // add metrics for buffers
            this.metrics.getIOMetricGroup().initializeBufferMetrics(this);

            // register detailed network metrics, if configured
            if (taskManagerConfig.getConfiguration().getBoolean(TaskManagerOptions.NETWORK_DETAILED_METRICS)) {
                // similar to MetricUtils.instantiateNetworkMetrics() but inside this IOMetricGroup
                MetricGroup networkGroup = this.metrics.getIOMetricGroup().addGroup("Network");
                MetricGroup outputGroup = networkGroup.addGroup("Output");
                MetricGroup inputGroup = networkGroup.addGroup("Input");

                // output metrics
                for (int i = 0; i < producedPartitions.length; i++) {
                    ResultPartitionMetrics.registerQueueLengthMetrics(
                        outputGroup.addGroup(i), producedPartitions[i]);
                }

                for (int i = 0; i < inputGates.length; i++) {
                    InputGateMetrics.registerQueueLengthMetrics(
                        inputGroup.addGroup(i), inputGates[i]);
                }
            }

            // next, kick off the background copying of files for the distributed cache
            try {
                for (Map.Entry entry :
                        DistributedCache.readFileInfoFromConfig(jobConfiguration)) {
                    LOG.info("Obtaining local cache file for '{}'.", entry.getKey());
                    Future cp = fileCache.createTmpFile(entry.getKey(), entry.getValue(), jobId, executionId);
                    distributedCacheEntries.put(entry.getKey(), cp);
                }
            }
            catch (Exception e) {
                throw new Exception(
                    String.format("Exception while adding files to distributed cache of task %s (%s).", taskNameWithSubtask, executionId), e);
            }

            if (isCanceledOrFailed()) {
                throw new CancelTaskException();
            }

            // ----------------------------------------------------------------
            //  call the user code initialization methods
            // ----------------------------------------------------------------

            TaskKvStateRegistry kvStateRegistry = network.createKvStateTaskRegistry(jobId, getJobVertexId());

            Environment env = new RuntimeEnvironment(
                jobId,
                vertexId,
                executionId,
                executionConfig,
                taskInfo,
                jobConfiguration,
                taskConfiguration,
                userCodeClassLoader,
                memoryManager,
                ioManager,
                broadcastVariableManager,
                taskStateManager,
                accumulatorRegistry,
                kvStateRegistry,
                inputSplitProvider,
                distributedCacheEntries,
                producedPartitions,
                inputGates,
                network.getTaskEventDispatcher(),
                checkpointResponder,
                taskManagerConfig,
                metrics,
                this);

            // now load and instantiate the task's invokable code
            invokable = loadAndInstantiateInvokable(userCodeClassLoader, nameOfInvokableClass, env);

            // ----------------------------------------------------------------
            //  actual task core work
            // ----------------------------------------------------------------

            // we must make strictly sure that the invokable is accessible to the cancel() call
            // by the time we switched to running.
            this.invokable = invokable;

            // switch to the RUNNING state, if that fails, we have been canceled/failed in the meantime
            if (!transitionState(ExecutionState.DEPLOYING, ExecutionState.RUNNING)) {
                throw new CancelTaskException();
            }

            // notify everyone that we switched to running
            taskManagerActions.updateTaskExecutionState(new TaskExecutionState(jobId, executionId, ExecutionState.RUNNING));

            // make sure the user code classloader is accessible thread-locally
            executingThread.setContextClassLoader(userCodeClassLoader);

            // run the invokable
            invokable.invoke();

            // make sure, we enter the catch block if the task leaves the invoke() method due
            // to the fact that it has been canceled
            if (isCanceledOrFailed()) {
                throw new CancelTaskException();
            }

            // ----------------------------------------------------------------
            //  finalization of a successful execution
            // ----------------------------------------------------------------

            // finish the produced partitions. if this fails, we consider the execution failed.
            for (ResultPartition partition : producedPartitions) {
                if (partition != null) {
                    partition.finish();
                }
            }

            // try to mark the task as finished
            // if that fails, the task was canceled/failed in the meantime
            if (!transitionState(ExecutionState.RUNNING, ExecutionState.FINISHED)) {
                throw new CancelTaskException();
            }
        }
        catch (Throwable t) {

            // unwrap wrapped exceptions to make stack traces more compact
            if (t instanceof WrappingRuntimeException) {
                t = ((WrappingRuntimeException) t).unwrap();
            }

            // ----------------------------------------------------------------
            // the execution failed. either the invokable code properly failed, or
            // an exception was thrown as a side effect of cancelling
            // ----------------------------------------------------------------

            try {
                // check if the exception is unrecoverable
                if (ExceptionUtils.isJvmFatalError(t) ||
                        (t instanceof OutOfMemoryError && taskManagerConfig.shouldExitJvmOnOutOfMemoryError())) {

                    // terminate the JVM immediately
                    // don't attempt a clean shutdown, because we cannot expect the clean shutdown to complete
                    try {
                        LOG.error("Encountered fatal error {} - terminating the JVM", t.getClass().getName(), t);
                    } finally {
                        Runtime.getRuntime().halt(-1);
                    }
                }

                // transition into our final state. we should be either in DEPLOYING, RUNNING, CANCELING, or FAILED
                // loop for multiple retries during concurrent state changes via calls to cancel() or
                // to failExternally()
                while (true) {
                    ExecutionState current = this.executionState;

                    if (current == ExecutionState.RUNNING || current == ExecutionState.DEPLOYING) {
                        if (t instanceof CancelTaskException) {
                            if (transitionState(current, ExecutionState.CANCELED)) {
                                cancelInvokable(invokable);
                                break;
                            }
                        }
                        else {
                            if (transitionState(current, ExecutionState.FAILED, t)) {
                                // proper failure of the task. record the exception as the root cause
                                failureCause = t;
                                cancelInvokable(invokable);

                                break;
                            }
                        }
                    }
                    else if (current == ExecutionState.CANCELING) {
                        if (transitionState(current, ExecutionState.CANCELED)) {
                            break;
                        }
                    }
                    else if (current == ExecutionState.FAILED) {
                        // in state failed already, no transition necessary any more
                        break;
                    }
                    // unexpected state, go to failed
                    else if (transitionState(current, ExecutionState.FAILED, t)) {
                        LOG.error("Unexpected state in task {} ({}) during an exception: {}.", taskNameWithSubtask, executionId, current);
                        break;
                    }
                    // else fall through the loop and
                }
            }
            catch (Throwable tt) {
                String message = String.format("FATAL - exception in exception handler of task %s (%s).", taskNameWithSubtask, executionId);
                LOG.error(message, tt);
                notifyFatalError(message, tt);
            }
        }
        finally {
            try {
                LOG.info("Freeing task resources for {} ({}).", taskNameWithSubtask, executionId);

                // clear the reference to the invokable. this helps guard against holding references
                // to the invokable and its structures in cases where this Task object is still referenced
                this.invokable = null;

                // stop the async dispatcher.
                // copy dispatcher reference to stack, against concurrent release
                ExecutorService dispatcher = this.asyncCallDispatcher;
                if (dispatcher != null && !dispatcher.isShutdown()) {
                    dispatcher.shutdownNow();
                }

                // free the network resources
                network.unregisterTask(this);

                // free memory resources
                if (invokable != null) {
                    memoryManager.releaseAll(invokable);
                }

                // remove all of the tasks library resources
                libraryCache.unregisterTask(jobId, executionId);
                fileCache.releaseJob(jobId, executionId);
                blobService.getPermanentBlobService().releaseJob(jobId);

                // close and de-activate safety net for task thread
                LOG.info("Ensuring all FileSystem streams are closed for task {}", this);
                FileSystemSafetyNet.closeSafetyNetAndGuardedResourcesForThread();

                notifyFinalState();
            }
            catch (Throwable t) {
                // an error in the resource cleanup is fatal
                String message = String.format("FATAL - exception in resource cleanup of task %s (%s).", taskNameWithSubtask, executionId);
                LOG.error(message, t);
                notifyFatalError(message, t);
            }

            // un-register the metrics at the end so that the task may already be
            // counted as finished when this happens
            // errors here will only be logged
            try {
                metrics.close();
            }
            catch (Throwable t) {
                LOG.error("Error during metrics de-registration of task {} ({}).", taskNameWithSubtask, executionId, t);
            }
        }
    }

StreamTask.invoke()

  • 创建一个后端状态，stateBackend,此时为MemoryStateBackend

    stateBackend = createStateBackend();

  • 如果没有调置时间服务，就创建SystemProcessingTimeService,它将当前处理时间指定为调用的结果(时间)

                // if the clock is not already set, then assign a default TimeServiceProvider
        if (timerService == null) {
            ThreadFactory timerThreadFactory = new DispatcherThreadFactory(TRIGGER_THREAD_GROUP,
                "Time Trigger for " + getName(), getUserCodeClassLoader());

            timerService = new SystemProcessingTimeService(this, getCheckpointLock(), timerThreadFactory);
        }

  • 当前流任务对应的操作链条，此处不同的流任务对应的操作链条不一样，像source流中，用户自定义的函数链不一样，这个operatorChain也不一样，这里以WordCount为例说明

    operatorChain = new OperatorChain<>(this, streamRecordWriters);
  • Source流中的操作链条 operatorChain.allOperators

  • headOperator = operatorChain.getHeadOperator()为StreamSource

    allOperators = {StreamOperator[3]@5784} 
0 = {StreamMap@5793} 
1 = {StreamFlatMap@5794} 
2 = {StreamSource@5789}

  • 任务初使化

    // task specific initialization
        init();

  • 在所有的operators是opened之前所有的触发器调度不能被执行，就是需要先把operator.open

                // we need to make sure that any triggers scheduled in open() cannot be
        // executed before all operators are opened
        synchronized (lock) {

            // both the following operations are protected by the lock
            // so that we avoid race conditions in the case that initializeState()
            // registers a timer, that fires before the open() is called.

            initializeState();
            openAllOperators();
        }

  • 调用具体任务的run()函数去处理，这里分不同的类型

    • Source 调的是SourceStreamTask.run()函数

    • Window 调的是OneInputStreamTask.run()函数

          // let the task do its work
        isRunning = true;
        run();
  • 源码
public final void invoke() throws Exception {

        boolean disposed = false;
        try {
            // -------- Initialize ---------
            LOG.debug("Initializing {}.", getName());

            asyncOperationsThreadPool = Executors.newCachedThreadPool();

            CheckpointExceptionHandlerFactory cpExceptionHandlerFactory = createCheckpointExceptionHandlerFactory();

            synchronousCheckpointExceptionHandler = cpExceptionHandlerFactory.createCheckpointExceptionHandler(
                getExecutionConfig().isFailTaskOnCheckpointError(),
                getEnvironment());

            asynchronousCheckpointExceptionHandler = new AsyncCheckpointExceptionHandler(this);

            stateBackend = createStateBackend();
            checkpointStorage = stateBackend.createCheckpointStorage(getEnvironment().getJobID());

            // if the clock is not already set, then assign a default TimeServiceProvider
            if (timerService == null) {
                ThreadFactory timerThreadFactory = new DispatcherThreadFactory(TRIGGER_THREAD_GROUP,
                    "Time Trigger for " + getName(), getUserCodeClassLoader());

                timerService = new SystemProcessingTimeService(this, getCheckpointLock(), timerThreadFactory);
            }

            operatorChain = new OperatorChain<>(this, streamRecordWriters);
            headOperator = operatorChain.getHeadOperator();

            // task specific initialization
            init();

            // save the work of reloading state, etc, if the task is already canceled
            if (canceled) {
                throw new CancelTaskException();
            }

            // -------- Invoke --------
            LOG.debug("Invoking {}", getName());

            // we need to make sure that any triggers scheduled in open() cannot be
            // executed before all operators are opened
            synchronized (lock) {

                // both the following operations are protected by the lock
                // so that we avoid race conditions in the case that initializeState()
                // registers a timer, that fires before the open() is called.

                initializeState();
                openAllOperators();
            }

            // final check to exit early before starting to run
            if (canceled) {
                throw new CancelTaskException();
            }

            // let the task do its work
            isRunning = true;
            run();

            // if this left the run() method cleanly despite the fact that this was canceled,
            // make sure the "clean shutdown" is not attempted
            if (canceled) {
                throw new CancelTaskException();
            }

            LOG.debug("Finished task {}", getName());

            // make sure no further checkpoint and notification actions happen.
            // we make sure that no other thread is currently in the locked scope before
            // we close the operators by trying to acquire the checkpoint scope lock
            // we also need to make sure that no triggers fire concurrently with the close logic
            // at the same time, this makes sure that during any "regular" exit where still
            synchronized (lock) {
                // this is part of the main logic, so if this fails, the task is considered failed
                closeAllOperators();

                // make sure no new timers can come
                timerService.quiesce();

                // only set the StreamTask to not running after all operators have been closed!
                // See FLINK-7430
                isRunning = false;
            }

            // make sure all timers finish
            timerService.awaitPendingAfterQuiesce();

            LOG.debug("Closed operators for task {}", getName());

            // make sure all buffered data is flushed
            operatorChain.flushOutputs();

            // make an attempt to dispose the operators such that failures in the dispose call
            // still let the computation fail
            tryDisposeAllOperators();
            disposed = true;
        }
        finally {
            // clean up everything we initialized
            isRunning = false;

            // Now that we are outside the user code, we do not want to be interrupted further
            // upon cancellation. The shutdown logic below needs to make sure it does not issue calls
            // that block and stall shutdown.
            // Additionally, the cancellation watch dog will issue a hard-cancel (kill the TaskManager
            // process) as a backup in case some shutdown procedure blocks outside our control.
            setShouldInterruptOnCancel(false);

            // clear any previously issued interrupt for a more graceful shutdown
            Thread.interrupted();

            // stop all timers and threads
            tryShutdownTimerService();

            // stop all asynchronous checkpoint threads
            try {
                cancelables.close();
                shutdownAsyncThreads();
            }
            catch (Throwable t) {
                // catch and log the exception to not replace the original exception
                LOG.error("Could not shut down async checkpoint threads", t);
            }

            // we must! perform this cleanup
            try {
                cleanup();
            }
            catch (Throwable t) {
                // catch and log the exception to not replace the original exception
                LOG.error("Error during cleanup of stream task", t);
            }

            // if the operators were not disposed before, do a hard dispose
            if (!disposed) {
                disposeAllOperators();
            }

            // release the output resources. this method should never fail.
            if (operatorChain != null) {
                // beware: without synchronization, #performCheckpoint() may run in
                //         parallel and this call is not thread-safe
                synchronized (lock) {
                    operatorChain.releaseOutputs();
                }
            }
        }
    }

SourceStreamTask.run()

  • headOperator,会依次从StreamSource.operatorChain中调用(StreamSource,StreamFlatMap,StreamMap),这个就是链式调用，把这一个类型的任务，可以依次调用执行对应的operator，不需要每次一次operator输出中间结果
  • StreamSource操作会调用SocketTextStreamFunction.run()函数来处理
  • 源码
    protected void run() throws Exception {
        headOperator.run(getCheckpointLock(), getStreamStatusMaintainer());
    }

SocketTextStreamFunction.run()

  • 建立Source的Sorcket连接，读取流中的数据，每次读取8K的数据放到缓存中，再按行进行解析
  • 把一行数据放到ctx.collect(record);进行后续的处理
  • 此处调用的是NonTimestampContext.collect(record)
public void run(SourceContext ctx) throws Exception {
        final StringBuilder buffer = new StringBuilder();
        long attempt = 0;

        while (isRunning) {

            try (Socket socket = new Socket()) {
                currentSocket = socket;

                LOG.info("Connecting to server socket " + hostname + ':' + port);
                socket.connect(new InetSocketAddress(hostname, port), CONNECTION_TIMEOUT_TIME);
                try (BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()))) {

                    char[] cbuf = new char[8192];
                    int bytesRead;
                    while (isRunning && (bytesRead = reader.read(cbuf)) != -1) {
                        buffer.append(cbuf, 0, bytesRead);
                        int delimPos;
                        while (buffer.length() >= delimiter.length() && (delimPos = buffer.indexOf(delimiter)) != -1) {
                            String record = buffer.substring(0, delimPos);
                            // truncate trailing carriage return
                            if (delimiter.equals("\n") && record.endsWith("\r")) {
                                record = record.substring(0, record.length() - 1);
                            }
                            ctx.collect(record);
                            buffer.delete(0, delimPos + delimiter.length());
                        }
                    }
                }
            }

            // if we dropped out of this loop due to an EOF, sleep and retry
            if (isRunning) {
                attempt++;
                if (maxNumRetries == -1 || attempt < maxNumRetries) {
                    LOG.warn("Lost connection to server socket. Retrying in " + delayBetweenRetries + " msecs...");
                    Thread.sleep(delayBetweenRetries);
                }
                else {
                    // this should probably be here, but some examples expect simple exists of the stream source
                    // throw new EOFException("Reached end of stream and reconnects are not enabled.");
                    break;
                }
            }
        }

        // collect trailing data
        if (buffer.length() > 0) {
            ctx.collect(buffer.toString());
        }
    }

RecordWriter.emit

  • numChannels 为并行度，即为DataStrea.setParallelism(2) 设置的并行度
  • channelSelector.selectChannels(record, numChannels),分区算法，给当前数据分区(分区是为了给下游并行计算使用，在这里是发给不同的Window，并行计算)
  • 调用KeyGroupStreamPartitioner.selectChannels具体的分区算法
  • 源码
    public void emit(T record) throws IOException, InterruptedException {
        emit(record, channelSelector.selectChannels(record, numChannels));
    }

KeyGroupStreamPartitioner.selectChannels

  • 分区实现KeyGroupRangeAssignment.assignKeyToParallelOperator(key, maxParallelism, numberOfOutputChannels);

        分区代码
numberOfOutputChannels: 一共分为多少个分区,即并行度为多少
maxParallelism:最大并行度，默认为128
key:处理的数据，对应的key的值

KeyGroupRangeAssignment.assignKeyToParallelOperator(key, maxParallelism, numberOfOutputChannels);
  • 源码
    @Override
    public int[] selectChannels(
        SerializationDelegate> record,
        int numberOfOutputChannels) {

        K key;
        try {
            key = keySelector.getKey(record.getInstance().getValue());
        } catch (Exception e) {
            throw new RuntimeException("Could not extract key from " + record.getInstance().getValue(), e);
        }
        returnArray[0] = KeyGroupRangeAssignment.assignKeyToParallelOperator(key, maxParallelism, numberOfOutputChannels);
        return returnArray;
    }

OneInputStreamTask.run()

  • StreamTask.run().run()函数调用，当为Window时调用OneInputStreamTask.run()
  • 调用StreamInputProcessor.processInput()函数
  • 源码
    protected void run() throws Exception {
        // cache processor reference on the stack, to make the code more JIT friendly
        final StreamInputProcessor inputProcessor = this.inputProcessor;

        while (running && inputProcessor.processInput()) {
            // all the work happens in the "processInput" method
        }
    }

StreamInputProcessor.processInput()

  • 调用BarrierTracker.getNextNonBlocked()得到一个元素(key,value)得值，也就是source进行flatMap,map 函数之后的数据，此时，还没有进行聚合操作,注意这里会得到
  • 此时的数据还没有进行分配给不同的Window,当Source有数据发送过来后，就一条一条调用streamOperator.processElement(record),即WindowOperator.processElement进行处理
public boolean processInput() throws Exception {
        if (isFinished) {
            return false;
        }
        if (numRecordsIn == null) {
            try {
                numRecordsIn = ((OperatorMetricGroup) streamOperator.getMetricGroup()).getIOMetricGroup().getNumRecordsInCounter();
            } catch (Exception e) {
                LOG.warn("An exception occurred during the metrics setup.", e);
                numRecordsIn = new SimpleCounter();
            }
        }

        while (true) {
            if (currentRecordDeserializer != null) {
                DeserializationResult result = currentRecordDeserializer.getNextRecord(deserializationDelegate);

                if (result.isBufferConsumed()) {
                    currentRecordDeserializer.getCurrentBuffer().recycleBuffer();
                    currentRecordDeserializer = null;
                }

                if (result.isFullRecord()) {
                    StreamElement recordOrMark = deserializationDelegate.getInstance();

                    if (recordOrMark.isWatermark()) {
                        // handle watermark
                        statusWatermarkValve.inputWatermark(recordOrMark.asWatermark(), currentChannel);
                        continue;
                    } else if (recordOrMark.isStreamStatus()) {
                        // handle stream status
                        statusWatermarkValve.inputStreamStatus(recordOrMark.asStreamStatus(), currentChannel);
                        continue;
                    } else if (recordOrMark.isLatencyMarker()) {
                        // handle latency marker
                        synchronized (lock) {
                            streamOperator.processLatencyMarker(recordOrMark.asLatencyMarker());
                        }
                        continue;
                    } else {
                        // now we can do the actual processing
                        StreamRecord record = recordOrMark.asRecord();
                        synchronized (lock) {
                            numRecordsIn.inc();
                            streamOperator.setKeyContextElement1(record);
                            streamOperator.processElement(record);
                        }
                        return true;
                    }
                }
            }

            final BufferOrEvent bufferOrEvent = barrierHandler.getNextNonBlocked();
            if (bufferOrEvent != null) {
                if (bufferOrEvent.isBuffer()) {
                    currentChannel = bufferOrEvent.getChannelIndex();
                    currentRecordDeserializer = recordDeserializers[currentChannel];
                    currentRecordDeserializer.setNextBuffer(bufferOrEvent.getBuffer());
                }
                else {
                    // Event received
                    final AbstractEvent event = bufferOrEvent.getEvent();
                    if (event.getClass() != EndOfPartitionEvent.class) {
                        throw new IOException("Unexpected event: " + event);
                    }
                }
            }
            else {
                isFinished = true;
                if (!barrierHandler.isEmpty()) {
                    throw new IllegalStateException("Trailing data in checkpoint barrier handler.");
                }
                return false;
            }
        }
    }

WindowOperator.processElement(StreamRecord element)

  • WindowOperator.processElement,给每一个WordWithCount(1,1) 这样的元素分配window,也就是确认每一个元素属于哪一个窗口，因为需要对同一个窗口的相同key进行聚合操作

    final Collection elementWindows = windowAssigner.assignWindows(
            element.getValue(), element.getTimestamp(), windowAssignerContext);

  • 把当前元素增加到state中保存,add函数中会对相同key进行聚合操作(reduce),对同一个window中相同key进行求和就是在这个方法中进行的

    windowState.add(element.getValue());
  • triggerContext.onElement(element),对当前元素设置trigger,也就是当前元素的window在哪个时间点触发(结束的时间点),
    把当前元素的key,增加到InternalTimerServiceImpl.processingTimeTimersQueue中，每一条数据会加一次，加完后会去重，相当于Set,对相同Key的处理，

后面发送给Sink的数据，就是遍历这个processingTimeTimersQueue中的数据，当然，每次发送第一个元素，发送后，会把最后一个元素放到第一个元素

TriggerResult triggerResult = triggerContext.onElement(element);

public void processElement(StreamRecord element) throws Exception {
        final Collection elementWindows = windowAssigner.assignWindows(
            element.getValue(), element.getTimestamp(), windowAssignerContext);

        //if element is handled by none of assigned elementWindows
        boolean isSkippedElement = true;

        final K key = this.getKeyedStateBackend().getCurrentKey();

        if (windowAssigner instanceof MergingWindowAssigner) {
            MergingWindowSet mergingWindows = getMergingWindowSet();

            for (W window: elementWindows) {

                // adding the new window might result in a merge, in that case the actualWindow
                // is the merged window and we work with that. If we don't merge then
                // actualWindow == window
                W actualWindow = mergingWindows.addWindow(window, new MergingWindowSet.MergeFunction() {
                    @Override
                    public void merge(W mergeResult,
                            Collection mergedWindows, W stateWindowResult,
                            Collection mergedStateWindows) throws Exception {

                        if ((windowAssigner.isEventTime() && mergeResult.maxTimestamp() + allowedLateness <= internalTimerService.currentWatermark())) {
                            throw new UnsupportedOperationException("The end timestamp of an " +
                                    "event-time window cannot become earlier than the current watermark " +
                                    "by merging. Current watermark: " + internalTimerService.currentWatermark() +
                                    " window: " + mergeResult);
                        } else if (!windowAssigner.isEventTime() && mergeResult.maxTimestamp() <= internalTimerService.currentProcessingTime()) {
                            throw new UnsupportedOperationException("The end timestamp of a " +
                                    "processing-time window cannot become earlier than the current processing time " +
                                    "by merging. Current processing time: " + internalTimerService.currentProcessingTime() +
                                    " window: " + mergeResult);
                        }

                        triggerContext.key = key;
                        triggerContext.window = mergeResult;

                        triggerContext.onMerge(mergedWindows);

                        for (W m: mergedWindows) {
                            triggerContext.window = m;
                            triggerContext.clear();
                            deleteCleanupTimer(m);
                        }

                        // merge the merged state windows into the newly resulting state window
                        windowMergingState.mergeNamespaces(stateWindowResult, mergedStateWindows);
                    }
                });

                // drop if the window is already late
                if (isWindowLate(actualWindow)) {
                    mergingWindows.retireWindow(actualWindow);
                    continue;
                }
                isSkippedElement = false;

                W stateWindow = mergingWindows.getStateWindow(actualWindow);
                if (stateWindow == null) {
                    throw new IllegalStateException("Window " + window + " is not in in-flight window set.");
                }

                windowState.setCurrentNamespace(stateWindow);
                windowState.add(element.getValue());

                triggerContext.key = key;
                triggerContext.window = actualWindow;

                TriggerResult triggerResult = triggerContext.onElement(element);

                if (triggerResult.isFire()) {
                    ACC contents = windowState.get();
                    if (contents == null) {
                        continue;
                    }
                    emitWindowContents(actualWindow, contents);
                }

                if (triggerResult.isPurge()) {
                    windowState.clear();
                }
                registerCleanupTimer(actualWindow);
            }

            // need to make sure to update the merging state in state
            mergingWindows.persist();
        } else {
            for (W window: elementWindows) {

                // drop if the window is already late
                if (isWindowLate(window)) {
                    continue;
                }
                isSkippedElement = false;

                windowState.setCurrentNamespace(window);
                windowState.add(element.getValue());

                triggerContext.key = key;
                triggerContext.window = window;

                TriggerResult triggerResult = triggerContext.onElement(element);

                if (triggerResult.isFire()) {
                    ACC contents = windowState.get();
                    if (contents == null) {
                        continue;
                    }
                    emitWindowContents(window, contents);
                }

                if (triggerResult.isPurge()) {
                    windowState.clear();
                }
                registerCleanupTimer(window);
            }
        }

        // side output input event if
        // element not handled by any window
        // late arriving tag has been set
        // windowAssigner is event time and current timestamp + allowed lateness no less than element timestamp
        if (isSkippedElement && isElementLate(element)) {
            if (lateDataOutputTag != null){
                sideOutput(element);
            } else {
                this.numLateRecordsDropped.inc();
            }
        }
    }

InternalTimerServiceImpl.onProcessingTime

  • processingTimeTimersQueue(HeapPriorityQueueSet) 该对象中存储了所有的key,这些key是去重后，按处理顺序排序
  • processingTimeTimersQueue.peek() 取出第一条数据进行处理
  • processingTimeTimersQueue.poll();会移除第一条数据，并且，拿最后一条数据，放第1一个元素，导致，所有元素的处理顺序是，先处理第一个元素，然后，把最后一个元素放第一个，
    最后一个就置为空，再循环处理所有数据，相当于处理完第一个元素，处后从最后一个元素开始处理，一直处理到完成,举例
1 2 1 3 2 5 4
存为 1 2 3 5 4 
顺序就变为
 1
 4
 5
 3
 2
  • keyContext.setCurrentKey(timer.getKey());//设置当前的key,当前需要处理的

  • triggerTarget.onProcessingTime(timer);// 调用 WindowOperator.onProcessingTime(timer)处理

queue = {HeapPriorityQueueElement[129]@8184} 
 1 = {TimerHeapInternalTimer@12441} "Timer{timestamp=1551505439999, key=(1), namespace=TimeWindow{start=1551505380000, end=1551505440000}}"
 2 = {TimerHeapInternalTimer@12442} "Timer{timestamp=1551505439999, key=(2), namespace=TimeWindow{start=1551505380000, end=1551505440000}}"
 3 = {TimerHeapInternalTimer@12443} "Timer{timestamp=1551505439999, key=(3), namespace=TimeWindow{start=1551505380000, end=1551505440000}}"
 5 = {TimerHeapInternalTimer@12443} "Timer{timestamp=1551505439999, key=(3), namespace=TimeWindow{start=1551505380000, end=1551505440000}}"
 4 = {TimerHeapInternalTimer@12443} "Timer{timestamp=1551505439999, key=(3), namespace=TimeWindow{start=1551505380000, end=1551505440000}}"
  • 调用 WindowOperator.onProcessingTime(timer)处理当前key;

public void onProcessingTime(long time) throws Exception {
        // null out the timer in case the Triggerable calls registerProcessingTimeTimer()
        // inside the callback.
        nextTimer = null;

        InternalTimer timer;

        while ((timer = processingTimeTimersQueue.peek()) != null && timer.getTimestamp() <= time) {
            processingTimeTimersQueue.poll();
            keyContext.setCurrentKey(timer.getKey());
            triggerTarget.onProcessingTime(timer);
        }

        if (timer != null && nextTimer == null) {
            nextTimer = processingTimeService.registerTimer(timer.getTimestamp(), this);
        }
    }

WindowOperator.onProcessingTime

  • triggerResult.isFire()// 当前元素对应的window已经可以发射了，即过了结束时间
  • windowState.get() //取出当前key对应的(key,value)此时已经是相同key聚合后的值
  • emitWindowContents(triggerContext.window, contents);//发送给Sink进行处理
public void onProcessingTime(InternalTimer timer) throws Exception {
        triggerContext.key = timer.getKey();
        triggerContext.window = timer.getNamespace();

        MergingWindowSet mergingWindows;

        if (windowAssigner instanceof MergingWindowAssigner) {
            mergingWindows = getMergingWindowSet();
            W stateWindow = mergingWindows.getStateWindow(triggerContext.window);
            if (stateWindow == null) {
                // Timer firing for non-existent window, this can only happen if a
                // trigger did not clean up timers. We have already cleared the merging
                // window and therefore the Trigger state, however, so nothing to do.
                return;
            } else {
                windowState.setCurrentNamespace(stateWindow);
            }
        } else {
            windowState.setCurrentNamespace(triggerContext.window);
            mergingWindows = null;
        }

        TriggerResult triggerResult = triggerContext.onProcessingTime(timer.getTimestamp());

        if (triggerResult.isFire()) {
            ACC contents = windowState.get();
            if (contents != null) {
                emitWindowContents(triggerContext.window, contents);
            }
        }

        if (triggerResult.isPurge()) {
            windowState.clear();
        }

        if (!windowAssigner.isEventTime() && isCleanupTime(triggerContext.window, timer.getTimestamp())) {
            clearAllState(triggerContext.window, windowState, mergingWindows);
        }

        if (mergingWindows != null) {
            // need to make sure to update the merging state in state
            mergingWindows.persist();
        }
    }

SingleInputGate

  • 中间数据处理流程(数据交互)
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.flink.runtime.io.network.partition.consumer;

import org.apache.flink.api.common.JobID;
import org.apache.flink.core.memory.MemorySegment;
import org.apache.flink.runtime.deployment.InputChannelDeploymentDescriptor;
import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
import org.apache.flink.runtime.deployment.ResultPartitionLocation;
import org.apache.flink.runtime.event.AbstractEvent;
import org.apache.flink.runtime.event.TaskEvent;
import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
import org.apache.flink.runtime.io.network.NetworkEnvironment;
import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
import org.apache.flink.runtime.io.network.buffer.BufferProvider;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
import org.apache.flink.runtime.io.network.partition.consumer.InputChannel.BufferAndAvailability;
import org.apache.flink.runtime.jobgraph.DistributionPattern;
import org.apache.flink.runtime.jobgraph.IntermediateDataSetID;
import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
import org.apache.flink.runtime.metrics.groups.TaskIOMetricGroup;
import org.apache.flink.runtime.taskmanager.TaskActions;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.BitSet;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Timer;

import static org.apache.flink.util.Preconditions.checkArgument;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.flink.util.Preconditions.checkState;

/**
 * An input gate consumes one or more partitions of a single produced intermediate result.
 *
 * 
Each intermediate result is partitioned over its producing parallel subtasks; each of these
 * partitions is furthermore partitioned into one or more subpartitions.
 *
 * 
As an example, consider a map-reduce program, where the map operator produces data and the
 * reduce operator consumes the produced data.
 *
 * 
{@code
 * +-----+              +---------------------+              +--------+
 * | Map | = produce => | Intermediate Result | <= consume = | Reduce |
 * +-----+              +---------------------+              +--------+
 * }

 *
 * 
When deploying such a program in parallel, the intermediate result will be partitioned over its
 * producing parallel subtasks; each of these partitions is furthermore partitioned into one or more
 * subpartitions.
 *
 * 
{@code
 *                            Intermediate result
 *               +-----------------------------------------+
 *               |                      +----------------+ |              +-----------------------+
 * +-------+     | +-------------+  +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
 * | Map 1 | ==> | | Partition 1 | =|   +----------------+ |         |    +-----------------------+
 * +-------+     | +-------------+  +=> | Subpartition 2 | | <==+    |
 *               |                      +----------------+ |    |    | Subpartition request
 *               |                                         |    |    |
 *               |                      +----------------+ |    |    |
 * +-------+     | +-------------+  +=> | Subpartition 1 | | <==+====+
 * | Map 2 | ==> | | Partition 2 | =|   +----------------+ |    |         +-----------------------+
 * +-------+     | +-------------+  +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
 *               |                      +----------------+ |              +-----------------------+
 *               +-----------------------------------------+
 * }

 *
 * 
In the above example, two map subtasks produce the intermediate result in parallel, resulting
 * in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two
 * subpartitions -- one for each parallel reduce subtask.
 */
public class SingleInputGate implements InputGate {

end

你可能感兴趣的:(Flink1.7.2 并行计算源码分析)

  • JVM源码分析之堆外内存完全解读 HeapDump性能社区
    概述广义的堆外内存说到堆外内存，那大家肯定想到堆内内存，这也是我们大家接触最多的，我们在jvm参数里通常设置-Xmx来指定我们的堆的最大值，不过这还不是我们理解的Java堆，-Xmx的值是新生代和老生代的和的最大值，我们在jvm参数里通常还会加一个参数-XX:MaxPermSize来指定持久代的最大值，那么我们认识的Java堆的最大值其实是-Xmx和-XX:MaxPermSize的总和，在分代算法
  • Golang Channel PandaSkr golang
    Channel解析1.Channel源码分析1.1Channel数据结构typehchanstruct{qcountuint//channel的元素数量dataqsizuint//channel循环队列长度bufunsafe.Pointer//指向循环队列的指针elemsizeuint16//元素大小closeduint32//channel是否关闭0-未关闭elemtype*_type//元素类
  • Java集合类框架源码分析 之 RoleList源码解析 【6】 yunzhonghefei Java集合类源码分析RoleList源码解析
    该类继承于ArrayList，针对Role进行了一些扩展。其他方法和ArrayList中基本相同，源码不做针对性分析：看一下类简介：/***代表了一个roles的列表，作为方法setRoles()的参数，去创建一个关联关系，并且尝试在同一个关系中设置多个角色。*ARoleListrepresentsalistofroles(Roleobjects).Itisusedas*parameterwhen
  • 【鸿蒙OH-v5.0源码分析之 Linux Kernel 部分】004 - Kernel 启动引导代码head.S 源码逐行分析 "小夜猫&小懒虫&小财迷"的男人 鸿蒙OH-v5.0源码分析之Uboot+Kernel部分harmonyoslinux华为
    【鸿蒙OH-v5.0源码分析之LinuxKernel部分】004-Kernel启动引导代码head.S源码逐行分析系列文章汇总：《鸿蒙OH-v5.0源码分析之Uboot+Kernel部分】000-文章链接汇总》本文链接：《【鸿蒙OH-v5.0源码分析之LinuxKernel部分】004-Kernel启动引导代码head.S源码逐行分析》head.S主要工作如下：保存内核启动参数,无效化处理器缓存(
  • Spark 组件 GraphX、Streaming 叶域 大数据sparkspark大数据分布式
    Spark组件GraphX、Streaming一、SparkGraphX1.1GraphX的主要概念1.2GraphX的核心操作1.3示例代码1.4GraphX的应用场景二、SparkStreaming2.1SparkStreaming的主要概念2.2示例代码2.3SparkStreaming的集成2.4SparkStreaming的应用场景SparkGraphX用于处理图和图并行计算。Graph
  • Java高并发编程详解系列-深入理解Thread构造 nihui123 高并发Java高并发Java高并发
    上篇分享中主要是对线程的基本概念和基本操作做了一个分享，同时提出了两种常用的创建多线程的方法，当然在后期的分享中也会提及到更多的创建线程的方式，到后期的分享的时候再说。这次主要是深入的理解一下Thread的构造函数，通过构造函数对于Thread有一个更加深入的了解。这里首先提供一个JDK1.6的ThreadAPI截图线程命名规范  从源码分析可以看到在Thread类中默认提供了线程的命名方式，这个
  • 【QT教程】QT6硬件图形界面编程 QT硬件编程 QT性能优化QT原理源码QT界面美化 qtqt6.3qt5c++QT教程
    QT6硬件图形界面编程使用AI技术辅助生成QT界面美化视频课程QT性能优化视频课程QT原理与源码分析视频课程QTQMLC++扩展开发视频课程免费QT视频课程您可以看免费1000+个QT技术视频免费QT视频课程QT统计图和QT数据可视化视频免费看免费QT视频课程QT性能优化视频免费看免费QT视频课程QT界面美化视频免费看1QT6硬件图形界面编程概述1.1QT6硬件图形界面编程简介1.1.1QT6硬件
  • Jetpack LiveData源码分析 xiangxiongfly915 #AndroidJetpack系列JetpackLiveData源码分析
    文章目录JetpackLiveData源码分析前提源码分析注册订阅流程LiveData#observe()LifecycleRegistry#addObserver()ObserverWithState#dispatchEvent()LiveData#removeObserver()LiveData$LifecycleBoundObserver类LifecycleBoundObserver#sho
  • ExoPlayer架构详解与源码分析（17）——TrackSelector 山雨楼 ExoPlayer架构android音视频ExoPlayerMedia3源码
    系列文章目录ExoPlayer架构详解与源码分析（1）——前言ExoPlayer架构详解与源码分析（2）——PlayerExoPlayer架构详解与源码分析（3）——TimelineExoPlayer架构详解与源码分析（4）——整体架构ExoPlayer架构详解与源码分析（5）——MediaSourceExoPlayer架构详解与源码分析（6）——MediaPeriodExoPlayer架构详解与
  • ExoPlayer架构详解与源码分析（12）——Cache 山雨楼 ExoPlayer架构android音视频ExoPlayerMedia3源码
    系列文章目录ExoPlayer架构详解与源码分析（1）——前言ExoPlayer架构详解与源码分析（2）——PlayerExoPlayer架构详解与源码分析（3）——TimelineExoPlayer架构详解与源码分析（4）——整体架构ExoPlayer架构详解与源码分析（5）——MediaSourceExoPlayer架构详解与源码分析（6）——MediaPeriodExoPlayer架构详解与
  • ExoPlayer架构详解与源码分析（7）——SampleQueue(4) 2401_83740189 程序员架构
    long[]newSourceIds=newlong[newCapacity];long[]newOffsets=newlong[newCapacity];long[]newTimesUs=newlong[newCapacity];int[]newFlags=newint[newCapacity];int[]newSizes=newint[newCapacity];CryptoData[]newC
  • 【HarmonyOS】- 常见算法简单写法 数的羊都睡了 HarmonyOSArkTS鸿蒙
    文章目录知识回顾前言源码分析1.冒泡排序2.二分法查找拓展知识时间、空间复杂度总结知识回顾前言常见算法简单写法源码分析1.冒泡排序functionbubbleSort(arr:number[]):number[]{constn=arr.length;for(leti=0;iarr[j+1]){//交换元素consttemp=arr[j];arr[j]=arr[j+1];arr[j+1]=temp;
  • [ docker-ce源码分析系列 ] 修改resolv.conf文件被还原的原因 nangonghen dockerdocker
    1概述：1.1环境版本信息如下：a、操作系统：centos7.6，amd64b、服务器docker版本：v18.09.22resolv.conf文件被还原的现象：容器中的/etc/resolv.conf文件，是由宿主机/var/lib/docker/containers/xxxx/resolv.conf文件挂载。在dockerrestart容器之前，手动修改了/var/lib/docker/con
  • dispatch_once源码分析 福伟_Y
    GCD里的单例函数dispatch_once是我们经常会用到的，今天我们来稍做深入分析一下。GCD的源码都在libdispatch.dylib库里，这个库在libSystem_initializer被初始化，可理解为在dyld里被加载和初始化的(之前的文章有分析过)。dispatch_once作为单例的使用入口，通过分析得到它是一个宏定义，_dispatch_once函数在libdispatch.
  • 分布式离线计算—Spark—基础介绍 测试开发abbey 人工智能—大数据
    原文作者：饥渴的小苹果原文地址：【Spark】Spark基础教程目录Spark特点Spark相对于Hadoop的优势Spark生态系统Spark基本概念Spark结构设计Spark各种概念之间的关系Executor的优点Spark运行基本流程Spark运行架构的特点Spark的部署模式Spark三种部署方式Hadoop和Spark的统一部署摘要：Spark是基于内存计算的大数据并行计算框架Spar
  • jQuery Easyui 源码分析之combo组件 90后北京程序员 前端-easyuieasyui之combobox
    /***jQueryEasyUI1.3.1*该源码完全由压缩码翻译而来，并非网络上放出的源码，请勿索要。*/(function($){functionsetSize(target,width){varopts=$.data(target,"combo").options;varcombo=$.data(target,"combo").combo;varpanel=$.data(target,"co
  • html5carousel图片轮播,全面解析Bootstrap中Carousel轮播的使用方法 RemusrickCat
    本文实例为大家全面的解析了Bootstrap中Carousel的使用方法，供大家参考，具体内容如下源码文件：Carousel.scssCarousel.js实现原理：隐藏所有要显示的元素，然后指定当前要显示的为block，宽、高自适应源码分析：1、Html结构：主要分为以四个部分1.1、容器：最外层div，需要一个data-ride=”carousel”来指定为轮播放插件，并且提供一个Id，方便圆
  • Java集合-----List接口及其实现类：ArrayList、LinkedList、Vector Colourful． Java集合java集合
    文章目录List接口概述List接口的常用方法List接口的实现类ArrayList源码分析类继承关系ArrayList中的属性：ArrayList构造函数：ArrayList中常用方法添加操作：add()删除操作：remove()获取元素：get()ArrayList是如何实现序列化的？ArrayList的总结LinkedList源码分析类继承关系类成员属性类构造器LinkedList的List
  • React Native通讯原理 zbl_zbl androidReactNativ
    之前写过一篇文章ReactNativeAndroid源码分析，在此文章的基础上分析和总结下RN与Native的通讯流程。本文基于Android代码分析，iOS实现原理类似。1.通讯框架图先来解析下各个模块的角色与作用：Java层，这块的实现在ReactAndroid中-ReactContext:Android上下文子类，包含一个CatalystInstance实例，用于获取NativeModule
  • python ray分布式_取代 Python 多进程！伯克利开源分布式框架 Ray weixin_39946313 pythonray分布式
    Ray由伯克利开源，是一个用于并行计算和分布式Python开发的开源项目。本文将介绍如何使用Ray轻松构建可从笔记本电脑扩展到大型集群的应用程序。并行和分布式计算是现代应用程序的主要内容。我们需要利用多个核心或多台机器来加速应用程序或大规模运行它们。网络爬虫和搜索所使用的基础设施并不是在某人笔记本电脑上运行的单线程程序，而是相互通信和交互的服务的集合。云计算承诺在所有维度上(内存、计算、存储等)实
  • OPENCL之SIMT与SIMD在架构上的主要区别是什么？ 糯米宝宝 gpuopencv
    SIMT（单指令多线程）与SIMD（单指令多数据）在架构上的主要区别体现在以下几个方面：执行单元的组织方式：SIMD：采用的是多数据流架构，即同一条指令同时作用于多个数据元素。这种架构特别适合于多媒体应用等数据密集型运算。SIMT：采用的是多线程架构，即同一条指令由多个线程并行执行。每个线程可以有不同的分支行为和执行路径，从而实现线程级的并行计算。软件暴露的信息：SIMD：向软件公开SIMD宽度（
  • [linux 驱动]增加一个文件节点控制led灯亮灭 嵌入式成长家 linux内核的系统实战linux驱动linux驱动led灯驱动
    目录1修改设备树2修改驱动3驱动源码3.1驱动源码3.2设备树节点3.3驱动源码分析3.3.1##解释3.3.2class_create解释3.3.3class_create_file解释3.3.4of_get_named_gpio_flags解释3.3.5devm_gpio_request解释3.3.6platform_driver_register解释3.3.7platform_driver_
  • 【QT教程】QT6对话框编程 QT性能优化QT原理源码QT界面美化 qtqt6.3qt5c++QT教程
    QT6对话框编程使用AI技术辅助生成QT界面美化视频课程QT性能优化视频课程QT原理与源码分析视频课程QTQMLC++扩展开发视频课程免费QT视频课程您可以看免费1000+个QT技术视频免费QT视频课程QT统计图和QT数据可视化视频免费看免费QT视频课程QT性能优化视频免费看免费QT视频课程QT界面美化视频免费看1QT6对话框编程基础1.1QT6对话框简介1.1.1QT6对话框简介QT6对话框简介
  • Python | 使用Joblib模块加快任务处理速度 python收藏家 pythonpython
    在本文中，我们将了解如何通过使用Joblib模块在Python中并行执行代码来大幅减少大型代码的执行时间。Joblib模块简介Joblib是一个用于Python的开源库，它提供了一些用于并行计算和内存映射的工具，旨在提高科学计算和数据分析的效率。Python中的Joblib模块特别用于使用Pipelines并行执行任务，而不是一个接一个地顺序执行任务。Joblib模块允许用户通过利用设备中存在的所
  • 鸿蒙轻内核M核源码分析系列十二 事件Event OpenHarmony_小贾 OpenHarmonyHarmonyOS鸿蒙开发harmonyosopenharmony鸿蒙内核鸿蒙开发移动开发嵌入式硬件驱动开发
    事件（Event）是一种任务间通信的机制，可用于任务间的同步。多任务环境下，任务之间往往需要同步操作，一个等待即是一个同步。事件可以提供一对多、多对多的同步操作。本文通过分析鸿蒙轻内核事件模块的源码，深入掌握事件的使用。本文中所涉及的源码，以OpenHarmonyLiteOS-M内核为例，均可以在开源站点https://gitee.com/openharmony/kernel_liteos_m获取
  • 鸿蒙轻内核M核源码分析系列四 中断Hwi OpenHarmony_小贾 鸿蒙开发OpenHarmonyHarmonyOSharmonyos单片机OpenHarmony嵌入式硬件鸿蒙开发移动开发鸿蒙内核
    在鸿蒙轻内核源码分析系列前几篇文章中，剖析了重要的数据结构。本文，我们讲述一下中断，会给读者介绍中断的概念，鸿蒙轻内核的中断模块的源代码。本文中所涉及的源码，以OpenHarmonyLiteOS-M内核为例。1、中断概念介绍中断是指出现需要时，CPU暂停执行当前程序，转而执行新程序的过程。当外设需要CPU时，将通过产生中断信号使CPU立即中断当前任务来响应中断请求。在剖析中断源代码之前，下面介绍些
  • 鸿蒙轻内核M核源码分析系列五 时间管理 OpenHarmony_小贾 HarmonyOSOpenHarmony鸿蒙开发harmonyosopenharmony鸿蒙开发NAPI鸿蒙内核移动开发嵌入式
    在鸿蒙轻内核源码分析上一篇文章中，我们剖析了中断的源码，简单提到了Tick中断。本文会继续分析Tick和时间相关的源码，给读者介绍鸿蒙轻内核的时间管理模块。本文中所涉及的源码，以OpenHarmonyLiteOS-M内核为例，均可以在开源站点https://gitee.com/openharmony/kernel_liteos_m获取。时间管理模块以系统时钟为基础，可以分为2部分，一部分是SysT
  • 鸿蒙轻内核M核源码分析系列六 任务及任务调度（1）任务栈 OpenHarmony_小贾 鸿蒙开发OpenHarmonyHarmonyOSHarmonyOSopenharmony鸿蒙开发移动开发鸿蒙内核驱动开发嵌入式硬件
    继续分析鸿蒙轻内核源码，我们本文开始要分析下任务及任务调度模块。首先，我们介绍下任务栈的基础概念。任务栈是高地址向低地址生长的递减栈，栈指针指向即将入栈的元素位置。初始化后未使用过的栈空间初始化的内容为宏OS_TASK_STACK_INIT代表的数值0xCACACACA，栈顶初始化为宏OS_TASK_MAGIC_WORD代表的数值0xCCCCCCCC。一个任务栈的示意图如下，其中，栈底指针是栈的最
  • 鸿蒙轻内核M核源码分析系列三 数据结构-任务排序链表 OpenHarmony_小贾 HarmonyOSOpenHarmony鸿蒙开发数据结构harmonyos移动开发OpenHarmony鸿蒙内核鸿蒙开发嵌入式硬件
    在鸿蒙轻内核源码分析系列一和系列二，我们分析了双向循环链表、优先级就绪队列的源码。本文会继续给读者介绍鸿蒙轻内核源码中重要的数据结构：任务排序链表TaskSortLinkAttr。鸿蒙轻内核的任务排序链表，用于任务延迟到期/超时唤醒等业务场景，是一个非常重要、非常基础的数据结构。本文中所涉及的源码，以OpenHarmonyLiteOS-M内核为例。1任务排序链表我们先看下任务排序链接的数据结构。任
  • 鸿蒙轻内核A核源码分析系列七 进程管理 (2) OpenHarmony_小贾 HarmonyOS鸿蒙开发OpenHarmonyharmonyosOpenHarmony移动开发驱动开发鸿蒙内核LiteOS-A内核进程通信
    本文先熟悉下进程管理的文件kernel\base\core\los_process.c中的内部接口，读读代码，做些记录。1、LiteOS-A内核进程全局变量⑴是进程池，存放各个进程控制块LosProcessCB的信息。⑵处开始的g_freeProcess是空闲进程链表，挂载各个空闲进程控制块；g_processRecycleList是待回收进程控制块链表，挂载各个等待回收的进程控制块。⑶处开始的g
  • 如何用ruby来写hadoop的mapreduce并生成jar包 wudixiaotie mapreduce
    ruby来写hadoop的mapreduce，我用的方法是rubydoop。怎么配置环境呢： 1.安装rvm：     不说了 网上有 2.安装ruby：     由于我以前是做ruby的，所以习惯性的先安装了ruby，起码调试起来比jruby快多了。 3.安装jruby：     rvm install jruby然后等待安
  • java编程思想 -- 访问控制权限 百合不是茶 java访问控制权限单例模式
    访问权限是java中一个比较中要的知识点,它规定者什么方法可以访问,什么不可以访问   一:包访问权限;   自定义包: package com.wj.control; //包 public class Demo { //定义一个无参的方法 public void DemoPackage(){ System.out.println("调用
  • [生物与医学]请审慎食用小龙虾 comsci 生物
         现在的餐馆里面出售的小龙虾,有一些是在野外捕捉的,这些小龙虾身体里面可能带有某些病毒和细菌,人食用以后可能会导致一些疾病,严重的甚至会死亡.....      所以,参加聚餐的时候,最好不要点小龙虾...就吃养殖的猪肉,牛肉,羊肉和鱼,等动物蛋白质     
  • org.apache.jasper.JasperException: Unable to compile class for JSP: 商人shang maven2.2jdk1.8
    环境： jdk1.8    maven  tomcat7-maven-plugin  2.0 原因： tomcat7-maven-plugin  2.0 不知吃 jdk 1.8，换成 tomcat7-maven-plugin  2.2就行，即     <plugin>
  • 你的垃圾你处理掉了吗?GC oloz GC
    前序:本人菜鸟，此文研究学习来自网络，各位牛牛多指教　 1.垃圾收集算法的核心思想 　　Java语言建立了垃圾收集机制，用以跟踪正在使用的对象和发现并回收不再使用(引用)的对象。该机制可以有效防范动态内存分配中可能发生的两个危险：因内存垃圾过多而引发的内存耗尽，以及不恰当的内存释放所造成的内存非法引用。 　　垃圾收集算法的核心思想是：对虚拟机可用内存空间，即堆空间中的对象进行识别
  • shiro 和 SESSSION 杨白白 shiro
    shiro 在web项目里默认使用的是web容器提供的session，也就是说shiro使用的session是web容器产生的，并不是自己产生的，在用于非web环境时可用其他来源代替。在web工程启动的时候它就和容器绑定在了一起，这是通过web.xml里面的shiroFilter实现的。通过session.getSession()方法会在浏览器cokkice产生JESSIONID，当关闭浏览器，此
  • 移动互联网终端 淘宝客如何实现盈利 小桔子 移動客戶端淘客淘寶App
           2012年淘宝联盟平台为站长和淘宝客带来的分成收入突破30亿元，同比增长100%。而来自移动端的分成达1亿元，其中美丽说、蘑菇街、果库、口袋购物等App运营商分成近5000万元。 可以看出，虽然目前阶段PC端对于淘客而言仍旧是盈利的大头，但移动端已经呈现出爆发之势。而且这个势头将随着智能终端(手机，平板)的加速普及而更加迅猛
  • wordpress小工具制作 aichenglong wordpress小工具
    wordpress 使用侧边栏的小工具，很方便调整页面结构 小工具的制作过程 1 在自己的主题文件中新建一个文件夹(如widget)，在文件夹中创建一个php(AWP_posts-category.php) 小工具是一个类,想侧边栏一样，还得使用代码注册，他才可以再后台使用，基本的代码一层不变 <?php class AWP_Post_Category extends WP_Wi
  • JS微信分享 AILIKES js
    // 所有功能必须包含在 WeixinApi.ready 中进行    WeixinApi.ready(function(Api) {        // 微信分享的数据            var wxData = {       &nb
  • 封装探讨 百合不是茶 JAVA面向对象 封装
    //封装   属性 方法 将某些东西包装在一起，通过创建对象或使用静态的方法来调用，称为封装；封装其实就是有选择性地公开或隐藏某些信息，它解决了数据的安全性问题，增加代码的可读性和可维护性   在 Aname类中申明三个属性，将其封装在一个类中：通过对象来调用   例如   1： //属性 将其设为私有 姓名 name 可以公开
  • jquery radio/checkbox change事件不能触发的问题 bijian1013 JavaScriptjquery
    我想让radio来控制当前我选择的是机动车还是特种车，如下所示：  <html> <head> <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"><
  • AngularJS中安全性措施 bijian1013 JavaScriptAngularJS安全性XSRFJSON漏洞
            在使用web应用中，安全性是应该首要考虑的一个问题。AngularJS提供了一些辅助机制，用来防护来自两个常见攻击方向的网络攻击。 一.JSON漏洞         当使用一个GET请求获取JSON数组信息的时候（尤其是当这一信息非常敏感，
  • [Maven学习笔记九]Maven发布web项目 bit1129 maven
    基于Maven的web项目的标准项目结构 user-project     user-core     user-service     user-web        src      
  • 【Hive七】Hive用户自定义聚合函数(UDAF) bit1129 hive
    用户自定义聚合函数，用户提供的多个入参通过聚合计算(求和、求最大值、求最小值)得到一个聚合计算结果的函数。 问题：UDF也可以提供输入多个参数然后输出一个结果的运算，比如加法运算add(3，5)，add这个UDF需要实现UDF的evaluate方法,那么UDF和UDAF的实质分别究竟是什么？   Double evaluate(Double a, Double b)  
  • 通过 nginx-lua 给 Nginx 增加 OAuth 支持 ronin47
    前言：我们使用Nginx的Lua中间件建立了OAuth2认证和授权层。如果你也有此打算，阅读下面的文档，实现自动化并获得收益。SeatGeek 在过去几年中取得了发展，我们已经积累了不少针对各种任务的不同管理接口。我们通常为新的展示需求创建新模块，比如我们自己的博客、图表等。我们还定期开发内部工具来处理诸如部署、可视化操作及事件处理等事务。在处理这些事务中，我们使用了几个不同的接口来认证： &n
  • 利用tomcat-redis-session-manager做session同步时自定义类对象属性保存不上的解决方法 bsr1983 session
    在利用tomcat-redis-session-manager做session同步时，遇到了在session保存一个自定义对象时，修改该对象中的某个属性，session未进行序列化，属性没有被存储到redis中。 在 tomcat-redis-session-manager的github上有如下说明： Session Change Tracking As noted in the &qu
  • 《代码大全》表驱动法-Table Driven Approach-1 bylijinnan java算法
    关于Table Driven Approach的一篇非常好的文章： http://www.codeproject.com/Articles/42732/Table-driven-Approach package com.ljn.base; import java.util.Random; public class TableDriven { public
  • Sybase封锁原理 chicony Sybase
           昨天在操作Sybase IQ12.7时意外操作造成了数据库表锁定，不能删除被锁定表数据也不能往其中写入数据。由于着急往该表抽入数据，因此立马着手解决该表的解锁问题。     无奈此前没有接触过Sybase IQ12.7这套数据库产品，加之当时已属于下班时间无法求助于支持人员支持，因此只有借助搜索引擎强大的
  • java异常处理机制 CrazyMizzz java
    java异常关键字有以下几个，分别为 try catch final throw throws 他们的定义分别为 try：    Opening exception-handling statement. catch：  Captures the exception. finally： Runs its code before terminating
  • hive 数据插入DML语法汇总 daizj hiveDML数据插入
    Hive的数据插入DML语法汇总1、Loading files into tables语法：1) LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]解释：1)、上面命令执行环境为hive客户端环境下： hive>l
  • 工厂设计模式 dcj3sjt126com 设计模式
      使用设计模式是促进最佳实践和良好设计的好办法。设计模式可以提供针对常见的编程问题的灵活的解决方案。 工厂模式 工厂模式（Factory）允许你在代码执行时实例化对象。它之所以被称为工厂模式是因为它负责“生产”对象。工厂方法的参数是你要生成的对象对应的类名称。 Example #1 调用工厂方法（带参数） <?phpclass Example{ 
  • mysql字符串查找函数 dcj3sjt126com mysql
      FIND_IN_SET(str,strlist) 假如字符串str 在由N 子链组成的字符串列表strlist 中，则返回值的范围在1到 N 之间。一个字符串列表就是一个由一些被‘,’符号分开的自链组成的字符串。如果第一个参数是一个常数字符串，而第二个是type SET列，则   FIND_IN_SET() 函数被优化，使用比特计算。如果str不在strlist 或st
  • jvm内存管理 easterfly jvm
    一、JVM堆内存的划分 分为年轻代和年老代。年轻代又分为三部分：一个eden,两个survivor。 工作过程是这样的：e区空间满了后，执行minor gc，存活下来的对象放入s0, 对s0仍会进行minor gc，存活下来的的对象放入s1中，对s1同样执行minor gc，依旧存活的对象就放入年老代中； 年老代满了之后会执行major gc，这个是stop the word模式，执行
  • CentOS-6.3安装配置JDK-8 gengzg centos
    JAVA_HOME=/usr/java/jdk1.8.0_45 JRE_HOME=/usr/java/jdk1.8.0_45/jre PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib export JAVA_HOME
  • 【转】关于web路径的获取方法 huangyc1210 Web路径
    假定你的web application 名称为news,你在浏览器中输入请求路径：  http://localhost:8080/news/main/list.jsp  则执行下面向行代码后打印出如下结果：  1、 System.out.println(request.getContextPath()); //可返回站点的根路径。也就是项
  • php里获取第一个中文首字母并排序 远去的渡口 数据结构PHP
    很久没来更新博客了，还是觉得工作需要多总结的好。今天来更新一个自己认为比较有成就的问题吧。 最近在做储值结算，需求里结算首页需要按门店的首字母A-Z排序。我的数据结构原本是这样的： Array ( [0] => Array ( [sid] => 2885842 [recetcstoredpay] =&g
  • java内部类 hm4123660 java内部类匿名内部类成员内部类方法内部类
         　在Java中，可以将一个类定义在另一个类里面或者一个方法里面，这样的类称为内部类。内部类仍然是一个独立的类，在编译之后内部类会被编译成独立的.class文件，但是前面冠以外部类的类名和$符号。内部类可以间接解决多继承问题,可以使用内部类继承一个类，外部类继承一个类，实现多继承。      &nb
  • Caused by: java.lang.IncompatibleClassChangeError: class org.hibernate.cfg.Exten zhb8015
    maven pom.xml关于hibernate的配置和异常信息如下，查了好多资料，问题还是没有解决。只知道是包冲突，就是不知道是哪个包....遇到这个问题的分享下是怎么解决的。。   maven pom:   <dependency> <groupId>org.hibernate</groupId> <ar
  • Spark 性能相关参数配置详解－任务调度篇 Stark_Summer sparkcachecpu任务调度yarn
    随着Spark的逐渐成熟完善, 越来越多的可配置参数被添加到Spark中来, 本文试图通过阐述这其中部分参数的工作原理和配置思路, 和大家一起探讨一下如何根据实际场合对Spark进行配置优化。   由于篇幅较长，所以在这里分篇组织，如果要看最新完整的网页版内容，可以戳这里：http://spark-config.readthedocs.org/，主要是便
  • css3滤镜 wangkeheng htmlcss
    经常看到一些网站的底部有一些灰色的图标，鼠标移入的时候会变亮，开始以为是js操作src或者bg呢，搜索了一下，发现了一个更好的方法：通过css3的滤镜方法。 html代码： <a href='' class='icon'><img src='utv.jpg' /></a> css代码： .icon{-webkit-filter: graysc
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他