MasterT-J
MasterT-J

Apache Storm 实时流处理系统通信机制源码分析

        我们今天就来仔细研究一下Apache Storm 2.0.0-SNAPSHOT的通信机制。下面我将从大致思想以及源码分析，然后我们细致分析实时流处理系统中源码通信机制研究。

1. 简介

        Worker间的通信经常需要通过网络跨节点进行，Storm使用ZeroMQ或Netty(0.9以后默认使用)作为进程间通信的消息框架。

Worker进程内部通信：不同worker的thread通信使用LMAX Disruptor来完成。

不同topologey之间的通信，Storm不负责，需要自己想办法实现，例如使用kafka等；

Worker进程间消息传递机制，消息的接收和处理的大概流程见下图：

Apache Storm 实时流处理系统通信机制源码分析_第1张图片

        实时流处理系统一般都有一个Worker进程用来分配资源，Worker进程是资源分配的最小单位。每个Worker进程中又包含多个Executor，Executor是用来真正执行Task的组件，里面包含一个工作线程和发送线程。每个Executor都有自己的接收队列和发送队列。
1.每个Worker进程都有一个单独的接受线程监听接收端口。Worker接收线程将收到的消息通过Task编号传递给对应的Executor(一个或则多个)接收队列。Worker接收线程将每个从网络上传来的消息发送到相应的Executor的接收队列中。Executor接收队列存放Worker或者Worker内部其他xecutor发送过来的消息。
2. Executor工作线程从接收队列中拿出数据，然后调用execute方法，发送Tuple到Executor的发送队列。
3. Executor的发送线程从发送队列中获取消息，按照消息目的地址选择发送到Worker的传输队列中或者其他Executor的接收队列中。

4. 最后Worker的发送线程从传输队列中读取消息，然后将Tuple元组发送到网络中。

1.1 具体细节

       对于worker进程来说，为了管理流入和传出的消息，每个worker进程有一个独立的接收线程(对配置的TCP端口supervisor.slots.ports进行监听);

        每个executor有自己的sendQueue和receiveQueue。

        Worker接收线程将收到的消息通过task编号传递给对应的executor(一个或多个)的receiveQueue;

        每个executor有单独的线程分别来处理spout/bolt的业务逻辑，业务逻辑输出的中间数据会存放在sendQueue中，当executor的sendQueue中的tuple达到一定的阀值，executor的发送线程将批量获取sendQueue中的tuple,并发送到TransferQueue中。

        每个worker进程控制一个或多个executor线程，用户可在代码中进行配置。其实就是我们在代码中设置的并发度个数。

        一个worker进程运行一个专用的接收线程来负责将外部发送过来的消息移动到对应的executor线程的receiveQueue中

        TransferQueue的大小由参数topology.transfer.buffer.size来设置。TransferQueue的每个元素实际上代表一个tuple的集合
        TransferQueue的大小由参数topology.transfer.buffer.size来设置。
        executor的sendQueue的大小用户可以自定义配置。

        executor的receiveQueue的大小用户可以自定义配置

1.2 Worker进程间通信分析

Apache Storm 实时流处理系统通信机制源码分析_第2张图片

1、 Worker接受线程通过网络接受数据，并根据Tuple中包含的taskId，匹配到对应的executor；然后根据executor找到对应的incoming-queue，将数据存发送到receiveQueue队列中。
2、 业务逻辑执行现成消费receiveQueue的数据，通过调用Bolt的execute(xxxx)方法，将Tuple作为参数传输给用户自定义的方法
3、 业务逻辑执行完毕之后，将计算的中间数据发送给sendQueue队列，当sendQueue中的tuple达到一定的阀值，executor的发送线程将批量获取sendQueue中的tuple,并发送到Worker的TransferQueue中
4、 Worker发送线程消费TransferQueue中数据，计算Tuple的目的地，连接不同的node+port将数据通过网络传输的方式传送给另一个的Worker。
5、 另一个worker执行以上步骤1的操作。

2. Storm 通信机制源码分析

2.1 Spout/Bolt 发送Tuple数据

        SpoutOutputCollector 调用emit方法，然后在emit方法中调用sendSpoutMsg方法。我们仔细分析sendSpoutMsg方法，首先根据emit方法中指定的stream和values，调用taskData的taskData.getOutgoingTasks(stream, values)的方法，获取数据要发往哪个TaskID（根据上游spout和下游bolt之间的分组信息）。然后根据这个TaskID，循环遍历将数据封装成TupleImpl。然后outputCollector通过调用executor的ExecutorTransfer类的transfer方法(）将tuple添加目标taskId信息，封装成AddressTuple，将数据发送到相应的目标Task。

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.executor.spout;

public class SpoutOutputCollectorImpl implements ISpoutOutputCollector {

    private final SpoutExecutor executor;
    private final Task taskData;
    private final int taskId;
    private final MutableLong emittedCount;
    private final boolean hasAckers;
    private final Random random;
    private final Boolean isEventLoggers;
    private final Boolean isDebug;
    private final RotatingMap pending;

    @SuppressWarnings("unused")
    public SpoutOutputCollectorImpl(ISpout spout, SpoutExecutor executor, Task taskData, int taskId,
                                    MutableLong emittedCount, boolean hasAckers, Random random,
                                    Boolean isEventLoggers, Boolean isDebug, RotatingMap pending) {
        this.executor = executor;
        this.taskData = taskData;
        this.taskId = taskId;
        this.emittedCount = emittedCount;
        this.hasAckers = hasAckers;
        this.random = random;
        this.isEventLoggers = isEventLoggers;
        this.isDebug = isDebug;
        this.pending = pending;
    }

    @Override
    public List emit(String streamId, List tuple, Object messageId) {
        return sendSpoutMsg(streamId, tuple, messageId, null);
    }

    @Override
    public void emitDirect(int taskId, String streamId, List tuple, Object messageId) {
        sendSpoutMsg(streamId, tuple, messageId, taskId);
    }

    @Override
    public long getPendingCount() {
        return pending.size();
    }

    @Override
    public void reportError(Throwable error) {
        executor.getErrorReportingMetrics().incrReportedErrorCount();
        executor.getReportError().report(error);
    }

    /**
     * 1. 首先Spout调用sendSpoutMsg() 发送一个tuple到下游bolt
     * @param stream
     * @param values
     * @param messageId
     * @param outTaskId
     * @return
     */
    private List sendSpoutMsg(String stream, List values, Object messageId, Integer outTaskId) {
        emittedCount.increment();

        List outTasks;
        if (outTaskId != null) {
            outTasks = taskData.getOutgoingTasks(outTaskId, stream, values);
        } else {
            outTasks = taskData.getOutgoingTasks(stream, values);
        }

        List ackSeq = new ArrayList<>();
        boolean needAck = (messageId != null) && hasAckers;

        long rootId = MessageId.generateId(random);
        //2.根据上游spout和下游bolt之间的分组信息，将tuple发送到下游相应的task中，并且封装成TupleImpl类
        for (Integer t : outTasks) {
            MessageId msgId;
            if (needAck) {
                long as = MessageId.generateId(random);
                msgId = MessageId.makeRootId(rootId, as);
                ackSeq.add(as);
            } else {
                msgId = MessageId.makeUnanchored();
            }

            TupleImpl tuple = new TupleImpl(executor.getWorkerTopologyContext(), values, this.taskId, stream, msgId);
            //3.outputCollector调用executor的ExecutorTransfer类的transfer方法(）将tuple添加目标taskId信息，封装成AddressTuple
            executor.getExecutorTransfer().transfer(t, tuple);
        }
        if (isEventLoggers) {
            executor.sendToEventLogger(executor, taskData, values, executor.getComponentId(), messageId, random);
        }

        boolean sample = false;
        try {
            sample = executor.getSampler().call();
        } catch (Exception ignored) {
        }
        if (needAck) {
            TupleInfo info = new TupleInfo();
            info.setTaskId(this.taskId);
            info.setStream(stream);
            info.setMessageId(messageId);
            if (isDebug) {
                info.setValues(values);
            }
            if (sample) {
                info.setTimestamp(System.currentTimeMillis());
            }

            pending.put(rootId, info);
            List ackInitTuple = new Values(rootId, Utils.bitXorVals(ackSeq), this.taskId);
            executor.sendUnanchored(taskData, Acker.ACKER_INIT_STREAM_ID, ackInitTuple, executor.getExecutorTransfer());
        } else if (messageId != null) {
            TupleInfo info = new TupleInfo();
            info.setStream(stream);
            info.setValues(values);
            info.setMessageId(messageId);
            info.setTimestamp(0);
            Long timeDelta = sample ? 0L : null;
            info.setId("0:");
            executor.ackSpoutMsg(executor, taskData, timeDelta, info);
        }

        return outTasks;
    }
} 
  
 
  
 
  
        BoltOutputCollector 调用emit方法，然后在emit方法中调用BoltEmit方法。我们仔细分析BoltEmit方法，首先根据emit方法中指定的stream和values，调用taskData的taskData.getOutgoingTasks(stream, values)的方法，获取数据要发往哪个TaskID。然后根据这个TaskID，循环遍历将数据封装成TupleImpl。然后再调用executor.getExecutorTransfer().transfer(t, tuple) 将数据发送到相应的目标Task。
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.executor.bolt;

public class BoltOutputCollectorImpl implements IOutputCollector {

    private static final Logger LOG = LoggerFactory.getLogger(BoltOutputCollectorImpl.class);

    private final BoltExecutor executor;
    private final Task taskData;
    private final int taskId;
    private final Random random;
    private final boolean isEventLoggers;
    private final boolean isDebug;

    public BoltOutputCollectorImpl(BoltExecutor executor, Task taskData, int taskId, Random random,
                                   boolean isEventLoggers, boolean isDebug) {
        this.executor = executor;
        this.taskData = taskData;
        this.taskId = taskId;
        this.random = random;
        this.isEventLoggers = isEventLoggers;
        this.isDebug = isDebug;
    }

    public List emit(String streamId, Collection anchors, List tuple) {
        return boltEmit(streamId, anchors, tuple, null);
    }

    @Override
    public void emitDirect(int taskId, String streamId, Collection anchors, List tuple) {
        boltEmit(streamId, anchors, tuple, taskId);
    }

    /**
     * 1.首先Bolt调用boltEmit() 发送一个tuple到下游bolt
     * @param streamId
     * @param anchors
     * @param values
     * @param targetTaskId
     * @return
     */
    private List boltEmit(String streamId, Collection anchors, List values, Integer targetTaskId) {
        List outTasks;
        if (targetTaskId != null) {
            outTasks = taskData.getOutgoingTasks(targetTaskId, streamId, values);
        } else {
            outTasks = taskData.getOutgoingTasks(streamId, values);
        }

        for (Integer t : outTasks) {
            Map anchorsToIds = new HashMap<>();
            if (anchors != null) {
                for (Tuple a : anchors) {
                    Set rootIds = a.getMessageId().getAnchorsToIds().keySet();
                    if (rootIds.size() > 0) {
                        long edgeId = MessageId.generateId(random);
                        ((TupleImpl) a).updateAckVal(edgeId);
                        for (Long root_id : rootIds) {
                            putXor(anchorsToIds, root_id, edgeId);
                        }
                    }
                }
            }
            MessageId msgId = MessageId.makeId(anchorsToIds);
            TupleImpl tupleExt = new TupleImpl(executor.getWorkerTopologyContext(), values, taskId, streamId, msgId);
            executor.getExecutorTransfer().transfer(t, tupleExt);
        }
        if (isEventLoggers) {
            executor.sendToEventLogger(executor, taskData, values, executor.getComponentId(), null, random);
        }
        return outTasks;
    }

    @Override
    public void ack(Tuple input) {
        long ackValue = ((TupleImpl) input).getAckVal();
        Map anchorsToIds = input.getMessageId().getAnchorsToIds();
        for (Map.Entry entry : anchorsToIds.entrySet()) {
            executor.sendUnanchored(taskData, Acker.ACKER_ACK_STREAM_ID,
                    new Values(entry.getKey(), Utils.bitXor(entry.getValue(), ackValue)),
                    executor.getExecutorTransfer());
        }
        long delta = tupleTimeDelta((TupleImpl) input);
        if (isDebug) {
            LOG.info("BOLT ack TASK: {} TIME: {} TUPLE: {}", taskId, delta, input);
        }
        BoltAckInfo boltAckInfo = new BoltAckInfo(input, taskId, delta);
        boltAckInfo.applyOn(taskData.getUserContext());
        if (delta >= 0) {
            ((BoltExecutorStats) executor.getStats()).boltAckedTuple(
                    input.getSourceComponent(), input.getSourceStreamId(), delta);
        }
    }

    @Override
    public void fail(Tuple input) {
        Set roots = input.getMessageId().getAnchors();
        for (Long root : roots) {
            executor.sendUnanchored(taskData, Acker.ACKER_FAIL_STREAM_ID,
                    new Values(root), executor.getExecutorTransfer());
        }
        long delta = tupleTimeDelta((TupleImpl) input);
        if (isDebug) {
            LOG.info("BOLT fail TASK: {} TIME: {} TUPLE: {}", taskId, delta, input);
        }
        BoltFailInfo boltFailInfo = new BoltFailInfo(input, taskId, delta);
        boltFailInfo.applyOn(taskData.getUserContext());
        if (delta >= 0) {
            ((BoltExecutorStats) executor.getStats()).boltFailedTuple(
                    input.getSourceComponent(), input.getSourceStreamId(), delta);
        }
    }

    @Override
    public void resetTimeout(Tuple input) {
        Set roots = input.getMessageId().getAnchors();
        for (Long root : roots) {
            executor.sendUnanchored(taskData, Acker.ACKER_RESET_TIMEOUT_STREAM_ID,
                    new Values(root), executor.getExecutorTransfer());
        }
    }

    @Override
    public void reportError(Throwable error) {
        executor.getErrorReportingMetrics().incrReportedErrorCount();
        executor.getReportError().report(error);
    }

    private long tupleTimeDelta(TupleImpl tuple) {
        Long ms = tuple.getProcessSampleStartTime();
        if (ms != null) {
            return Time.deltaMs(ms);
        }
        return -1;
    }

    private void putXor(Map pending, Long key, Long id) {
        Long curr = pending.get(key);
        if (curr == null) {
            curr = 0l;
        }
        pending.put(key, Utils.bitXor(curr, id));
    }
}
 
  
2.2 ExecutorTransfer类调用transfer方法发送Tuple
 
  
        ExecutorTransfer类是Executor类的一个成员变量，用来将发送Executor中的Task数据的。在这里，ExecutorTransfer将tuple添加目标task信息，将tuple封装成AddressedTuple。并将封装后的结果AddressedTuple publish到batchTransferQueue队列中。这个batchTransferQueue就说Executor中的发送队列sendQueue。
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.executor;

public class ExecutorTransfer implements EventHandler, Callable {
    private static final Logger LOG = LoggerFactory.getLogger(ExecutorTransfer.class);

    private final WorkerState workerData;
    private final DisruptorQueue batchTransferQueue;
    private final Map topoConf;
    private final KryoTupleSerializer serializer;
    private final MutableObject cachedEmit;
    private final boolean isDebug;

    public ExecutorTransfer(WorkerState workerData, DisruptorQueue batchTransferQueue, Map topoConf) {
        this.workerData = workerData;
        this.batchTransferQueue = batchTransferQueue;
        this.topoConf = topoConf;
        this.serializer = new KryoTupleSerializer(topoConf, workerData.getWorkerTopologyContext());
        this.cachedEmit = new MutableObject(new ArrayList<>());
        this.isDebug = ObjectReader.getBoolean(topoConf.get(Config.TOPOLOGY_DEBUG), false);
    }

    //4.ExecutorTransfer将tuple添加目标task信息，将tuple封装成AddressedTuple。并将封装后的结果AddressedTuple publish到batchTransferQueue队列中。
    // batchTransferQueue也就是Executor的发送队列。
    public void transfer(int task, Tuple tuple) {
        AddressedTuple val = new AddressedTuple(task, tuple);
        if (isDebug) {
            LOG.info("TRANSFERRING tuple {}", val);
        }
        batchTransferQueue.publish(val);
    }


    @VisibleForTesting
    public DisruptorQueue getBatchTransferQueue() {
        return this.batchTransferQueue;
    }

    /**
     * 6.ExecutorTransfer的Call方法被调用。batchTransferQueue批量的消费消息
     * @return
     * @throws Exception
     */
    @Override
    public Object call() throws Exception {
        batchTransferQueue.consumeBatchWhenAvailable(this);
        return 0L;
    }

    public String getName() {
        return batchTransferQueue.getName();
    }

    /**
     * 7.相应事件，不断的批量消费batchTransferQueue中的AddressedTuple对象
     * @param event
     * @param sequence
     * @param endOfBatch
     * @throws Exception
     */
    @Override
    public void onEvent(Object event, long sequence, boolean endOfBatch) throws Exception {
        ArrayList cachedEvents = (ArrayList) cachedEmit.getObject();
        cachedEvents.add(event);
        if (endOfBatch) {
            //8.调用WorkerState的transfer方法。对AddressedTuple进行序列化操作
            workerData.transfer(serializer, cachedEvents);
            cachedEmit.setObject(new ArrayList<>());
        }
    }
}

 
  
2.3 Executor 中发送队列达到一定的阈值后，就开始调用WorkerState的transfer方法
 
  
        Executor是Storm中正在执行任务的线程，Worker进程中会启动所有的Executor。每个Executor都调用execute方法，在Executor方法中的execute线程中。Executor线程 执行execute()方法后，不断的Loop调用executorTransfer的Callable接口。一旦sendQueue buffer达到一定的阈值后，就开始调用ExecutorTransfer的Call方法。
 
  
       ExecutorTransfer类是Executor类的一个成员变量，用来将发送Executor中的Task数据的（如上节中所述）。当Executor中的sendQueue buffer达到一定的阈值后，就开始调用ExecutorTransfer的Call方法。Executor的batchTransferQueue批量的消费消息，也就是Executor的sendQueue被批量消费消息，不断的批量消费batchTransferQueue中的AddressedTuple对象，最后调用WorkerState的transfer方法，对AddressedTuple进行序列化操作，并且将数据发送到Worker的传输队列中去。
 
  
public abstract class Executor implements Callable, EventHandler {

    private static final Logger LOG = LoggerFactory.getLogger(Executor.class);
........
    /**
     * separated from mkExecutor in order to replace executor transfer in executor data for testing
     */
    public ExecutorShutdown execute() throws Exception {
        LOG.info("Loading executor tasks " + componentId + ":" + executorId);

        registerBackpressure();
        //5.在Executor线程 执行execute()方法后，不断的Loop调用executorTransfer的Callable接口。一旦sendQueue buffer达到一定的阈值后。
        // 调用ExecutorTransfer的Call方法
        Utils.SmartThread systemThreads =
                Utils.asyncLoop(executorTransfer, executorTransfer.getName(), reportErrorDie);

        String handlerName = componentId + "-executor" + executorId;
        Utils.SmartThread handlers =
                Utils.asyncLoop(this, false, reportErrorDie, Thread.NORM_PRIORITY, true, true, handlerName);
        setupTicks(StatsUtil.SPOUT.equals(type));

        LOG.info("Finished loading executor " + componentId + ":" + executorId);
        return new ExecutorShutdown(this, Lists.newArrayList(systemThreads, handlers), idToTask, receiveQueue, sendQueue);
    .............}} 
  
ExecutorTransfer源码
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.executor;

public class ExecutorTransfer implements EventHandler, Callable {
    private static final Logger LOG = LoggerFactory.getLogger(ExecutorTransfer.class);

    private final WorkerState workerData;
    private final DisruptorQueue batchTransferQueue;
    private final Map topoConf;
    private final KryoTupleSerializer serializer;
    private final MutableObject cachedEmit;
    private final boolean isDebug;

    public ExecutorTransfer(WorkerState workerData, DisruptorQueue batchTransferQueue, Map topoConf) {
        this.workerData = workerData;
        this.batchTransferQueue = batchTransferQueue;
        this.topoConf = topoConf;
        this.serializer = new KryoTupleSerializer(topoConf, workerData.getWorkerTopologyContext());
        this.cachedEmit = new MutableObject(new ArrayList<>());
        this.isDebug = ObjectReader.getBoolean(topoConf.get(Config.TOPOLOGY_DEBUG), false);
    }

    //4.ExecutorTransfer将tuple添加目标task信息，将tuple封装成AddressedTuple。并将封装后的结果AddressedTuple publish到batchTransferQueue队列中。
    // batchTransferQueue也就是Executor的发送队列。
    public void transfer(int task, Tuple tuple) {
        AddressedTuple val = new AddressedTuple(task, tuple);
        if (isDebug) {
            LOG.info("TRANSFERRING tuple {}", val);
        }
        batchTransferQueue.publish(val);
    }


    @VisibleForTesting
    public DisruptorQueue getBatchTransferQueue() {
        return this.batchTransferQueue;
    }

    /**
     * 6.ExecutorTransfer的Call方法被调用。batchTransferQueue批量的消费消息
     * @return
     * @throws Exception
     */
    @Override
    public Object call() throws Exception {
        batchTransferQueue.consumeBatchWhenAvailable(this);
        return 0L;
    }

    public String getName() {
        return batchTransferQueue.getName();
    }

    /**
     * 7.相应事件，不断的批量消费batchTransferQueue中的AddressedTuple对象
     * @param event
     * @param sequence
     * @param endOfBatch
     * @throws Exception
     */
    @Override
    public void onEvent(Object event, long sequence, boolean endOfBatch) throws Exception {
        ArrayList cachedEvents = (ArrayList) cachedEmit.getObject();
        cachedEvents.add(event);
        if (endOfBatch) {
            //8.调用WorkerState的transfer方法。对AddressedTuple进行序列化操作
            workerData.transfer(serializer, cachedEvents);
            cachedEmit.setObject(new ArrayList<>());
        }
    }
}
 
  
2.4 WorkerState调用transfer方法
 
  
        WorkerState是Worker类的一个成员变量，保存着Worker中大量的存在的状态以及对Worker进程通信以及一些操作。这里介绍WorkerState的transfer方法。在WorkerState的Transfer方法中，首先定义了两个局部变量。一个是List local 还有一个是 Map> remoteMap。分别用来保存Executor中AddressTuple发送到本地Worker进程和远程Worke进程中相应TaskID的Map集合。
 
  
        然后Executor方法中不断的对AddressedTuple进行序列化操作，并将要发送到相同的task的AddressedTuple进行打包批量的发送消息。如果需要发送到本地worker的taskid，我们调用WorkerState的transferLocal方法发送到本地，本地发送不需要序列化。需要发送到远程Worker的消息，序列化后进行打包成Map>对象Pulish到Worker的传输队列中去。
 
  
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.storm.daemon.worker;



public class WorkerState {

........
  
    //12.注册回调函数，WorkerState中的registerCallbacks()方法中注册反序列化连接回调函数。
    public void registerCallbacks() {
        LOG.info("Registering IConnectionCallbacks for {}:{}", assignmentId, port);
        receiver.registerRecv(new DeserializingConnectionCallback(topologyConf,
            getWorkerTopologyContext(),
            this::transferLocal));
    }

    //14.调用用第一个步骤声明的transferLocal()方法 在Worker内部本地发送到相应的线程
    public void transferLocal(List tupleBatch) {
        Map> grouped = new HashMap<>();
        for (AddressedTuple tuple : tupleBatch) {
            Integer executor = taskToShortExecutor.get(tuple.dest);
            if (null == executor) {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
                continue;
            }
            List current = grouped.get(executor);
            if (null == current) {
                current = new ArrayList<>();
                grouped.put(executor, current);
            }
            current.add(tuple);
        }

        for (Map.Entry> entry : grouped.entrySet()) {
            DisruptorQueue queue = shortExecutorReceiveQueueMap.get(entry.getKey());
            if (null != queue) {
                queue.publish(entry.getValue());
            } else {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
            }
        }
    }

    //9.不断的对AddressedTuple进行序列化操作，并将要发送到相同的task的AddressedTuple进行打包批量的发送消息。
    // 如果需要发送到本地worker的taskid，我们调用WorkerState的transferLocal方法发送到本地。本地发送不需要序列化
    // 需要发送到远程Worker的消息，序列化后进行打包成Map>对象发送到Worker的传输队列中去
    public void transfer(KryoTupleSerializer serializer, List tupleBatch) {
        if (trySerializeLocal) {
            assertCanSerialize(serializer, tupleBatch);
        }
        List local = new ArrayList<>();
        Map> remoteMap = new HashMap<>();
        LOG.info("the time of start serializing : {}", System.currentTimeMillis());
        for (AddressedTuple addressedTuple : tupleBatch) {
            int destTask = addressedTuple.getDest();
            if (taskIds.contains(destTask)) {
                // Local task
                local.add(addressedTuple);
            } else {
                // Using java objects directly to avoid performance issues in java code
                if (! remoteMap.containsKey(destTask)) {
                    remoteMap.put(destTask, new ArrayList<>());
                }
                remoteMap.get(destTask).add(new TaskMessage(destTask, serializer.serialize(addressedTuple.getTuple())));
            }
        }
        LOG.info("the time of end serializing : {}", System.currentTimeMillis());

        if (!local.isEmpty()) {
            transferLocal(local);
        }
        if (!remoteMap.isEmpty()) {
            transferQueue.publish(remoteMap);
        }
    }

    .........
}

 
  
2.5 Worker进程发送TaskMessage到相应的Task任务
 
  
        Worker进程是分配资源的最小单位。这里我们主要分析Worker进程之间的通信过程。Worker的传输线程不断的异步从Worker的传输队列中循环调用，不断的批量消费传输队列中的消息，发送到相应的远程Worker中。这里调用WorkerState的sendTuplesToRemoteWorker方法将(HashMap>) packets数据packets发送到相应的远程Worker中。
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 * 

 * http://www.apache.org/licenses/LICENSE-2.0
 * 

 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.storm.daemon.worker;

public class Worker implements Shutdownable, DaemonCommon {
    .........
    public void start() throws Exception {
        LOG.info("Launching worker for {} on {}:{} with id {} and conf {}", topologyId, assignmentId, port, workerId,
            conf);
        // because in local mode, its not a separate
        // process. supervisor will register it in this case
        // if ConfigUtils.isLocalMode(conf) returns false then it is in distributed mode.
        if (!ConfigUtils.isLocalMode(conf)) {
            // Distributed mode
            SysOutOverSLF4J.sendSystemOutAndErrToSLF4J();
            String pid = Utils.processPid();
            FileUtils.touch(new File(ConfigUtils.workerPidPath(conf, workerId, pid)));
            FileUtils.writeStringToFile(new File(ConfigUtils.workerArtifactsPidPath(conf, topologyId, port)), pid,
                Charset.forName("UTF-8"));
        }
        final Map topologyConf =
            ConfigUtils.overrideLoginConfigWithSystemProperty(ConfigUtils.readSupervisorStormConf(conf, topologyId));
        List acls = Utils.getWorkerACL(topologyConf);
        IStateStorage stateStorage =
            ClusterUtils.mkStateStorage(conf, topologyConf, acls, new ClusterStateContext(DaemonType.WORKER));
        IStormClusterState stormClusterState =
            ClusterUtils.mkStormClusterState(stateStorage, acls, new ClusterStateContext());
        Credentials initialCredentials = stormClusterState.credentials(topologyId, null);
        Map initCreds = new HashMap<>();
        if (initialCredentials != null) {
            initCreds.putAll(initialCredentials.get_creds());
        }
        autoCreds = AuthUtils.GetAutoCredentials(topologyConf);
        subject = AuthUtils.populateSubject(null, autoCreds, initCreds);
        backpressureZnodeTimeoutMs = ObjectReader.getInt(topologyConf.get(Config.BACKPRESSURE_ZNODE_TIMEOUT_SECS)) * 1000;

        Subject.doAs(subject, new PrivilegedExceptionAction() {
            @Override public Object run() throws Exception {
                workerState =
                    new WorkerState(conf, context, topologyId, assignmentId, port, workerId, topologyConf, stateStorage,
                        stormClusterState);

                // Heartbeat here so that worker process dies if this fails
                // it's important that worker heartbeat to supervisor ASAP so that supervisor knows
                // that worker is running and moves on
                doHeartBeat();

                executorsAtom = new AtomicReference<>(null);

                // launch heartbeat threads immediately so that slow-loading tasks don't cause the worker to timeout
                // to the supervisor
                workerState.heartbeatTimer
                    .scheduleRecurring(0, (Integer) conf.get(Config.WORKER_HEARTBEAT_FREQUENCY_SECS), () -> {
                        try {
                            doHeartBeat();
                        } catch (IOException e) {
                            throw new RuntimeException(e);
                        }
                    });

                workerState.executorHeartbeatTimer
                    .scheduleRecurring(0, (Integer) conf.get(Config.WORKER_HEARTBEAT_FREQUENCY_SECS),
                        Worker.this::doExecutorHeartbeats);

                //11.worker注册相应的回调函数用来接受远程worker发送来的消息。
                workerState.registerCallbacks();

                workerState.refreshConnections(null);

                workerState.activateWorkerWhenAllConnectionsReady();

                workerState.refreshStormActive(null);

                workerState.runWorkerStartHooks();

                List newExecutors = new ArrayList();
                for (List e : workerState.getExecutors()) {
                    if (ConfigUtils.isLocalMode(topologyConf)) {
                        newExecutors.add(
                            LocalExecutor.mkExecutor(workerState, e, initCreds)
                                .execute());
                    } else {
                        newExecutors.add(
                            Executor.mkExecutor(workerState, e, initCreds)
                                .execute());
                    }
                }
                executorsAtom.set(newExecutors);

                //10.Worker的传输线程不断的异步从Worker的传输队列中循环调用，不断的批量消费传输队列中的消息。发送到相应的远程Worker中
                EventHandler tupleHandler = (packets, seqId, batchEnd) -> workerState
                    .sendTuplesToRemoteWorker((HashMap>) packets, seqId, batchEnd);

                // This thread will publish the messages destined for remote tasks to remote connections
                transferThread = Utils.asyncLoop(() -> {
                    workerState.transferQueue.consumeBatchWhenAvailable(tupleHandler);
                    return 0L;
                });

                DisruptorBackpressureCallback disruptorBackpressureHandler =
                    mkDisruptorBackpressureHandler(workerState);
                workerState.transferQueue.registerBackpressureCallback(disruptorBackpressureHandler);
                workerState.transferQueue
                    .setEnableBackpressure((Boolean) topologyConf.get(Config.TOPOLOGY_BACKPRESSURE_ENABLE));
                workerState.transferQueue
                    .setHighWaterMark(ObjectReader.getDouble(topologyConf.get(Config.BACKPRESSURE_DISRUPTOR_HIGH_WATERMARK)));
                workerState.transferQueue
                    .setLowWaterMark(ObjectReader.getDouble(topologyConf.get(Config.BACKPRESSURE_DISRUPTOR_LOW_WATERMARK)));

                WorkerBackpressureCallback backpressureCallback = mkBackpressureHandler(topologyConf);
                backpressureThread = new WorkerBackpressureThread(workerState.backpressureTrigger, workerState, backpressureCallback);
                if ((Boolean) topologyConf.get(Config.TOPOLOGY_BACKPRESSURE_ENABLE)) {
                    backpressureThread.start();
                    stormClusterState.topologyBackpressure(topologyId, backpressureZnodeTimeoutMs, workerState::refreshThrottle);
                    
                    int pollingSecs = ObjectReader.getInt(topologyConf.get(Config.TASK_BACKPRESSURE_POLL_SECS));
                    workerState.refreshBackpressureTimer.scheduleRecurring(0, pollingSecs, workerState::refreshThrottle);
                }

                credentialsAtom = new AtomicReference(initialCredentials);

                establishLogSettingCallback();

                workerState.stormClusterState.credentials(topologyId, Worker.this::checkCredentialsChanged);

                workerState.refreshCredentialsTimer.scheduleRecurring(0,
                    (Integer) conf.get(Config.TASK_CREDENTIALS_POLL_SECS), new Runnable() {
                        @Override public void run() {
                            checkCredentialsChanged();
                            if ((Boolean) topologyConf.get(Config.TOPOLOGY_BACKPRESSURE_ENABLE)) {
                               checkThrottleChanged();
                            }
                        }
                    });

                workerState.checkForUpdatedBlobsTimer.scheduleRecurring(0,
                        (Integer) conf.getOrDefault(Config.WORKER_BLOB_UPDATE_POLL_INTERVAL_SECS, 10), new Runnable() {
                            @Override public void run() {
                                try {
                                    LOG.debug("Checking if blobs have updated");
                                    updateBlobUpdates();
                                } catch (IOException e) {
                                    // IOException from reading the version files to be ignored
                                    LOG.error(e.getStackTrace().toString());
                                }
                            }
                        });

                // The jitter allows the clients to get the data at different times, and avoids thundering herd
                if (!(Boolean) topologyConf.get(Config.TOPOLOGY_DISABLE_LOADAWARE_MESSAGING)) {
                    workerState.refreshLoadTimer.scheduleRecurringWithJitter(0, 1, 500, Worker.this::doRefreshLoad);
                }

                workerState.refreshConnectionsTimer.scheduleRecurring(0,
                    (Integer) conf.get(Config.TASK_REFRESH_POLL_SECS), workerState::refreshConnections);

                workerState.resetLogLevelsTimer.scheduleRecurring(0,
                    (Integer) conf.get(Config.WORKER_LOG_LEVEL_RESET_POLL_SECS), logConfigManager::resetLogLevels);

                workerState.refreshActiveTimer.scheduleRecurring(0, (Integer) conf.get(Config.TASK_REFRESH_POLL_SECS),
                    workerState::refreshStormActive);

                LOG.info("Worker has topology config {}", Utils.redactValue(topologyConf, Config.STORM_ZOOKEEPER_TOPOLOGY_AUTH_PAYLOAD));
                LOG.info("Worker {} for storm {} on {}:{}  has finished loading", workerId, topologyId, assignmentId, port);
                return this;
            };
        });
        .......................
    } 
  
        WorkerState调用sendTuplesToRemoteWorker将packets发送到相应的远程Worker中。不断的将数据添加到TransferDrainer贮水池中。 当数据批次到达批次末尾的时候，然后调用TransferDrainer的send方法将数据将数据发送到远程的Worker中去。
 
  
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.storm.daemon.worker;



public class WorkerState {

........
   /**
     * WorkerState调用sendTuplesToRemoteWorker将packets发送到相应的远程Worker中。不断的将数据添加到TransferDrainer贮水池中。
     * 当数据批次到达批次末尾的时候，然后调用TransferDrainer的send方法将数据将数据发送到远程的Worker中去。
     * @param packets
     * @param seqId
     * @param batchEnd
     */
     public void sendTuplesToRemoteWorker(HashMap> packets, long seqId, boolean batchEnd) {
        drainer.add(packets);
        if (batchEnd) {
            ReentrantReadWriteLock.ReadLock readLock = endpointSocketLock.readLock();
            try {
                readLock.lock();
                drainer.send(cachedTaskToNodePort.get(), cachedNodeToPortSocket.get());
            } finally {
                readLock.unlock();
            }
            drainer.clear();
        }
    }

    .........
}
 
  
        TransferDrainer调用Send方法将数据发送到相应的远程Worker进程。
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.utils;

public class TransferDrainer {

  private Map>> bundles = new HashMap();
  private static final Logger LOG = LoggerFactory.getLogger(TransferDrainer.class);
  
  public void add(HashMap> taskTupleSetMap) {
    for (Map.Entry> entry : taskTupleSetMap.entrySet()) {
      addListRefToMap(this.bundles, entry.getKey(), entry.getValue());
    }
  }
  
  public void send(Map taskToNode, Map connections) {
    HashMap>> bundleMapByDestination = groupBundleByDestination(taskToNode);

    for (Map.Entry>> entry : bundleMapByDestination.entrySet()) {
      NodeInfo hostPort = entry.getKey();
      IConnection connection = connections.get(hostPort);
      if (null != connection) {
        ArrayList> bundle = entry.getValue();
        Iterator iter = getBundleIterator(bundle);
        if (null != iter && iter.hasNext()) {
          connection.send(iter);
        }
      } else {
        LOG.warn("Connection is not available for hostPort {}", hostPort);
      }
    }
  }

  private HashMap>> groupBundleByDestination(Map taskToNode) {
    HashMap>> bundleMap = Maps.newHashMap();
    for (Integer task : this.bundles.keySet()) {
      NodeInfo hostPort = taskToNode.get(task);
      if (hostPort != null) {
        for (ArrayList chunk : this.bundles.get(task)) {
          addListRefToMap(bundleMap, hostPort, chunk);
        }
      } else {
        LOG.warn("No remote destination available for task {}", task);
      }
    }
    return bundleMap;
  }

  private  void addListRefToMap(Map>> bundleMap,
                                   T key, ArrayList tuples) {
    ArrayList> bundle = bundleMap.get(key);

    if (null == bundle) {
      bundle = new ArrayList>();
      bundleMap.put(key, bundle);
    }

    if (null != tuples && tuples.size() > 0) {
      bundle.add(tuples);
    }
  }

  private Iterator getBundleIterator(final ArrayList> bundle) {
    
    if (null == bundle) {
      return null;
    }
    
    return new Iterator () {
      
      private int offset = 0;
      private int size = 0;
      {
        for (ArrayList list : bundle) {
            size += list.size();
        }
      }
      
      private int bundleOffset = 0;
      private Iterator iter = bundle.get(bundleOffset).iterator();
      
      @Override
      public boolean hasNext() {
          return offset < size;
      }

      @Override
      public TaskMessage next() {
        TaskMessage msg;
        if (iter.hasNext()) {
          msg = iter.next(); 
        } else {
          bundleOffset++;
          iter = bundle.get(bundleOffset).iterator();
          msg = iter.next();
        }
        if (null != msg) {
          offset++;
        }
        return msg;
      }

      @Override
      public void remove() {
        throw new RuntimeException("not supported");
      }
    };
  }
  
  public void clear() {
    bundles.clear();
  }
}

 
  
2.6 Worker进程接收从其他Worker进程发送过来的数据
 
  
 
  
        首先Worker进程在初始化的时候。Worker进程会注册相应的回调函数用来接收远程worker注册发送过来的消息。
 
  
                //11.worker注册相应的回调函数用来接受远程worker发送来的消息。
                workerState.registerCallbacks();
 
  
        在workerState的registerCallbacks方法中，它会注册接收数据的回调用函数，并且注册反序列化接口DeserializingConnectionCallback。
 
  
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.storm.daemon.worker;



public class WorkerState {

........
  
    //12.注册回调函数，WorkerState中的registerCallbacks()方法中注册反序列化连接回调函数。
    public void registerCallbacks() {
        LOG.info("Registering IConnectionCallbacks for {}:{}", assignmentId, port);
        receiver.registerRecv(new DeserializingConnectionCallback(topologyConf,
            getWorkerTopologyContext(),
            this::transferLocal));
    }

    //14.调用用第一个步骤声明的transferLocal()方法 在Worker内部本地发送到相应的线程
    public void transferLocal(List tupleBatch) {
        Map> grouped = new HashMap<>();
        for (AddressedTuple tuple : tupleBatch) {
            Integer executor = taskToShortExecutor.get(tuple.dest);
            if (null == executor) {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
                continue;
            }
            List current = grouped.get(executor);
            if (null == current) {
                current = new ArrayList<>();
                grouped.put(executor, current);
            }
            current.add(tuple);
        }

        for (Map.Entry> entry : grouped.entrySet()) {
            DisruptorQueue queue = shortExecutorReceiveQueueMap.get(entry.getKey());
            if (null != queue) {
                queue.publish(entry.getValue());
            } else {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
            }
        }
    }

    //9.不断的对AddressedTuple进行序列化操作，并将要发送到相同的task的AddressedTuple进行打包批量的发送消息。
    // 如果需要发送到本地worker的taskid，我们调用WorkerState的transferLocal方法发送到本地。本地发送不需要序列化
    // 需要发送到远程Worker的消息，序列化后进行打包成Map>对象发送到Worker的传输队列中去
    public void transfer(KryoTupleSerializer serializer, List tupleBatch) {
        if (trySerializeLocal) {
            assertCanSerialize(serializer, tupleBatch);
        }
        List local = new ArrayList<>();
        Map> remoteMap = new HashMap<>();
        LOG.info("the time of start serializing : {}", System.currentTimeMillis());
        for (AddressedTuple addressedTuple : tupleBatch) {
            int destTask = addressedTuple.getDest();
            if (taskIds.contains(destTask)) {
                // Local task
                local.add(addressedTuple);
            } else {
                // Using java objects directly to avoid performance issues in java code
                if (! remoteMap.containsKey(destTask)) {
                    remoteMap.put(destTask, new ArrayList<>());
                }
                remoteMap.get(destTask).add(new TaskMessage(destTask, serializer.serialize(addressedTuple.getTuple())));
            }
        }
        LOG.info("the time of end serializing : {}", System.currentTimeMillis());

        if (!local.isEmpty()) {
            transferLocal(local);
        }
        if (!remoteMap.isEmpty()) {
            transferQueue.publish(remoteMap);
        }
    }

    .........
}
 
  
        在Worker进程中的反序列化接口类中DeserializingConnectionCallback，对消息进行反序列化，将List batch反序列化成ArrayList ret。Worker接收线程从接收队列中读取TaskMessage序列化后的数据，然后将其进行反序列化操作。最终得到带有消息头的AddressTuple。然后调用WorkerState的TransferLocal方法，在Worker进程中本地发送AddressedTuple。
 
  
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.storm.messaging;

/**
 * A class that is called when a TaskMessage arrives.
 */
public class DeserializingConnectionCallback implements IConnectionCallback, IMetric {

    private static final Logger LOG = LoggerFactory.getLogger(DeserializingConnectionCallback.class);

    private final WorkerState.ILocalTransferCallback cb;
    private final Map conf;
    private final GeneralTopologyContext context;

    private final ThreadLocal _des =
        new ThreadLocal() {
            @Override
            protected KryoTupleDeserializer initialValue() {
                return new KryoTupleDeserializer(conf, context);
            }
        };

    // Track serialized size of messages.
    private final boolean sizeMetricsEnabled;
    private final ConcurrentHashMap byteCounts = new ConcurrentHashMap<>();


    public DeserializingConnectionCallback(final Map conf, final GeneralTopologyContext context, WorkerState.ILocalTransferCallback callback) {
        this.conf = conf;
        this.context = context;
        cb = callback;
        sizeMetricsEnabled = ObjectReader.getBoolean(conf.get(Config.TOPOLOGY_SERIALIZED_MESSAGE_SIZE_METRICS), false);

    }

    //13.当有消息发送到Worker中时。Worker接收线程从接收队列中读取TaskMessage序列化后的数据，然后将其进行反序列化操作。最终得到带有消息头的AddressTuple。
    //然后调用回调函数的transfer方法。
    @Override
    public void recv(List batch) {
        KryoTupleDeserializer des = _des.get();
        ArrayList ret = new ArrayList<>(batch.size());
        LOG.info("the time of start deserializing : {}", System.currentTimeMillis());
        for (TaskMessage message: batch) {
            Tuple tuple = des.deserialize(message.message());
            AddressedTuple addrTuple = new AddressedTuple(message.task(), tuple);
            updateMetrics(tuple.getSourceTask(), message);
            ret.add(addrTuple);
        }
        LOG.info("the time of start deserializing : {}", System.currentTimeMillis());
        cb.transfer(ret);
    }

    /**
     * Returns serialized byte count traffic metrics.
     * @return Map of metric counts, or null if disabled
     */
    @Override
    public Object getValueAndReset() {
        if (!sizeMetricsEnabled) {
            return null;
        }
        HashMap outMap = new HashMap<>();
        for (Map.Entry ent : byteCounts.entrySet()) {
            AtomicLong count = ent.getValue();
            if (count.get() > 0) {
                outMap.put(ent.getKey(), count.getAndSet(0L));
            }
        }
        return outMap;
    }

    /**
     * Update serialized byte counts for each message.
     * @param sourceTaskId source task
     * @param message serialized message
     */
    protected void updateMetrics(int sourceTaskId, TaskMessage message) {
        if (sizeMetricsEnabled) {
            int dest = message.task();
            int len = message.message().length;
            String key = Integer.toString(sourceTaskId) + "-" + Integer.toString(dest);
            byteCounts.computeIfAbsent(key, k -> new AtomicLong(0L)).addAndGet(len);
        }
    }

}

 
  
        在WorkerState的TransferLocal（）方法中，将AddressedTuple发送到相应的Executor的接收队列中去。
 
  
 
  
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.storm.daemon.worker;



public class WorkerState {

........
  
    //14.调用用第一个步骤声明的transferLocal()方法 在Worker内部本地发送到相应的线程
    public void transferLocal(List tupleBatch) {
        Map> grouped = new HashMap<>();
        for (AddressedTuple tuple : tupleBatch) {
            Integer executor = taskToShortExecutor.get(tuple.dest);
            if (null == executor) {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
                continue;
            }
            List current = grouped.get(executor);
            if (null == current) {
                current = new ArrayList<>();
                grouped.put(executor, current);
            }
            current.add(tuple);
        }

        for (Map.Entry> entry : grouped.entrySet()) {
            DisruptorQueue queue = shortExecutorReceiveQueueMap.get(entry.getKey());
            if (null != queue) {
                queue.publish(entry.getValue());
            } else {
                LOG.warn("Received invalid messages for unknown tasks. Dropping... ");
            }
        }
    }
    .........
}
 
  
2.7 Executor 从本地receiveQueue中消费数据
 
  
        Executor从本地Executor 接收队列中不断的消费数据。将接收到的ArrayList addressedTuples循环遍历。然后调用tupleActionFn发送到相应的BoltExecutor或者SpoutExecutor，执行相应的tupleActionFn 函数进行逻辑处理
 
  
 public abstract class Executor implements Callable, EventHandler {
........
//15.Executor调用OnEvent()方法接收从Worker发送过来的AddressTuple.然后调用tupleActionFn发送到相应的BoltExecutor或者SpoutExecutor
    @SuppressWarnings("unchecked")
    @Override
    public void onEvent(Object event, long seq, boolean endOfBatch) throws Exception {
        ArrayList addressedTuples = (ArrayList) event;
        for (AddressedTuple addressedTuple : addressedTuples) {
            TupleImpl tuple = (TupleImpl) addressedTuple.getTuple();
            int taskId = addressedTuple.getDest();
            if (isDebug) {
                LOG.info("Processing received message FOR {} TUPLE: {}", taskId, tuple);
            }
            LOG.info("the time of receiving message: {}", System.currentTimeMillis());
            if (taskId != AddressedTuple.BROADCAST_DEST) {
                tupleActionFn(taskId, tuple);
            } else {
                for (Integer t : taskIds) {
                    tupleActionFn(t, tuple);
                }
            }
        }
    }
...........
}
 
  
 
  

                            
                        
                    
                    
                    

                    
                    
                    

                
                

                    
                

            
        
    
    

        
你可能感兴趣的:(实时流处理,Storm框架)

        

            

                
    
                    
  • android系统selinux中添加新属性property
                        辉色投像

                        
    1.定位/android/system/sepolicy/private/property_contexts声明属性开头：persist.charge声明属性类型：u:object_r:system_prop:s0图12.定位到android/system/sepolicy/public/domain.te删除neverallow{domain-init}default_prop:property
    
                    
    • 
                    
  • OC语言多界面传值五大方式
                        Magnetic_h
iosui学习objective-c开发语言
                        
    前言在完成暑假仿写项目时，遇到了许多需要用到多界面传值的地方，这篇博客来总结一下比较常用的五种多界面传值的方式。属性传值属性传值一般用前一个界面向后一个界面传值，简单地说就是通过访问后一个视图控制器的属性来为它赋值，通过这个属性来做到从前一个界面向后一个界面传值。首先在后一个界面中定义属性@interfaceBViewController:UIViewController@propertyNSSt
    
                    
    • 
                    
  • Cell Insight | 单细胞测序技术又一新发现，可用于HIV-1和Mtb共感染个体诊断
                        尐尐呅

                        
    结核病是艾滋病合并其他疾病中导致患者死亡的主要原因。其中结核病由结核分枝杆菌（Mycobacteriumtuberculosis,Mtb）感染引起，获得性免疫缺陷综合症（艾滋病）由人免疫缺陷病毒（Humanimmunodeficiencyvirustype1,HIV-1）感染引起。国家感染性疾病临床医学研究中心/深圳市第三人民医院张国良团队携手深圳华大生命科学研究院吴靓团队，共同研究得出单细胞测序
    
                    
    • 
                    
  • c++ 的iostream 和 c++的stdio的区别和联系
                        黄卷青灯77
c++算法开发语言iostreamstdio
                        
    在C++中，iostream和C语言的stdio.h都是用于处理输入输出的库，但它们在设计、用法和功能上有许多不同。以下是两者的区别和联系：区别1.编程风格iostream（C++风格）：C++标准库中的输入输出流类库，支持面向对象的输入输出操作。典型用法是cin（输入）和cout（输出），使用>操作符来处理数据。更加类型安全，支持用户自定义类型的输入输出。#includeintmain(){in
    
                    
    • 
                    
  • LocalDateTime 转 String
                        igotyback
java开发语言
                        
    importjava.time.LocalDateTime;importjava.time.format.DateTimeFormatter;publicclassMain{publicstaticvoidmain(String[]args){//获取当前时间LocalDateTimenow=LocalDateTime.now();//定义日期格式化器DateTimeFormatterformat
    
                    
    • 
                    
  • mysql禁用远程登录
                        igotyback
mysql
                        
    去mysql库中的user表里，将host都改成localhost之后刷新权限FLUSHPRIVILEGES;
    
                    
    • 
                    
  • ArcGIS栅格计算器常见公式（赋值、0和空值的转换、补充栅格空值）
                        研学随笔
arcgis经验分享
                        
    我们在使用ArcGIS时通常经常用到栅格计算器，今天主要给大家介绍我日常中经常用到的几个公式，供大家参考学习。将特定值（-9999）赋值为0，例如-9999.Con("raster"==-9999,0,"raster")2.给空值赋予特定的值（如0）Con(IsNull("raster"),0,"raster")3.将特定的栅格值(如1)赋值为空值，其他保留原值SetNull("raster"==
    
                    
    • 
                    
  • 水平垂直居中的几种方法（总结）
                        LJ小番茄
CSS_玄学语言htmljavascript前端csscss3
                        
    1.使用flexbox的justify-content和align-items.parent{display:flex;justify-content:center;/*水平居中*/align-items:center;/*垂直居中*/height:100vh;/*需要指定高度*/}2.使用grid的place-items:center.parent{display:grid;place-item
    
                    
    • 
                    
  • 【一起学Rust | 设计模式】习惯语法——使用借用类型作为参数、格式化拼接字符串、构造函数
                        广龙宇
一起学Rust#Rust设计模式rust设计模式开发语言
                        
    提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档文章目录前言一、使用借用类型作为参数二、格式化拼接字符串三、使用构造函数总结前言Rust不是传统的面向对象编程语言，它的所有特性，使其独一无二。因此，学习特定于Rust的设计模式是必要的。本系列文章为作者学习《Rust设计模式》的学习笔记以及自己的见解。因此，本系列文章的结构也与此书的结构相同（后续可能会调成结构），基本上分为三个部分
    
                    
    • 
                    
  • 每日一题——第八十九题
                        互联网打工人no1
C语言程序设计每日一练c语言
                        
    题目：在字符串中找到提取数字，并统计一共找到多少整数，a123xxyu23&8889，那么找到的整数为123，23，8889//思想：#include#include#includeintmain(){charstr[]="a123xxyu23&8889";intcount=0;intnum=0;//用于临时存放当前正在构建的整数。boolinNum=false;//用于标记当前是否正在读取一个整
    
                    
    • 
                    
  • 每日一题——第九十题
                        互联网打工人no1
C语言程序设计每日一练c语言
                        
    题目：判断子串是否与主串匹配#include#include#include//////判断子串是否在主串中匹配//////主串///子串///boolisSubstring(constchar*str,constchar*substr){intlenstr=strlen(str);//计算主串的长度intlenSub=strlen(substr);//计算子串的长度//遍历主字符串，对每个可能得
    
                    
    • 
                    
  • 每日一题——第八十四题
                        互联网打工人no1
C语言程序设计每日一练c语言
                        
    题目：编写函数1、输入10个职工的姓名和职工号2、按照职工由大到小顺序排列，姓名顺序也随之调整3、要求输入一个职工号，用折半查找法找出该职工的姓名#define_CRT_SECURE_NO_WARNINGS#include#include#defineMAX_EMPLOYEES10typedefstruct{intid;charname[50];}Empolyee;voidinputEmploye
    
                    
    • 
                    
  • 2022-04-18
                        Apbenz

                        
    语重心长的和我说，不要老是说不行，人至而立之年危机四伏，内在的，外在的，感觉就是心力憔悴，让人无所适从。面对职场的无情，突然好羡慕干体力劳动的外卖小哥。难道命运是想让我去送外卖了吗？干体力活才能让我活下去？fastadmin打卡成功,淘宝金币任务完成。ㅏㅓㅗㅜㅡㅣㅐㅔㅑㅕㅛㅠㅢㅒㅖY行。야자여자요리우유의사얘기예
    
                    
    • 
                    
  • C#中使用split分割字符串
                        互联网打工人no1
c#
                        
    1、用字符串分隔：usingSystem.Text.RegularExpressions;stringstr="aaajsbbbjsccc";string[]sArray=Regex.Split(str,"js",RegexOptions.IgnoreCase);foreach(stringiinsArray)Response.Write(i.ToString()+"");输出结果：aaabbbc
    
                    
    • 
                    
  • WPF中的ComboBox控件几种数据绑定的方式
                        互联网打工人no1
wpfc#
                        
    一、用字典给ItemsSource赋值（此绑定用的地方很多，建议熟练掌握）在XMAL中：在CS文件中privatevoidBindData(){DictionarydicItem=newDictionary();dicItem.add(1,"北京");dicItem.add(2,"上海");dicItem.add(3,"广州");cmb_list.ItemsSource=dicItem;cmb_l
    
                    
    • 
                    
  • 第四天旅游线路预览——从换乘中心到喀纳斯湖
                        陟彼高冈yu
基于Googleearthstudio的旅游规划和预览旅游
                        
    第四天：从贾登峪到喀纳斯风景区入口，晚上住宿贾登峪；换乘中心有4路车，喀纳斯①号车，去喀纳斯湖，路程时长约5分钟；将上面的的行程安排进行动态展示，具体步骤见”Googleearthstudio进行动态轨迹显示制作过程“、“Googleearthstudio入门教程”和“Googleearthstudio进阶教程“相关内容，得到行程如下所示：Day4-2-480p
    
                    
    • 
                    
  • Goolge earth studio 进阶4——路径修改与平滑
                        陟彼高冈yu
Googleearthstudio进阶教程旅游
                        
    如果我们希望在大约中途时获得更多的城市鸟瞰视角。可以将相机拖动到这里并创建一个新的关键帧。camera_target_clip_7EarthStudio会自动平滑我们的路径，所以当我们通过这个关键帧时，不是一个生硬的角度，而是一个平滑的曲线。camera_target_clip_8路径上有贝塞尔控制手柄，允许我们调整路径的形状。右键单击，我们可以选择“平滑路径”，这是默认的自动平滑算法，或者我们可
    
                    
    • 
                    
  • Python教程：一文了解使用Python处理XPath
                        旦莫
Python进阶python开发语言
                        
    目录1.环境准备1.1安装lxml1.2验证安装2.XPath基础2.1什么是XPath？2.2XPath语法2.3示例XML文档3.使用lxml解析XML3.1解析XML文档3.2查看解析结果4.XPath查询4.1基本路径查询4.2使用属性查询4.3查询多个节点5.XPath的高级用法5.1使用逻辑运算符5.2使用函数6.实战案例6.1从网页抓取数据6.1.1安装Requests库6.1.2代
    
                    
    • 
                    
  • Google earth studio 简介
                        陟彼高冈yu
旅游
                        
    GoogleEarthStudio是一个基于Web的动画工具，专为创作使用GoogleEarth数据的动画和视频而设计。它利用了GoogleEarth强大的三维地图和卫星影像数据库，使用户能够轻松地创建逼真的地球动画、航拍视频和动态地图可视化。网址为https://www.google.com/earth/studio/。GoogleEarthStudio是一个基于Web的动画工具，专为创作使用G
    
                    
    • 
                    
  • 四章-32-点要素的聚合
                        彩云飘过

                        
    本文基于腾讯课堂老胡的课《跟我学Openlayers--基础实例详解》做的学习笔记，使用的openlayers5.3.xapi。源码见1032.html，对应的官网示例https://openlayers.org/en/latest/examples/cluster.htmlhttps://openlayers.org/en/latest/examples/earthquake-clusters.
    
                    
    • 
                    
  • 使用LLaVa和Ollama实现多模态RAG示例
                        llzwxh888
python人工智能开发语言
                        
    本文将详细介绍如何使用LLaVa和Ollama实现多模态RAG（检索增强生成），通过提取图像中的结构化数据、生成图像字幕等功能来展示这一技术的强大之处。安装环境首先，您需要安装以下依赖包：!pipinstallllama-index-multi-modal-llms-ollama!pipinstallllama-index-readers-file!pipinstallunstructured!p
    
                    
    • 
                    
  • python是什么意思中文-在python中%是什么意思
                        编程大乐趣

                        
    Python中%有两种：1、数值运算：%代表取模，返回除法的余数。如：>>>7%212、%操作符（字符串格式化，stringformatting），说明如下：%[(name)][flags][width].[precision]typecode(name)为命名flags可以有+，-，''或0。+表示右对齐。-表示左对齐。''为一个空格，表示在正数的左侧填充一个空格，从而与负数对齐。0表示使用0填
    
                    
    • 
                    
  • 利用Requests Toolkit轻松完成HTTP请求
                        nseejrukjhad
http网络协议网络python
                        
    RequestsToolkit的力量：轻松构建HTTP请求Agent在现代软件开发中，API请求是与外部服务交互的核心。RequestsToolkit提供了一种便捷的方式，帮助开发者构建自动化的HTTP请求Agent。本文旨在详细介绍RequestsToolkit的设置、使用和潜在挑战。引言RequestsToolkit是一个强大的工具包，可用于构建执行HTTP请求的智能代理。这对于想要自动化与外
    
                    
    • 
                    
  • 利用LangChain的StackExchange组件实现智能问答系统
                        nseejrukjhad
langchainmicrosoft数据库python
                        
    利用LangChain的StackExchange组件实现智能问答系统引言在当今的软件开发世界中，StackOverflow已经成为程序员解决问题的首选平台之一。而LangChain作为一个强大的AI应用开发框架，提供了StackExchange组件，使我们能够轻松地将StackOverflow的海量知识库集成到我们的应用中。本文将详细介绍如何使用LangChain的StackExchange组件
    
                    
    • 
                    
  • Python快速入门 —— 第三节：类与对象
                        孤华暗香
Python快速入门python开发语言
                        
    第三节：类与对象目标：了解面向对象编程的基础概念，并学会如何定义类和创建对象。内容：类与对象：定义类：class关键字。类的构造函数：__init__()。类的属性和方法。对象的创建与使用。示例：classStudent:def__init__(self,name,age,major):self.name&#
    
                    
    • 
                    
  • webpack图片等资源的处理
                        dmengmeng

                        
    需要的loaderfile-loader（让我们可以引入这些资源文件）url-loader（其实是file-loader的二次封装）img-loader（处理图片所需要的）在没有使用任何处理图片的loader之前，比如说css中用到了背景图片，那么最后打包会报错的，因为他没办法处理图片。其实你只想能够使用图片的话。只加一个file-loader就可以，打开网页能准确看到图片。{test:/\.(p
    
                    
    • 
                    
  • Faiss Tips：高效向量搜索与聚类的利器
                        焦习娜Samantha

                        
    FaissTips：高效向量搜索与聚类的利器faiss_tipsSomeusefultipsforfaiss项目地址:https://gitcode.com/gh_mirrors/fa/faiss_tips项目介绍Faiss是由FacebookAIResearch开发的一个用于高效相似性搜索和密集向量聚类的库。它支持多种硬件平台，包括CPU和GPU，能够在海量数据集上实现快速的近似最近邻搜索（AN
    
                    
    • 
                    
  • Python 实现图片裁剪（附代码） | Python工具
                        剑客阿良_ALiang

                        
    前言本文提供将图片按照自定义尺寸进行裁剪的工具方法，一如既往的实用主义。环境依赖ffmpeg环境安装，可以参考我的另一篇文章：windowsffmpeg安装部署_阿良的博客-CSDN博客本文主要使用到的不是ffmpeg，而是ffprobe也在上面这篇文章中的zip包中。ffmpy安装：pipinstallffmpy-ihttps://pypi.douban.com/simple代码不废话了，上代码
    
                    
    • 
                    
  • python os 环境变量
                        CV矿工
python开发语言numpy
                        
    环境变量：环境变量是程序和操作系统之间的通信方式。有些字符不宜明文写进代码里，比如数据库密码，个人账户密码，如果写进自己本机的环境变量里，程序用的时候通过os.environ.get（）取出来就行了。os.environ是一个环境变量的字典。环境变量的相关操作importos"""设置/修改环境变量：os.environ[‘环境变量名称’]=‘环境变量值’#其中key和value均为string类
    
                    
    • 
                    
  • Redis系列：Geo 类型赋能亿级地图位置计算
                        Ly768768
redisbootstrap数据库
                        
    1前言我们在篇深刻理解高性能Redis的本质的时候就介绍过Redis的几种基本数据结构，它是基于不同业务场景而设计的：动态字符串(REDIS_STRING)：整数(REDIS_ENCODING_INT)、字符串(REDIS_ENCODING_RAW)双端列表(REDIS_ENCODING_LINKEDLIST)压缩列表(REDIS_ENCODING_ZIPLIST)跳跃表(REDIS_ENCODI
    
                    
    • 
                                
  • PHP，安卓，UI，java，linux视频教程合集
                                    cocos2d-x小菜
javaUIPHPandroidlinux
                                    
    ╔-----------------------------------╗┆                           
    
                                
    • 
                                
  • 各表中的列名必须唯一。在表 'dbo.XXX' 中多次指定了列名 'XXX'。
                                    bozch
.net.net mvc
                                    
    在.net mvc5中，在执行某一操作的时候，出现了如下错误： 
      各表中的列名必须唯一。在表 'dbo.XXX' 中多次指定了列名 'XXX'。 
经查询当前的操作与错误内容无关，经过对错误信息的排查发现，事故出现在数据库迁移上。 
回想过去： 在迁移之前已经对数据库进行了添加字段操作，再次进行迁移插入XXX字段的时候，就会提示如上错误。  
&
    
                                
    • 
                                
  • Java 对象大小的计算
                                    e200702084
java
                                    
                              Java对象的大小 
 
如何计算一个对象的大小呢？ 
 
 
   
    
                                
    • 
                                
  • Mybatis Spring
                                    171815164
mybatis
                                    
    ApplicationContext ac = new ClassPathXmlApplicationContext("applicationContext.xml");
CustomerService userService = (CustomerService) ac.getBean("customerService");
Customer cust
    
                                
    • 
                                
  • JVM 不稳定参数
                                    g21121
jvm
                                    
            -XX 参数被称为不稳定参数，之所以这么叫是因为此类参数的设置很容易引起JVM 性能上的差异，使JVM 存在极大的不稳定性。当然这是在非合理设置的前提下，如果此类参数设置合理讲大大提高JVM 的性能及稳定性。        可以说“不稳定参数”
    
                                
    • 
                                
  • 用户自动登录网站
                                    永夜-极光
用户
                                    
    1.目标:实现用户登录后,再次登录就自动登录,无需用户名和密码 
2.思路:将用户的信息保存为cookie 
           每次用户访问网站,通过filter拦截所有请求,在filter中读取所有的cookie,如果找到了保存登录信息的cookie,那么在cookie中读取登录信息,然后直接
    
                                
    • 
                                
  • centos7 安装后失去win7的引导记录
                                    程序员是怎么炼成的
操作系统
                                    
    1.使用root身份(必须)打开 /boot/grub2/grub.cfg 2.找到 ### BEGIN /etc/grub.d/30_os-prober ###   在后面添加    menuentry "Windows 7 (loader) (on /dev/sda1)" { 
    
                                
    • 
                                
  • Oracle 10g 官方中文安装帮助文档以及Oracle官方中文教程文档下载
                                    aijuans
oracle
                                    
    Oracle 10g 官方中文安装帮助文档下载：http://download.csdn.net/tag/Oracle%E4%B8%AD%E6%96%87API%EF%BC%8COracle%E4%B8%AD%E6%96%87%E6%96%87%E6%A1%A3%EF%BC%8Coracle%E5%AD%A6%E4%B9%A0%E6%96%87%E6%A1%A3 Oracle 10g 官方中文教程
    
                                
    • 
                                
  • JavaEE开源快速开发平台G4Studio_V3.2发布了
                                    無為子
AOPoraclemysqljavaeeG4Studio
                                    
      
我非常高兴地宣布,今天我们最新的JavaEE开源快速开发平台G4Studio_V3.2版本已经正式发布。大家可以通过如下地址下载。 
  
访问G4Studio网站  
http://www.g4it.org      
G4Studio_V3.2版本变更日志 
功能新增 
(1).新增了系统右下角滑出提示窗口功能。 
(2).新增了文件资源的Zip压缩和解压缩
    
                                
    • 
                                
  • Oracle常用的单行函数应用技巧总结
                                    百合不是茶
日期函数转换函数(核心)数字函数通用函数(核心)字符函数
                                    
    单行函数;   字符函数,数字函数,日期函数,转换函数(核心),通用函数(核心) 
一:字符函数: 
   .UPPER(字符串) 将字符串转为大写
   .LOWER (字符串) 将字符串转为小写 
   .INITCAP(字符串) 将首字母大写
   .LENGTH (字符串) 字符串的长度
   .REPLACE(字符串,'A','_') 将字符串字符A转换成_

    
                                
    • 
                                
  • Mockito异常测试实例
                                    bijian1013
java单元测试mockito
                                    
    Mockito异常测试实例： 
package com.bijian.study;

import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

import org.junit.Assert;
import org.junit.Test;

import org.mockito.
    
                                
    • 
                                
  • GA与量子恒道统计
                                    Bill_chen
JavaScript浏览器百度Google防火墙
                                    
    前一阵子，统计**网址时，Google Analytics（GA） 和量子恒道统计（也称量子统计），数据有较大的偏差，仔细找相关资料研究了下，总结如下： 
  
为何GA和量子网站统计（量子统计前身为雅虎统计）结果不同？ 
首先：没有一种网站统计工具能保证百分之百的准确出现该问题可能有以下几个原因：（1）不同的统计分析系统的算法机制不同；（2）统计代码放置的位置和前后
    
                                
    • 
                                
  • 【Linux命令三】Top命令
                                    bit1129
linux命令
                                    
    Linux的Top命令类似于Windows的任务管理器，可以查看当前系统的运行情况，包括CPU、内存的使用情况等。如下是一个Top命令的执行结果： 
  
  
top - 21:22:04 up 1 day, 23:49, 1 user, load average: 1.10, 1.66, 1.99
Tasks: 202 total,   4 running, 198 sl
    
                                
    • 
                                
  • spring四种依赖注入方式
                                    白糖_
spring
                                    
       平常的java开发中，程序员在某个类中需要依赖其它类的方法，则通常是new一个依赖类再调用类实例的方法，这种开发存在的问题是new的类实例不好统一管理，spring提出了依赖注入的思想，即依赖类不由程序员实例化，而是通过spring容器帮我们new指定实例并且将实例注入到需要该对象的类中。依赖注入的另一种说法是“控制反转”，通俗的理解是：平常我们new一个实例，这个实例的控制权是我
    
                                
    • 
                                
  • angular.injector
                                    boyitech
AngularJSAngularJS API
                                    
    angular.injector 
   描述:   创建一个injector对象, 调用injector对象的方法可以获得angular的service, 或者用来做依赖注入.        使用方法:   angular.injector(modules, [strictDi])        参数详解:      Param Type Details   mod
    
                                
    • 
                                
  • java-同步访问一个数组Integer[10]，生产者不断地往数组放入整数1000，数组满时等待；消费者不断地将数组里面的数置零，数组空时等待
                                    bylijinnan
Integer
                                    
    

public class PC {

	/**
	 * 题目：生产者-消费者。
	 * 同步访问一个数组Integer[10]，生产者不断地往数组放入整数1000，数组满时等待；消费者不断地将数组里面的数置零，数组空时等待。
	 */
	
	private static final Integer[] val=new Integer[10];
	private static
    
                                
    • 
                                
  • 使用Struts2.2.1配置
                                    Chen.H
apachespringWebxmlstruts
                                    
    Struts2.2.1 需要如下 jar包: commons-fileupload-1.2.1.jar commons-io-1.3.2.jar commons-logging-1.0.4.jar freemarker-2.3.16.jar javassist-3.7.ga.jar ognl-3.0.jar spring.jar 
struts2-core-2.2.1.jar struts2-sp
    
                                
    • 
                                
  • [职业与教育]青春之歌
                                    comsci
教育
                                    
     
 
       每个人都有自己的青春之歌............但是我要说的却不是青春... 
 
       大家如果在自己的职业生涯没有给自己以后创业留一点点机会,仅仅凭学历和人脉关系,是难以在竞争激烈的市场中生存下去的.... 
 
  &nbs
    
                                
    • 
                                
  • oracle连接(join)中使用using关键字
                                    daizj
JOINoraclesqlusing
                                    
    在oracle连接(join)中使用using关键字 
34. View the Exhibit and examine the structure of the ORDERS and ORDER_ITEMS tables. 
Evaluate the following SQL statement: 
SELECT oi.order_id, product_id, order_date 
FRO
    
                                
    • 
                                
  • NIO示例
                                    daysinsun
nio
                                    
    NIO服务端代码： 
 

public class NIOServer {
	
	private Selector selector;

	
	public void startServer(int port) throws IOException {

		ServerSocketChannel serverChannel = ServerSocketChannel.open(
    
                                
    • 
                                
  • C语言学习homework1
                                    dcj3sjt126com
chomework
                                    
    0、 课堂练习做完 
1、使用sizeof计算出你所知道的所有的类型占用的空间。 
int x; 
sizeof(x); 
  
sizeof(int); 
  
# include <stdio.h>

int main(void)
{
	int x1;
	char x2;
	double x3;
	float x4;

	printf(&quo
    
                                
    • 
                                
  • select in order by , mysql排序
                                    dcj3sjt126com
mysql
                                    
    If i select like this: 
SELECT id FROM users WHERE id IN(3,4,8,1); 
This by default will select users in this order 
1,3,4,8, 
I would like to select them in the same order that i put IN() values so: 
    
                                
    • 
                                
  • 页面校验-新建项目
                                    fanxiaolong
页面校验
                                    
    $(document).ready(
	function() {
			var flag = true;
		$('#changeform').submit(function() {
			var projectScValNull = true;
			var s ="";
			var parent_id = $("#parent_id").v
    
                                
    • 
                                
  • Ehcache（02）——ehcache.xml简介
                                    234390216
ehcacheehcache.xml简介
                                    
    ehcache.xml简介 
  
       ehcache.xml文件是用来定义Ehcache的配置信息的，更准确的来说它是定义CacheManager的配置信息的。根据之前我们在《Ehcache简介》一文中对CacheManager的介绍我们知道一切Ehcache的应用都是从CacheManager开始的。在不指定配置信
    
                                
    • 
                                
  • junit 4.11中三个新功能
                                    jackyrong
java
                                    
    junit 4.11中两个新增的功能，首先是注解中可以参数化，比如 
 
 


import static org.junit.Assert.assertEquals;
 
import java.util.Arrays;
 
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runn
    
                                
    • 
                                
  • 国外程序员爱用苹果Mac电脑的10大理由
                                    php教程分享
windowsPHPunixMicrosoftperl
                                    
    Mac 在国外很受欢迎，尤其是在 设计/web开发/IT 人员圈子里。普通用户喜欢 Mac 可以理解，毕竟 Mac 设计美观，简单好用，没有病毒。那么为什么专业人士也对 Mac 情有独钟呢？从个人使用经验来看我想有下面几个原因： 
1、Mac OS X 是基于 Unix 的 
这一点太重要了，尤其是对开发人员，至少对于我来说很重要，这意味着Unix 下一堆好用的工具都可以随手捡到。如果你是个 wi
    
                                
    • 
                                
  • 位运算、异或的实际应用
                                    wenjinglian
位运算
                                    
    一． 位操作基础，用一张表描述位操作符的应用规则并详细解释。 
      二． 常用位操作小技巧，有判断奇偶、交换两数、变换符号、求绝对值。 
      三． 位操作与空间压缩，针对筛素数进行空间压缩。 
   &n
    
                                
    • 
                                
  • weblogic部署项目出现的一些问题（持续补充中……）
                                    Everyday都不同
weblogic部署失败
                                    
    好吧，weblogic的问题确实…… 
  
问题一： 
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [zip:E:/weblogic/user_projects/domains/base_domain/serve
    
                                
    • 
                                
  • tomcat7性能调优（01）
                                    toknowme
tomcat7
                                    
      
    
Tomcat优化：   1、最大连接数最大线程等设置    
<Connector port="8082" protocol="HTTP/1.1" 
               useBodyEncodingForURI="t
    
                                
    • 
                                
  • PO VO DAO DTO BO TO概念与区别
                                    xp9802
javaDAO设计模式bean领域模型
                                    
    O/R Mapping 是 Object Relational Mapping（对象关系映射）的缩写。通俗点讲，就是将对象与关系数据库绑定，用对象来表示关系数据。在O/R Mapping的世界里，有两个基本的也是重要的东东需要了解，即VO，PO。 
它们的关系应该是相互独立的，一个VO可以只是PO的部分，也可以是多个PO构成，同样也可以等同于一个PO（指的是他们的属性）。这样，PO独立出来，数据持
    
                                
    •