Flume 源码学习（一）Source介绍

Source是Flume NG三大组件的第一个，是数据收集的来源，也是我们今天学习的主题。

类图

先来看一下Source组件类图：

从上面类图，可以清晰看出，Source接口实现了NamedComponent接口和LifecycleWare接口。而AbstractSource抽象类可以看做是适配器模式的一种应用，实现了Source接口的所有业务不相关方法。

关键接口声明

接口名	作用
NamedComponent	Enables a component to be tagged with a name so that it can be referred to uniquely within the configuration system.
LifecycleAware	An interface implemented by any class that has a defined, stateful,lifecycle.

AbstractSource

private ChannelProcessor channelProcessor;
  private String name;

  private LifecycleState lifecycleState;

  public AbstractSource() {
    lifecycleState = LifecycleState.IDLE;
  }

  @Override
  public synchronized void start() {
    Preconditions.checkState(channelProcessor != null,
        "No channel processor configured");

    lifecycleState = LifecycleState.START;
  }

  @Override
  public synchronized void stop() {
    lifecycleState = LifecycleState.STOP;
  }

  @Override
  public synchronized void setChannelProcessor(ChannelProcessor cp) {
    channelProcessor = cp;
  }

  @Override
  public synchronized ChannelProcessor getChannelProcessor() {
    return channelProcessor;
  }

  @Override
  public synchronized LifecycleState getLifecycleState() {
    return lifecycleState;
  }

  @Override
  public synchronized void setName(String name) {
    this.name = name;
  }

  @Override
  public synchronized String getName() {
    return name;
  }

  public String toString() {
      return this.getClass().getName() + "{name:" + name + ",state:" + lifecycleState +"}";
  }  

实现了最基本的LifecycyleAware接口和NamedComponentNamed定义的方法，子类只需要专注于具体业务。

ExecSource

接下来就是具体的Source实现了，我们先来看用的最多的ExecSource

public class ExecSource extends AbstractSource implements EventDrivenSource,
Configurable {

  private static final Logger logger = LoggerFactory
      .getLogger(ExecSource.class);

  private String shell;//exec执行的是shell
  private String command;//exec执行的是command
  private SourceCounter sourceCounter;//用于JMX查看的计数器
  private ExecutorService executor;//具体的Exec执行器
  private Future runnerFuture;//Exec执行的结果Future
  private long restartThrottle;//重新执行的间隔
  private boolean restart;//判断是否命令一次执行成功后需要重新执行
  private boolean logStderr;//是否输出stderr
  private Integer bufferCount;//缓冲区数量
  private long batchTimeout;//批量flush的timeout
  private ExecRunnable runner;//具体命令的执行代码
  private Charset charset;//编码

  @Override
  public void start() {//重写了父类的start()方法
    logger.info("Exec source starting with command:{}", command);

    executor = Executors.newSingleThreadExecutor();//单线程的执行器

    runner = new ExecRunnable(shell, command, getChannelProcessor(), sourceCounter,
        restart, restartThrottle, logStderr, bufferCount, batchTimeout, charset);

    // FIXME: Use a callback-like executor / future to signal us upon failure.
    runnerFuture = executor.submit(runner);

    /*
     * NB: This comes at the end rather than the beginning of the method because
     * it sets our state to running. We want to make sure the executor is alive
     * and well first.
     */
    sourceCounter.start();
    super.start();//调用父类start()方法更改Source的状态为started

    logger.debug("Exec source started");
  }

  @Override
  public void stop() {
    logger.info("Stopping exec source with command:{}", command);
    if(runner != null) {
      runner.setRestart(false);
      runner.kill();
    }

    if (runnerFuture != null) {
      logger.debug("Stopping exec runner");
      runnerFuture.cancel(true);
      logger.debug("Exec runner stopped");
    }
    executor.shutdown();

    while (!executor.isTerminated()) {
      logger.debug("Waiting for exec executor service to stop");
      try {
        executor.awaitTermination(500, TimeUnit.MILLISECONDS);
      } catch (InterruptedException e) {
        logger.debug("Interrupted while waiting for exec executor service "
            + "to stop. Just exiting.");
        Thread.currentThread().interrupt();
      }
    }

    sourceCounter.stop();
    super.stop();

    logger.debug("Exec source with command:{} stopped. Metrics:{}", command,
        sourceCounter);
  }
  //获取用户的flume-ng启动是配置文件中配置的参数
  @Override
  public void configure(Context context) {
    command = context.getString("command");//

    Preconditions.checkState(command != null,
        "The parameter command must be specified");

    restartThrottle = context.getLong(ExecSourceConfigurationConstants.CONFIG_RESTART_THROTTLE,
        ExecSourceConfigurationConstants.DEFAULT_RESTART_THROTTLE);

    restart = context.getBoolean(ExecSourceConfigurationConstants.CONFIG_RESTART,
        ExecSourceConfigurationConstants.DEFAULT_RESTART);

    logStderr = context.getBoolean(ExecSourceConfigurationConstants.CONFIG_LOG_STDERR,
        ExecSourceConfigurationConstants.DEFAULT_LOG_STDERR);

    bufferCount = context.getInteger(ExecSourceConfigurationConstants.CONFIG_BATCH_SIZE,
        ExecSourceConfigurationConstants.DEFAULT_BATCH_SIZE);

    batchTimeout = context.getLong(ExecSourceConfigurationConstants.CONFIG_BATCH_TIME_OUT,
        ExecSourceConfigurationConstants.DEFAULT_BATCH_TIME_OUT);

    charset = Charset.forName(context.getString(ExecSourceConfigurationConstants.CHARSET,
        ExecSourceConfigurationConstants.DEFAULT_CHARSET));

    shell = context.getString(ExecSourceConfigurationConstants.CONFIG_SHELL, null);

    if (sourceCounter == null) {
      sourceCounter = new SourceCounter(getName());
    }
  }
  //这个类是重点，包含了Flume怎么执行命令
  private static class ExecRunnable implements Runnable {

    public ExecRunnable(String shell, String command, ChannelProcessor channelProcessor,
        SourceCounter sourceCounter, boolean restart, long restartThrottle,
        boolean logStderr, int bufferCount, long batchTimeout, Charset charset) {
      this.command = command;
      this.channelProcessor = channelProcessor;
      this.sourceCounter = sourceCounter;
      this.restartThrottle = restartThrottle;
      this.bufferCount = bufferCount;
      this.batchTimeout = batchTimeout;
      this.restart = restart;
      this.logStderr = logStderr;
      this.charset = charset;
      this.shell = shell;
    }

    private final String shell;
    private final String command;
    private final ChannelProcessor channelProcessor;
    private final SourceCounter sourceCounter;
    private volatile boolean restart;
    private final long restartThrottle;
    private final int bufferCount;
    private long batchTimeout;
    private final boolean logStderr;
    private final Charset charset;
    private Process process = null;
    private SystemClock systemClock = new SystemClock();
    private Long lastPushToChannel = systemClock.currentTimeMillis();
    ScheduledExecutorService timedFlushService;
    ScheduledFuture future;

    @Override
    public void run() {
      do {
        String exitCode = "unknown";
        BufferedReader reader = null;
        String line = null;
        final List eventList = new ArrayList();//从shell/command读取的内容的缓冲队列，定时刷新到channel

        timedFlushService = Executors.newSingleThreadScheduledExecutor(
                new ThreadFactoryBuilder().setNameFormat(
                "timedFlushExecService" +
                Thread.currentThread().getId() + "-%d").build());//定时Flush从shell/command的输出到channel的executor
        try {
          if(shell != null) {
            String[] commandArgs = formulateShellCommand(shell, command);
            process = Runtime.getRuntime().exec(commandArgs);//调用Runtime执行shell
          }  else {
            String[] commandArgs = command.split("\\s+");
            process = new ProcessBuilder(commandArgs).start();//调用ProcessBuilder启动进程
          }
          reader = new BufferedReader(
              new InputStreamReader(process.getInputStream(), charset));//获取进程的输出

          // StderrLogger dies as soon as the input stream is invalid
          StderrReader stderrReader = new StderrReader(new BufferedReader(
              new InputStreamReader(process.getErrorStream(), charset)), logStderr);
          stderrReader.setName("StderrReader-[" + command + "]");
          stderrReader.setDaemon(true);
          stderrReader.start();

          future = timedFlushService.scheduleWithFixedDelay(new Runnable() {
              @Override
              public void run() {
                try {
                  synchronized (eventList) {
                    if(!eventList.isEmpty() && timeout()) {
                      flushEventBatch(eventList);//如果eventList不为空并且timeout，执行一次flush操作
                    }
                  }
                } catch (Exception e) {
                  logger.error("Exception occured when processing event batch", e);
                  if(e instanceof InterruptedException) {
                      Thread.currentThread().interrupt();
                  }
                }
              }
          },
          batchTimeout, batchTimeout, TimeUnit.MILLISECONDS);

          while ((line = reader.readLine()) != null) {
            synchronized (eventList) {
              sourceCounter.incrementEventReceivedCount();//
              eventList.add(EventBuilder.withBody(line.getBytes(charset)));//读取到一行内容就对应一个Event
              if(eventList.size() >= bufferCount || timeout()) {//如果缓冲size大于等于最大的缓冲量或者timeout了，直接触发flush
                flushEventBatch(eventList);
              }
            }
          }

          synchronized (eventList) {
              if(!eventList.isEmpty()) {
                flushEventBatch(eventList);//最后一次性把没有flush的内容flush
              }
          }
        } catch (Exception e) {
          logger.error("Failed while running command: " + command, e);
          if(e instanceof InterruptedException) {
            Thread.currentThread().interrupt();
          }
        } finally {
          if (reader != null) {
            try {
              reader.close();
            } catch (IOException ex) {
              logger.error("Failed to close reader for exec source", ex);
            }
          }
          exitCode = String.valueOf(kill());
        }
        if(restart) {
          logger.info("Restarting in {}ms, exit code {}", restartThrottle,
              exitCode);
          try {
            Thread.sleep(restartThrottle);
          } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
          }
        } else {
          logger.info("Command [" + command + "] exited with " + exitCode);
        }
      } while(restart);
    }
    //具体的flush操作
    private void flushEventBatch(List eventList){
      channelProcessor.processEventBatch(eventList);//调用cp来处理
      sourceCounter.addToEventAcceptedCount(eventList.size());
      eventList.clear();
      lastPushToChannel = systemClock.currentTimeMillis();
    }
    //timeout 规则，当前时间距离上次flush时间毫秒数是否大于我们配置的batchTimeout
    private boolean timeout(){
      return (systemClock.currentTimeMillis() - lastPushToChannel) >= batchTimeout;
    }

    private static String[] formulateShellCommand(String shell, String command) {
      String[] shellArgs = shell.split("\\s+");
      String[] result = new String[shellArgs.length + 1];
      System.arraycopy(shellArgs, 0, result, 0, shellArgs.length);
      result[shellArgs.length] = command;
      return result;
    }

    public int kill() {
      if(process != null) {
        synchronized (process) {
          process.destroy();//杀死进程

          try {
            int exitValue = process.waitFor();

            // Stop the Thread that flushes periodically
            if (future != null) {
                future.cancel(true);//停止future
            }

            if (timedFlushService != null) {
              timedFlushService.shutdown();
              while (!timedFlushService.isTerminated()) {
                try {
                  timedFlushService.awaitTermination(500, TimeUnit.MILLISECONDS);//停止定时flush线程
                } catch (InterruptedException e) {
                  logger.debug("Interrupted while waiting for exec executor service "
                    + "to stop. Just exiting.");
                  Thread.currentThread().interrupt();
                }
              }
            }
            return exitValue;
          } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
          }
        }
        return Integer.MIN_VALUE;
      }
      return Integer.MIN_VALUE / 2;
    }
    public void setRestart(boolean restart) {
      this.restart = restart;
    }
  }
  private static class StderrReader extends Thread {
    private BufferedReader input;
    private boolean logStderr;

    protected StderrReader(BufferedReader input, boolean logStderr) {
      this.input = input;
      this.logStderr = logStderr;
    }

    @Override
    public void run() {
      try {
        int i = 0;
        String line = null;
        while((line = input.readLine()) != null) {
          if(logStderr) {
            // There is no need to read 'line' with a charset
            // as we do not to propagate it.
            // It is in UTF-16 and would be printed in UTF-8 format.
            logger.info("StderrLogger[{}] = '{}'", ++i, line);//调用logger输出process的stderr
          }
        }
      } catch (IOException e) {
        logger.info("StderrLogger exiting", e);
      } finally {
        try {
          if(input != null) {
            input.close();
          }
        } catch (IOException ex) {
          logger.error("Failed to close stderr reader for exec source", ex);
        }
      }
    }
  }
}

代码总体比较简单。

ChannelProcessor

接下来就是ChannelProcessor这个重要角色了，首先来看这个类的四个重要属性：

  private final ChannelSelector selector;
  private final InterceptorChain interceptorChain;
  private ExecutorService execService;
  BlockingQueue taskQueue;

四个属性的作用分别是：

• selector：Flume中的selector。selector的解释我们直接看Flume User guide：

As discussed in previous section, Flume supports fanning out the flow from one source to multiple channels. There are two modes of fan out, replicating and multiplexing. In the replicating flow, the event is sent to all the configured channels. In case of multiplexing, the event is sent to only a subset of qualifying channels. To fan out the flow, one needs to specify a list of channels for a source and the policy for the fanning it out. This is done by adding a channel “selector” that can be replicating or multiplexing. Then further specify the selection rules if it’s a multiplexer. If you don’t specify a selector, then by default it’s replicating。

描述的很清楚。

• interceptorChain： 拦截器链，interceptor是用来在Event 发完channal前进行拦截，过滤和处理。比如可以字自定义拦截器过滤掉不需要的events。
• execService：optional channel 对应的Event处理的线程池
• taskQuene：execService使用的缓冲队列。

四个属性的初始化和配置代码如下：

 @Override
  public void configure(Context context) {
    int queueSize = context.getInteger("pendingTransactions", 20);
    taskQueue = new ArrayBlockingQueue(queueSize, true);
    ThreadFactory factory = new ThreadFactoryBuilder()
      .setNameFormat("OptionalChannelProcessorThread").build();
    this.execService =
      new ThreadPoolExecutor(1, 1, 0, TimeUnit.MILLISECONDS, taskQueue,
        factory, new ThreadPoolExecutor.DiscardPolicy());
    configureInterceptors(context);
  }

  // WARNING: throws FlumeException (is that ok?)
  private void configureInterceptors(Context context) {

    List interceptors = Lists.newLinkedList();

    String interceptorListStr = context.getString("interceptors", "");
    if (interceptorListStr.isEmpty()) {
      return;
    }
    String[] interceptorNames = interceptorListStr.split("\\s+");

    Context interceptorContexts =
        new Context(context.getSubProperties("interceptors."));

    // run through and instantiate all the interceptors specified in the Context
    InterceptorBuilderFactory factory = new InterceptorBuilderFactory();
    for (String interceptorName : interceptorNames) {
      Context interceptorContext = new Context(
          interceptorContexts.getSubProperties(interceptorName + "."));
      String type = interceptorContext.getString("type");
      if (type == null) {
        LOG.error("Type not specified for interceptor " + interceptorName);
        throw new FlumeException("Interceptor.Type not specified for " +
          interceptorName);
      }
      try {
        Interceptor.Builder builder = factory.newInstance(type);
        builder.configure(interceptorContext);
        interceptors.add(builder.build());
      } catch (ClassNotFoundException e) {
        LOG.error("Builder class not found. Exception follows.", e);
        throw new FlumeException("Interceptor.Builder not found.", e);
      } catch (InstantiationException e) {
        LOG.error("Could not instantiate Builder. Exception follows.", e);
        throw new FlumeException("Interceptor.Builder not constructable.", e);
      } catch (IllegalAccessException e) {
        LOG.error("Unable to access Builder. Exception follows.", e);
        throw new FlumeException("Unable to access Interceptor.Builder.", e);
      }
    }

    interceptorChain.setInterceptors(interceptors);
  }

通过源代码学习可以帮助我们更好掌握怎么编写Flume的配置文件。

processEventBatch() 方法

好了，接下来就是最重要的processEventBatch()方法。还是先来看代码：

 public void processEventBatch(List events) {
    Preconditions.checkNotNull(events, "Event list must not be null");

    events = interceptorChain.intercept(events);

    Map> reqChannelQueue =
        new LinkedHashMap>();

    Map> optChannelQueue =
        new LinkedHashMap>();

    for (Event event : events) {
      List reqChannels = selector.getRequiredChannels(event);
      for (Channel ch : reqChannels) {
        List eventQueue = reqChannelQueue.get(ch);
        if (eventQueue == null) {
          eventQueue = new ArrayList();
          reqChannelQueue.put(ch, eventQueue);
        }
        eventQueue.add(event);
      }

      List optChannels = selector.getOptionalChannels(event);
      for (Channel ch: optChannels) {
        List eventQueue = optChannelQueue.get(ch);
        if (eventQueue == null) {
          eventQueue = new ArrayList();
          optChannelQueue.put(ch, eventQueue);
        }
        eventQueue.add(event);
      }
    }

    // Process required channels
    for (Channel reqChannel : reqChannelQueue.keySet()) {
      List batch = reqChannelQueue.get(reqChannel);
      executeChannelTransaction(reqChannel, batch, false);
    }

    // Process optional channels
    for (Channel optChannel : optChannelQueue.keySet()) {
      List batch = optChannelQueue.get(optChannel);
      execService.submit(new OptionalChannelTransactionRunnable(optChannel, batch));
    }
  }

从代码可知，events->required channels的工作直接由主线程立刻顺序执行，events->optional channels的工作丢给execService异步处理，处理方法也是直接执行executeChannelTransaction方法。

executeChannelTransaction()

再来看executeChannelTransaction()方法：

private static void executeChannelTransaction(Channel channel, List batch, boolean isOptional) {
    Transaction tx = channel.getTransaction();
    Preconditions.checkNotNull(tx, "Transaction object must not be null");
    try {
      tx.begin();

      for (Event event : batch) {
        channel.put(event);
      }

      tx.commit();
    } catch (Throwable t) {
      tx.rollback();
      if (t instanceof Error) {
        LOG.error("Error while writing to channel: " +
                channel, t);
        throw (Error) t;
      } else if(!isOptional) {
          throw new ChannelException("Unable to put batch on required " +
                  "channel: " + channel, t);
      }
    } finally {
      tx.close();
    }
  }

代码很简单，在事务上下文环境内往channel推送events，提交事务，出现异常事务回滚，最后事务关闭。
到现在，从Source产生Event再丢给Channel的流程也就结束了，其他Source如何产生Event大家可以根据源码自己学习。