flume定制 之 taildir 目录递归

转载：https://www.jianshu.com/p/9cd850b49dbf

背景 : flume taildir源码只支持监控一级目录下的文件
目标 : 实现目录递归, 即可以同时监控子目录.

定制步骤 :

1. 下载flume源码
2. 修改flume-taildir-source源码

flume版本: 1.9.2

flume taildir收集监控文件的代码位置:
flume-taildir-source模块 -> TaildirMatcher.java -> getMatchingFilesNoCache方法

新增方法: getMatchingFilesNoCacheRecurseFolder , recurseFolder

  List getMatchingFiles() {
    long now = TimeUnit.SECONDS.toMillis(
        TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis()));
    long currentParentDirMTime = parentDir.lastModified();
    List result;

    // calculate matched files if
    // - we don't want to use cache (recalculate every time) OR
    // - directory was clearly updated after the last check OR
    // - last mtime change wasn't already checked for sure
    //   (system clock hasn't passed that second yet)
    if (!cachePatternMatching ||
        lastSeenParentDirMTime < currentParentDirMTime ||
        !(currentParentDirMTime < lastCheckedTime)) {
//      lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCache());
//    使用自己的递归目录方法getMatchingFilesNoCacheRecurseFolder
      lastMatchedFiles = sortByLastModifiedTime(getMatchingFilesNoCacheRecurseFolder());
      lastSeenParentDirMTime = currentParentDirMTime;
      lastCheckedTime = now;
    }

 private List getMatchingFilesNoCacheRecurseFolder() {
    List result = Lists.newArrayList();

    List paths = recurseFolder(parentDir);
    for(Path path:paths){
      try (DirectoryStream stream = Files.newDirectoryStream(path, fileFilter)) {
        for (Path entry : stream) {
          result.add(entry.toFile());
        }
      } catch (IOException e) {
        e.printStackTrace();
      }
    }

    String matchedFileNames = result.stream().map(r-> r.getAbsolutePath()).collect(Collectors.joining("\n"));
    logger.debug("============================matched files=======================================");
    logger.debug(matchedFileNames);
    logger.debug("=============================matched files======================================");
    return result;
  }


 public List recurseFolder(File root){
    List allParentFolders = new ArrayList<>();
    allParentFolders.add(root.toPath());

    if (root.exists()) {
      File[] files = root.listFiles();
      if (null == files || files.length == 0) {
        return allParentFolders;
      } else {
        for (File subFile : files) {
          if (subFile.isDirectory()) {
            allParentFolders.addAll(recurseFolder(subFile));
          }
        }
      }
    }
    return allParentFolders;
  }
  

这种方法在设计上来讲, 不够雅观, "/logs/.*.log"这个正则表达式,从任何角度来看都应该只匹配/logs本目录下的文件,不适合表示其子目录. 你可以进一步考虑更完美的实现方式,但是本质实现是类似的.