Nutch学习笔记7---url的正则过滤机制研究

今天碰到问题，url正则过滤老是出问题，不爽之下，又打开源码了。

Crawl.java里有这么一段

for (i = 0; i < depth; i++) {             // generate new segment
      Path[] segs = generator.generate(crawlDb, segments, -1, topN, System
          .currentTimeMillis());
      if (segs == null) {
        LOG.info("Stopping at depth=" + i + " - no more URLs to fetch.");
        break;
      }
      fetcher.fetch(segs[0], threads);  // fetch it
      if (!Fetcher.isParsing(job)) {
        parseSegment.parse(segs[0]);    // parse it, if needed
      }
      crawlDbTool.update(crawlDb, segs, true, true); // update crawldb
    }

 

可以看到，生成下一步要去fetch抓取的url列表是由下面的代码决定：

Path[] segs = generator.generate(crawlDb, segments, -1, topN, System
          .currentTimeMillis());

 

跟踪org.apache.nutch.crawl.Generator类

public Path[] generate(Path dbDir, Path segments, int numLists, long topN, long curTime)
      throws IOException {

    JobConf job = new NutchJob(getConf());
    boolean filter = job.getBoolean(GENERATOR_FILTER, true);
    boolean normalise = job.getBoolean(GENERATOR_NORMALISE, true);
    return generate(dbDir, segments, numLists, topN, curTime, filter, normalise, false, 1);
  }

继续进入

generate(dbDir, segments, numLists, topN, curTime, filter, normalise, false, 1);

有这么一段

job.setMapperClass(Selector.class);
    job.setPartitionerClass(Selector.class);
    job.setReducerClass(Selector.class);

可见用了hadoop模式的map/reduce, 类是Selector.class.

 

跟踪Selector.java

map函数代码如下：

/** Select & invert subset due for fetch. */
    public void map(Text key, CrawlDatum value,
        OutputCollector<FloatWritable,SelectorEntry> output, Reporter reporter)
        throws IOException {
      Text url = key;
      if (filter) {
        // If filtering is on don't generate URLs that don't pass
        // URLFilters
        try {
          if (filters.filter(url.toString()) == null) return;
        } catch (URLFilterException e) {
          if (LOG.isWarnEnabled()) {
            LOG.warn("Couldn't filter url: " + url + " (" + e.getMessage() + ")");
          }
        }
      }
      CrawlDatum crawlDatum = value;

      // check fetch schedule
      if (!schedule.shouldFetch(url, crawlDatum, curTime)) {
        LOG.debug("-shouldFetch rejected '" + url + "', fetchTime="
            + crawlDatum.getFetchTime() + ", curTime=" + curTime);
        return;
      }

      LongWritable oldGenTime = (LongWritable) crawlDatum.getMetaData().get(
          Nutch.WRITABLE_GENERATE_TIME_KEY);
      if (oldGenTime != null) { // awaiting fetch & update
        if (oldGenTime.get() + genDelay > curTime) // still wait for
        // update
        return;
      }
      float sort = 1.0f;
      try {
        sort = scfilters.generatorSortValue(key, crawlDatum, sort);
      } catch (ScoringFilterException sfe) {
        if (LOG.isWarnEnabled()) {
          LOG.warn("Couldn't filter generatorSortValue for " + key + ": " + sfe);
        }
      }

      if (restrictStatus != null
        && !restrictStatus.equalsIgnoreCase(CrawlDatum.getStatusName(crawlDatum.getStatus()))) return;

      // consider only entries with a score superior to the threshold
      if (scoreThreshold != Float.NaN && sort < scoreThreshold) return;

      // consider only entries with a retry (or fetch) interval lower than threshold
      if (intervalThreshold != -1 && crawlDatum.getFetchInterval() > intervalThreshold) return;

      // sort by decreasing score, using DecreasingFloatComparator
      sortValue.set(sort);
      // record generation time
      crawlDatum.getMetaData().put(Nutch.WRITABLE_GENERATE_TIME_KEY, genTime);
      entry.datum = crawlDatum;
      entry.url = key;
      output.collect(sortValue, entry); // invert for sort by score
    }

重点在于：

filters.filter(url.toString())

看看filter函数

public String filter(String urlString) throws URLFilterException {
    for (int i = 0; i < this.filters.length; i++) {
      if (urlString == null)
        return null;
      urlString = this.filters[i].filter(urlString);

    }
    return urlString;
  }

也就是说：

任何一个url,都必须经过N个符合条件的url过滤器的检查，一旦其中一个检查没通过，那就没通过。

这里的url过滤器，不用说，又是插件。

~~~~~~~~~~~~~~~~~~~~~~~~~~

开辟第二战场，我们去看过滤器的代码，我的 conf/nutch-site.xml文件里是这么定义的。

<name>plugin.includes</name>
  <value>protocol-http|urlfilter-(domain|regex)|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic|urlnormalizer-(pass|regex|basic)|htmlparsefilter-youku</value>

这里有2个url过滤器，urlfilter-domain,urlfilter-regex.

重点放在urlfilter-regex的研究上！

~~~~~~~~~~~~~~~~~~~~~~~~~~

这个插件的代码位于$...nutch-1.7/src/plugin/urlfilterregex/src/java/org/apache/nutch/urlfilter/regex

目录下的 RegexURLFilter.java

那我们的任务就是研究这个类的filter函数。

这个类本身没做filter,那就看父类

import org.apache.nutch.urlfilter.api.RegexURLFilterBase;

这个类在哪呢：

src/plugin/lib-regex-filter/src/java/org/apache/nutch/urlfilter/api/RegexURLFilterBase.java

最重要的函数是filter

public String filter(String url) {
    for (RegexRule rule : rules) {
      if (rule.match(url)) {
        return rule.accept() ? url : null;
      }
    };
    return null;
  }

结论：

拿到一个url后，跟conf/regex-urlfilter.txt里的正则表达式挨个匹配

找到第一个匹配规则就肯定会返回结果，至于结果就根据第一个字符是+还是-表示通过还是失败。

正则过滤规则分析完毕！