Nutch1.7源码再研究之---15 Parse分析
这一节，开始分析Parse部分。
---------------------------------------
if (!Fetcher.isParsing(job)) {
parseSegment.parse(segs[0]); // parse it, if needed
}
如果fetch的过程中没有parse内容，则这里开始进行parse.
先看下parseSegment.parse函数代码。
-----------
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
long start = System.currentTimeMillis();
if (LOG.isInfoEnabled()) {
LOG.info("ParseSegment: starting at " + sdf.format(start));
LOG.info("ParseSegment: segment: " + segment);
}
JobConf job = new NutchJob(getConf());
job.setJobName("parse " + segment);
FileInputFormat.addInputPath(job, new Path(segment, Content.DIR_NAME));
job.set(Nutch.SEGMENT_NAME_KEY, segment.getName());
job.setInputFormat(SequenceFileInputFormat.class);
job.setMapperClass(ParseSegment.class);
job.setReducerClass(ParseSegment.class);
FileOutputFormat.setOutputPath(job, segment);
job.setOutputFormat(ParseOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(ParseImpl.class);
JobClient.runJob(job);
这里需要关注的是
job.setMapperClass(ParseSegment.class);
job.setReducerClass(ParseSegment.class);
所以我们开始分析 ParseSegment.class.
--------------------------------------------------
先是一段转换代码：
// convert on the fly from old UTF8 keys
if (key instanceof Text) {
newKey.set(key.toString());
key = newKey;
}
这个就不解释了。
---------------------------- 接下来是抓取状态校验
int status =
Integer.parseInt(content.getMetadata().get(Nutch.FETCH_STATUS_KEY));
if (status != CrawlDatum.STATUS_FETCH_SUCCESS) {
// content not fetched successfully, skip document
LOG.debug("Skipping " + key + " as content is not fetched successfully");
return;
}
如果不成功，则不用解析，这个是当然的，这个状态的值是fetch过程中写入的。
------------ 接下来是测试fetch的content是否是完整的，这个学过http协议的都知道怎么回事，就不解释了!
if (skipTruncated && isTruncated(content)) {
return;
}
-----------------------------------------------
接下来就是真正的解析的核心了
ParseResult parseResult = null;
try {
parseResult = new ParseUtil(getConf()).parse(content);
} catch (Exception e) {
LOG.warn("Error parsing: " + key + ": " + StringUtils.stringifyException(e));
return;
}
---我们进入parse函数
首先是获取<type,url>对应的解析器
Parser[] parsers = null;
try {
parsers = this.parserFactory.getParsers(content.getContentType(),
content.getUrl() != null ? content.getUrl():"");
} catch (ParserNotFound e) {
if (LOG.isWarnEnabled()) {
LOG.warn("No suitable parser found when trying to parse content " + content.getUrl() +
" of type " + content.getContentType());
}
throw new ParseException(e.getMessage());
}
这个解析器的插件配置项见：
<property>
<name>plugin.includes</name>
<value>urlnormalizer-(pass|regex|basic)|urlfilter-regex|protocol-http|parse-(html|tika)|index-(basic|anchor)|indexer-solr|scoring-opic</value>
<description>Regular expression naming plugin directory names to
include. Any plugin not matching this expression is excluded.
In any case you need at least include the nutch-extensionpoints plugin. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins. In order to use HTTPS please enable
protocol-httpclient, but be aware of possible intermittent problems with the
underlying commons-httpclient library.
</description>
</property>
一般情况下，对应的解析器是：
org.apache.nutch.parse.html.HtmlParser
－－－－－－－－－－－－－接下来循环执行各个解析器。
ParseResult parseResult = null;
for (int i=0; i<parsers.length; i++) {
if (LOG.isDebugEnabled()) {
LOG.debug("Parsing [" + content.getUrl() + "] with [" + parsers[i] + "]");
}
if (maxParseTime!=-1)
parseResult = runParser(parsers[i], content);
else
parseResult = parsers[i].getParse(content);
if (parseResult != null && !parseResult.isEmpty())
return parseResult;
}
这里由于默认的maxParseTime为30，所以这里执行的是
parseResult = runParser(parsers[i], content);
---------------这里有一个知识点就是java并发的一个东西，需要看2段代码
1）
private ParserFactory parserFactory;
/** Parser timeout set to 30 sec by default. Set -1 to deactivate **/
private int maxParseTime = 30;
private ExecutorService executorService;
public ParseUtil(Configuration conf) {
this.parserFactory = new ParserFactory(conf);
maxParseTime=conf.getInt("parser.timeout", 30);
executorService = Executors.newCachedThreadPool(new ThreadFactoryBuilder()
.setNameFormat("parse-%d").setDaemon(true).build());
}
2）
private ParseResult runParser(Parser p, Content content) {
ParseCallable pc = new ParseCallable(p, content);
Future<ParseResult> task = executorService.submit(pc);
ParseResult res = null;
try {
res = task.get(maxParseTime, TimeUnit.SECONDS);
} catch (Exception e) {
LOG.warn("Error parsing " + content.getUrl() + " with " + p, e);
task.cancel(true);
} finally {
pc = null;
}
return res;
}
解释：
在1）中初始化，创建线程池的方法的官方解释：
Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks.
创建一个线程池，这个线程池在需要的时候创建新的线程。
可以的话，会复用之前已经创建的线程。
当有很多异步的短任务需要执行时，会明显改善系统的性能。
Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool.
需要的话，会复用之前创建好的线程，如果没有可用的线程，新的线程被创建且加入到线程池里。
Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources.
60秒内没有被使用的线程会销毁，因此，一个闲置很长时间的线程池不会消耗任何资源，因为空了呗。
Note that pools with similar properties but different details (for example, timeout parameters) may be created using ThreadPoolExecutor constructors.
Returns:
the newly created thread pool
返回新创建的线程池。
----------------------------------------
executorService.submit(pc);------------任务提交
ParseCallable pc = new ParseCallable(p, content);------构造任务
有必要看看ParseCallable的源码
---
package org.apache.nutch.parse;
import java.util.concurrent.Callable;
import org.apache.nutch.protocol.Content;
class ParseCallable implements Callable<ParseResult> {
private Parser p;
private Content content;
public ParseCallable(Parser p, Content content) {
this.p = p;
this.content = content;
}
@Override
public ParseResult call() throws Exception {
return p.getParse(content);
}
}
重点就是： return p.getParse(content);
前面说过一般情况下解析器是HtmlParser.
关于HtmlParser.getParse()源码分析将新开一篇博客讲解。