Fetcher类的工作流程

Fetcher类工作流程：



FileInputFormat.addInputPath(job, new Path(segment, CrawlDatum.GENERATE_DIR_NAME));

job.setInputFormat(InputFormat.class);



----------------第一部分------------------------

job.setMapRunnerClass(Fetcher.class);



Fetcher类实现了MapRunnable<Text, CrawlDatum, Text, NutchWritable>这个接口，主要完成了生产者的启动与消费者的启动。

Fetcher extends Configured implements Tool,MapRunnable<Text, CrawlDatum, Text, NutchWritable>

public void run(RecordReader<Text, CrawlDatum> input,OutputCollector<Text, NutchWritable> output,Reporter reporter){

	启动生产者

	feeder = new QueueFeeder(input, fetchQueues, threadCount * queueDepthMuliplier);

	启动消费者

	for (int i = 0; i < threadCount; i++) {       // spawn threads

      		new FetcherThread(getConf()).start(); 

    	}

}

FetcherThread类完成网页的下载，并用如下的方法多路输出。





----------------第二部分：多路输出------------------------

FileOutputFormat.setOutputPath(job, segment);

job.setOutputFormat(FetcherOutputFormat.class);/** Splits FetcherOutput entries into multiple map files. */

output.collect(key, new NutchWritable(datum));

output.collect(key, new NutchWritable(content));

output.collect(url, new NutchWritable(new ParseImpl(new ParseText(parse.getText()),parseData, parse.isCanonical())));