ninjuli
ninjuli

Nutch1.0源码分析-----抓取部分

简单的分析了nutch抓取过程，涉及到的mapredue等内容在这不做讨论，时间仓促，很多地方写得不具体，以后有时间再慢慢修改，工作需要又得马上分析nutch相关配置文件，分析整理后会发布上来。转载请注明出处

 

1.1 抓取目录分析

一共生成5个文件夹,分别是:

l         crawldb目录存放下载的URL,以及下载的日期,用来页面更新检查时间.

l         linkdb目录存放URL的互联关系,是下载完成后分析得到的.

l         segments:存放抓取的页面,下面子目录的个数于获取的页面层数有关系,通常每一层页面会独立存放一个子目录,子目录名称为时间,便于管理.比如我这只抓取了一层页面就只生成了20090508173137目录.每个子目录里又有6个子文件夹如下:

Ø         content：每个下载页面的内容。

Ø         crawl_fetch：每个下载URL的状态。

Ø         crawl_generate：待下载URL集合。

Ø         crawl_parse：包含来更新crawldb的外部链接库。

Ø         parse_data：包含每个URL解析出的外部链接和元数据

Ø         parse_text：包含每个解析过的URL的文本内容。

l         indexs：存放每次下载的独立索引目录

l         index：符合Lucene格式的索引目录，是indexs里所有index合并后的完整索引

1.2 Crawl过程概述

引用到的类主要有以下9个：

1、  nutch.crawl.Inject

用来给抓取数据库添加URL的插入器

2、  nutch.crawl.Generator

用来生成待下载任务列表的生成器

3、  nutch.fetcher.Fetcher

完成抓取特定页面的抓取器

4、  nutch.parse.ParseSegment

负责内容提取和对下级URL提取的内容进行解析的解析器

5、  nutch.crawl.CrawlDb

负责数据库管理的数据库管理工具

6、  nutch.crawl.LinkDb

负责链接管理

7、  nutch.indexer.Indexer

负责创建索引的索引器

8、  nutch.indexer.DeleteDuplicates

删除重复数据

9、  nutch.indexer.IndexMerger

对当前下载内容局部索引和历史索引进行合并的索引合并器

1.3  抓取过程分析

1.3.1   inject方法

描述：初始化爬取的crawldb,读取URL配置文件,把内容注入爬取数据库.

首先会找到读取URL配置文件的目录urls.如果没创建此目录,nutch1.0下会报错.

得到hadoop处理的临时文件夹:

/tmp/hadoop-Administrator/mapred/

日志信息如下:

2009-05-08 15:41:36,640 INFO  Injector - Injector: starting

2009-05-08 15:41:37,031 INFO  Injector - Injector: crawlDb: 20090508/crawldb

2009-05-08 15:41:37,781 INFO  Injector - Injector: urlDir: urls

接着设置一些初始化信息.

调用hadoop包JobClient.runJob方法,跟踪进入JobClient下的submitJob方法进行提交整个过程.具体原理又涉及到另一个开源项目hadoop的分析，它包括了复杂的

MapReduce架构，此处不做分析。

查看submitJob方法,首先获得jobid,执行configureCommandLineOptions方法后会在上边的临时文件夹生成一个system文件夹,同时在它下边生成一个job_local_0001文件夹.执行writeSplitsFile后在job_local_0001下生成job.split文件.执行writeXml写入job.xml,然后执行jobSubmitClient.submitJob正式提交整个job流程,日志如下:

2009-05-08 15:41:36,640 INFO  Injector - Injector: starting

2009-05-08 15:41:37,031 INFO  Injector - Injector: crawlDb: 20090508/crawldb

2009-05-08 15:41:37,781 INFO  Injector - Injector: urlDir: urls

2009-05-08 15:52:41,734 INFO  Injector - Injector: Converting injected urls to crawl db entries.

2009-05-08 15:56:22,203 INFO  JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=

2009-05-08 16:08:20,796 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-08 16:08:20,984 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-08 16:24:42,593 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 16:38:29,437 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 16:38:29,546 INFO  MapTask - numReduceTasks: 1

2009-05-08 16:38:29,562 INFO  MapTask - io.sort.mb = 100

2009-05-08 16:38:29,687 INFO  MapTask - data buffer = 79691776/99614720

2009-05-08 16:38:29,687 INFO  MapTask - record buffer = 262144/327680

2009-05-08 16:38:29,718 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

2009-05-08 16:38:29,921 INFO  PluginRepository - Plugin Auto-activation mode: [true]

2009-05-08 16:38:29,921 INFO  PluginRepository - Registered Plugins:

2009-05-08 16:38:29,921 INFO  PluginRepository - the nutch core extension points (nutch-extensionpoints)

2009-05-08 16:38:29,921 INFO  PluginRepository - Basic Query Filter (query-basic)

2009-05-08 16:38:29,921 INFO  PluginRepository - Basic URL Normalizer (urlnormalizer-basic)

2009-05-08 16:38:29,921 INFO  PluginRepository - Basic Indexing Filter (index-basic)

2009-05-08 16:38:29,921 INFO  PluginRepository - Html Parse Plug-in (parse-html)

2009-05-08 16:38:29,921 INFO  PluginRepository - Site Query Filter (query-site)

2009-05-08 16:38:29,921 INFO  PluginRepository - Basic Summarizer Plug-in (summary-basic)

2009-05-08 16:38:29,921 INFO  PluginRepository - HTTP Framework (lib-http)

2009-05-08 16:38:29,921 INFO  PluginRepository - Text Parse Plug-in (parse-text)

2009-05-08 16:38:29,921 INFO  PluginRepository - Pass-through URL Normalizer (urlnormalizer-pass)

2009-05-08 16:38:29,921 INFO  PluginRepository - Regex URL Filter (urlfilter-regex)

2009-05-08 16:38:29,921 INFO  PluginRepository - Http Protocol Plug-in (protocol-http)

2009-05-08 16:38:29,921 INFO  PluginRepository - XML Response Writer Plug-in (response-xml)

2009-05-08 16:38:29,921 INFO  PluginRepository - Regex URL Normalizer (urlnormalizer-regex)

2009-05-08 16:38:29,921 INFO  PluginRepository - OPIC Scoring Plug-in (scoring-opic)

2009-05-08 16:38:29,921 INFO  PluginRepository - CyberNeko HTML Parser (lib-nekohtml)

2009-05-08 16:38:29,921 INFO  PluginRepository - Anchor Indexing Filter (index-anchor)

2009-05-08 16:38:29,921 INFO  PluginRepository - JavaScript Parser (parse-js)

2009-05-08 16:38:29,921 INFO  PluginRepository - URL Query Filter (query-url)

2009-05-08 16:38:29,921 INFO  PluginRepository - Regex URL Filter Framework (lib-regex-filter)

2009-05-08 16:38:29,921 INFO  PluginRepository - JSON Response Writer Plug-in (response-json)

2009-05-08 16:38:29,921 INFO  PluginRepository - Registered Extension-Points:

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Summarizer (org.apache.nutch.searcher.Summarizer)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Protocol (org.apache.nutch.protocol.Protocol)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Analysis (org.apache.nutch.analysis.NutchAnalyzer)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Field Filter (org.apache.nutch.indexer.field.FieldFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - HTML Parse Filter (org.apache.nutch.parse.HtmlParseFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Query Filter (org.apache.nutch.searcher.QueryFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Search Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch URL Normalizer (org.apache.nutch.net.URLNormalizer)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch URL Filter (org.apache.nutch.net.URLFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Online Search Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Indexing Filter (org.apache.nutch.indexer.IndexingFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Content Parser (org.apache.nutch.parse.Parser)

2009-05-08 16:38:29,921 INFO  PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)

2009-05-08 16:38:29,921 INFO  PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)

2009-05-08 16:38:29,968 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-08 16:38:29,984 WARN  RegexURLNormalizer - can't find rules for scope 'inject', using default

2009-05-08 16:38:29,984 INFO  MapTask - Starting flush of map output

2009-05-08 16:38:30,203 INFO  MapTask - Finished spill 0

2009-05-08 16:38:30,203 INFO  TaskRunner - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting

2009-05-08 16:38:30,218 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/urls/site.txt:0+19

2009-05-08 16:38:30,218 INFO  TaskRunner - Task 'attempt_local_0001_m_000000_0' done.

2009-05-08 16:38:30,234 INFO  LocalJobRunner -

2009-05-08 16:38:30,250 INFO  Merger - Merging 1 sorted segments

2009-05-08 16:38:30,265 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 53 bytes

2009-05-08 16:38:30,265 INFO  LocalJobRunner -

2009-05-08 16:38:30,390 INFO  TaskRunner - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting

2009-05-08 16:38:30,390 INFO  LocalJobRunner -

2009-05-08 16:38:30,390 INFO  TaskRunner - Task attempt_local_0001_r_000000_0 is allowed to commit now

2009-05-08 16:38:30,406 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to file:/tmp/hadoop-Administrator/mapred/temp/inject-temp-474192304

2009-05-08 16:38:30,406 INFO  LocalJobRunner - reduce > reduce

2009-05-08 16:38:30,406 INFO  TaskRunner - Task 'attempt_local_0001_r_000000_0' done.

 

执行完后返回的running值如下:

Job: job_local_0001

file: file:/tmp/hadoop-Administrator/mapred/system/job_local_0001/job.xml

tracking URL: http://localhost:8080/

2009-05-08 16:47:14,093 INFO  JobClient - Running job: job_local_0001

2009-05-08 16:49:51,859 INFO  JobClient - Job complete: job_local_0001

2009-05-08 16:51:36,062 INFO  JobClient - Counters: 11

2009-05-08 16:51:36,062 INFO  JobClient -   File Systems

2009-05-08 16:51:36,062 INFO  JobClient -     Local bytes read=51591

2009-05-08 16:51:36,062 INFO  JobClient -     Local bytes written=104337

2009-05-08 16:51:36,062 INFO  JobClient -   Map-Reduce Framework

2009-05-08 16:51:36,062 INFO  JobClient -     Reduce input groups=1

2009-05-08 16:51:36,062 INFO  JobClient -     Combine output records=0

2009-05-08 16:51:36,062 INFO  JobClient -     Map input records=1

2009-05-08 16:51:36,062 INFO  JobClient -     Reduce output records=1

2009-05-08 16:51:36,062 INFO  JobClient -     Map output bytes=49

2009-05-08 16:51:36,062 INFO  JobClient -     Map input bytes=19

2009-05-08 16:51:36,062 INFO  JobClient -     Combine input records=0

2009-05-08 16:51:36,062 INFO  JobClient -     Map output records=1

2009-05-08 16:51:36,062 INFO  JobClient -     Reduce input records=1

 

至此第一个runJob方法执行结束.

总结:待写

 

接下来就是生成crawldb文件夹,并把urls合并注入到它的里面.

JobClient.runJob(mergeJob);

CrawlDb.install(mergeJob, crawlDb);

这个过程首先会在前面提到的临时文件夹下生成job_local_0002目录,和上边一样同样会生成job.split和job.xml,接着完成crawldb的创建,最后删除临时文件夹temp下的文件.

至此inject过程结束.最后部分日志如下:

2009-05-08 17:03:57,250 INFO  Injector - Injector: Merging injected urls into crawl db.

2009-05-08 17:10:01,015 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-08 17:10:15,953 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-08 17:10:16,156 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-08 17:12:15,296 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 17:13:40,296 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 17:13:40,406 INFO  MapTask - numReduceTasks: 1

2009-05-08 17:13:40,406 INFO  MapTask - io.sort.mb = 100

2009-05-08 17:13:40,515 INFO  MapTask - data buffer = 79691776/99614720

2009-05-08 17:13:40,515 INFO  MapTask - record buffer = 262144/327680

2009-05-08 17:13:40,546 INFO  MapTask - Starting flush of map output

2009-05-08 17:13:40,765 INFO  MapTask - Finished spill 0

2009-05-08 17:13:40,765 INFO  TaskRunner - Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting

2009-05-08 17:13:40,765 INFO  LocalJobRunner - file:/tmp/hadoop-Administrator/mapred/temp/inject-temp-474192304/part-00000:0+143

2009-05-08 17:13:40,765 INFO  TaskRunner - Task 'attempt_local_0002_m_000000_0' done.

2009-05-08 17:13:40,796 INFO  LocalJobRunner -

2009-05-08 17:13:40,796 INFO  Merger - Merging 1 sorted segments

2009-05-08 17:13:40,796 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 53 bytes

2009-05-08 17:13:40,796 INFO  LocalJobRunner -

2009-05-08 17:13:40,906 WARN  NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2009-05-08 17:13:40,906 INFO  CodecPool - Got brand-new compressor

2009-05-08 17:13:40,906 INFO  TaskRunner - Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting

2009-05-08 17:13:40,906 INFO  LocalJobRunner -

2009-05-08 17:13:40,906 INFO  TaskRunner - Task attempt_local_0002_r_000000_0 is allowed to commit now

2009-05-08 17:13:40,921 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0002_r_000000_0' to file:/D:/work/workspace/nutch_crawl/20090508/crawldb/1896567745

2009-05-08 17:13:40,921 INFO  LocalJobRunner - reduce > reduce

2009-05-08 17:13:40,937 INFO  TaskRunner - Task 'attempt_local_0002_r_000000_0' done.

2009-05-08 17:13:46,781 INFO  JobClient - Running job: job_local_0002

2009-05-08 17:14:55,125 INFO  JobClient - Job complete: job_local_0002

2009-05-08 17:14:59,328 INFO  JobClient - Counters: 11

2009-05-08 17:14:59,328 INFO  JobClient -   File Systems

2009-05-08 17:14:59,328 INFO  JobClient -     Local bytes read=103875

2009-05-08 17:14:59,328 INFO  JobClient -     Local bytes written=209385

2009-05-08 17:14:59,328 INFO  JobClient -   Map-Reduce Framework

2009-05-08 17:14:59,328 INFO  JobClient -     Reduce input groups=1

2009-05-08 17:14:59,328 INFO  JobClient -     Combine output records=0

2009-05-08 17:14:59,328 INFO  JobClient -     Map input records=1

2009-05-08 17:14:59,328 INFO  JobClient -     Reduce output records=1

2009-05-08 17:14:59,328 INFO  JobClient -     Map output bytes=49

2009-05-08 17:14:59,328 INFO  JobClient -     Map input bytes=57

2009-05-08 17:14:59,328 INFO  JobClient -     Combine input records=0

2009-05-08 17:14:59,328 INFO  JobClient -     Map output records=1

2009-05-08 17:14:59,328 INFO  JobClient -     Reduce input records=1

2009-05-08 17:17:30,984 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-08 17:20:02,390 INFO  Injector - Injector: done

1.3.2   generate方法

描述：从爬取数据库中生成新的segment,然后从中生成待下载任务列表(fetchlist).

LockUtil.createLockFile(fs, lock, force);

首先执行上边方法后会在crawldb目录下生成.locked文件,猜测作用是防止crawldb的数据被修改,真实作用有待验证.

接着执行的过程和上边大同小异,可参考上边步骤,日志如下:

2009-05-08 17:37:18,218 INFO  Generator - Generator: Selecting best-scoring urls due for fetch.

2009-05-08 17:37:18,625 INFO  Generator - Generator: starting

2009-05-08 17:37:18,937 INFO  Generator - Generator: segment: 20090508/segments/20090508173137

2009-05-08 17:37:19,468 INFO  Generator - Generator: filtering: true

2009-05-08 17:37:22,312 INFO  Generator - Generator: topN: 50

2009-05-08 17:37:51,203 INFO  Generator - Generator: jobtracker is 'local', generating exactly one partition.

2009-05-08 17:39:57,609 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-08 17:40:05,234 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-08 17:40:05,406 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-08 17:40:05,437 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 17:40:06,062 INFO  FileInputFormat - Total input paths to process : 1

2009-05-08 17:40:06,109 INFO  MapTask - numReduceTasks: 1

省略插件加载日志……

2009-05-08 17:40:06,312 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-08 17:40:06,343 INFO  FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule

2009-05-08 17:40:06,343 INFO  AbstractFetchSchedule - defaultInterval=2592000

2009-05-08 17:40:06,343 INFO  AbstractFetchSchedule - maxInterval=7776000

2009-05-08 17:40:06,343 INFO  MapTask - io.sort.mb = 100

2009-05-08 17:40:06,437 INFO  MapTask - data buffer = 79691776/99614720

2009-05-08 17:40:06,437 INFO  MapTask - record buffer = 262144/327680

2009-05-08 17:40:06,453 WARN  RegexURLNormalizer - can't find rules for scope 'partition', using default

2009-05-08 17:40:06,453 INFO  MapTask - Starting flush of map output

2009-05-08 17:40:06,625 INFO  MapTask - Finished spill 0

2009-05-08 17:40:06,640 INFO  TaskRunner - Task:attempt_local_0003_m_000000_0 is done. And is in the process of commiting

2009-05-08 17:40:06,640 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/part-00000/data:0+143

2009-05-08 17:40:06,640 INFO  TaskRunner - Task 'attempt_local_0003_m_000000_0' done.

2009-05-08 17:40:06,656 INFO  LocalJobRunner -

2009-05-08 17:40:06,656 INFO  Merger - Merging 1 sorted segments

2009-05-08 17:40:06,656 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 78 bytes

2009-05-08 17:40:06,656 INFO  LocalJobRunner –

省略插件加载日志……

2009-05-08 17:40:06,875 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-08 17:40:06,906 INFO  FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule

2009-05-08 17:40:06,906 INFO  AbstractFetchSchedule - defaultInterval=2592000

2009-05-08 17:40:06,906 INFO  AbstractFetchSchedule - maxInterval=7776000

2009-05-08 17:40:06,906 WARN  RegexURLNormalizer - can't find rules for scope 'generate_host_count', using default

2009-05-08 17:40:06,906 INFO  TaskRunner - Task:attempt_local_0003_r_000000_0 is done. And is in the process of commiting

2009-05-08 17:40:06,906 INFO  LocalJobRunner -

2009-05-08 17:40:06,906 INFO  TaskRunner - Task attempt_local_0003_r_000000_0 is allowed to commit now

2009-05-08 17:40:06,906 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0003_r_000000_0' to file:/tmp/hadoop-Administrator/mapred/temp/generate-temp-1241774893937

2009-05-08 17:40:06,921 INFO  LocalJobRunner - reduce > reduce

2009-05-08 17:40:06,921 INFO  TaskRunner - Task 'attempt_local_0003_r_000000_0' done.

2009-05-08 17:40:21,468 INFO  JobClient - Running job: job_local_0003

2009-05-08 17:40:31,671 INFO  JobClient - Job complete: job_local_0003

2009-05-08 17:40:34,046 INFO  JobClient - Counters: 11

2009-05-08 17:40:34,046 INFO  JobClient -   File Systems

2009-05-08 17:40:34,046 INFO  JobClient -     Local bytes read=157400

2009-05-08 17:40:34,046 INFO  JobClient -     Local bytes written=316982

2009-05-08 17:40:34,046 INFO  JobClient -   Map-Reduce Framework

2009-05-08 17:40:34,046 INFO  JobClient -     Reduce input groups=1

2009-05-08 17:40:34,046 INFO  JobClient -     Combine output records=0

2009-05-08 17:40:34,046 INFO  JobClient -     Map input records=1

2009-05-08 17:40:34,046 INFO  JobClient -     Reduce output records=1

2009-05-08 17:40:34,046 INFO  JobClient -     Map output bytes=74

2009-05-08 17:40:34,046 INFO  JobClient -     Map input bytes=57

2009-05-08 17:40:34,046 INFO  JobClient -     Combine input records=0

2009-05-08 17:40:34,046 INFO  JobClient -     Map output records=1

2009-05-08 17:40:34,046 INFO  JobClient -     Reduce input records=1

 

接着还是执行submitJob方法提交整个generate过程,生成segments目录,删除临时文件,锁定文件等,当前segments下只生成了crawl_generate一个文件夹.

1.3.3   fetch 方法

描述：完成具体的下载任务

2009-05-11 09:45:13,984 WARN  Fetcher - Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.

2009-05-11 09:45:34,796 INFO  Fetcher - Fetcher: starting

2009-05-11 09:45:35,375 INFO  Fetcher - Fetcher: segment: 20090508/segments/20090511094102

2009-05-11 09:49:23,984 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 09:49:58,046 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 09:49:58,234 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 09:49:58,265 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 09:49:58,859 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 09:49:58,906 INFO  MapTask - numReduceTasks: 1

2009-05-11 09:49:58,906 INFO  MapTask - io.sort.mb = 100

2009-05-11 09:49:59,015 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 09:49:59,015 INFO  MapTask - record buffer = 262144/327680

2009-05-11 09:49:59,140 INFO  Fetcher - Fetcher: threads: 5

2009-05-11 09:49:59,140 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

2009-05-11 09:49:59,250 INFO  Fetcher - QueueFeeder finished: total 1 records.

省略插件加载日志….

2009-05-11 09:49:59,312 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 09:49:59,328 INFO  Configuration - found resource parse-plugins.xml at file:/D:/work/workspace/nutch_crawl/bin/parse-plugins.xml

2009-05-11 09:49:59,359 INFO  Fetcher - fetching http://www.163.com/

2009-05-11 09:49:59,375 INFO  Fetcher - -finishing thread FetcherThread, activeThreads=4

2009-05-11 09:49:59,375 INFO  Fetcher - -finishing thread FetcherThread, activeThreads=3

2009-05-11 09:49:59,375 INFO  Fetcher - -finishing thread FetcherThread, activeThreads=2

2009-05-11 09:49:59,375 INFO  Fetcher - -finishing thread FetcherThread, activeThreads=1

2009-05-11 09:49:59,421 INFO  Http - http.proxy.host = null

2009-05-11 09:49:59,421 INFO  Http - http.proxy.port = 8080

2009-05-11 09:49:59,421 INFO  Http - http.timeout = 10000

2009-05-11 09:49:59,421 INFO  Http - http.content.limit = 65536

2009-05-11 09:49:59,421 INFO  Http - http.agent = nutch/Nutch-1.0 (chinahui; http://www.163.com; [email protected])

2009-05-11 09:49:59,421 INFO  Http - protocol.plugin.check.blocking = false

2009-05-11 09:49:59,421 INFO  Http - protocol.plugin.check.robots = false

2009-05-11 09:50:00,109 INFO  Configuration - found resource tika-mimetypes.xml at file:/D:/work/workspace/nutch_crawl/bin/tika-mimetypes.xml

2009-05-11 09:50:00,156 WARN  ParserFactory - ParserFactory:Plugin: org.apache.nutch.parse.html.HtmlParser mapped to contentType application/xhtml+xml via parse-plugins.xml, but its plugin.xml file does not claim to support contentType: application/xhtml+xml

2009-05-11 09:50:00,375 INFO  Fetcher - -activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0

2009-05-11 09:50:00,671 INFO  SignatureFactory - Using Signature impl: org.apache.nutch.crawl.MD5Signature

2009-05-11 09:50:00,687 INFO  Fetcher - -finishing thread FetcherThread, activeThreads=0

2009-05-11 09:50:01,375 INFO  Fetcher - -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0

2009-05-11 09:50:01,375 INFO  Fetcher - -activeThreads=0

2009-05-11 09:50:01,375 INFO  MapTask - Starting flush of map output

2009-05-11 09:50:01,578 INFO  MapTask - Finished spill 0

2009-05-11 09:50:01,578 INFO  TaskRunner - Task:attempt_local_0005_m_000000_0 is done. And is in the process of commiting

2009-05-11 09:50:01,578 INFO  LocalJobRunner - 0 threads, 1 pages, 0 errors, 0.5 pages/s, 256 kb/s,

2009-05-11 09:50:01,578 INFO  TaskRunner - Task 'attempt_local_0005_m_000000_0' done.

2009-05-11 09:50:01,593 INFO  LocalJobRunner -

2009-05-11 09:50:01,593 INFO  Merger - Merging 1 sorted segments

2009-05-11 09:50:01,593 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 72558 bytes

2009-05-11 09:50:01,593 INFO  LocalJobRunner -

2009-05-11 09:50:01,671 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:01,734 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:01,765 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:01,765 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 09:50:01,921 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 09:50:01,984 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:02,015 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:02,062 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:02,093 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:02,125 INFO  CodecPool - Got brand-new compressor

2009-05-11 09:50:02,140 WARN  RegexURLNormalizer - can't find rules for scope 'outlink', using default

2009-05-11 09:50:02,171 INFO  TaskRunner - Task:attempt_local_0005_r_000000_0 is done. And is in the process of commiting

2009-05-11 09:50:02,171 INFO  LocalJobRunner - reduce > reduce

2009-05-11 09:50:02,187 INFO  TaskRunner - Task 'attempt_local_0005_r_000000_0' done.

2009-05-11 09:50:44,062 INFO  JobClient - Running job: job_local_0005

2009-05-11 09:51:31,328 INFO  JobClient - Job complete: job_local_0005

2009-05-11 09:51:32,984 INFO  JobClient - Counters: 11

2009-05-11 09:51:33,000 INFO  JobClient -   File Systems

2009-05-11 09:51:33,000 INFO  JobClient -     Local bytes read=336424

2009-05-11 09:51:33,000 INFO  JobClient -     Local bytes written=700394

2009-05-11 09:51:33,000 INFO  JobClient -   Map-Reduce Framework

2009-05-11 09:51:33,000 INFO  JobClient -     Reduce input groups=1

2009-05-11 09:51:33,000 INFO  JobClient -     Combine output records=0

2009-05-11 09:51:33,000 INFO  JobClient -     Map input records=1

2009-05-11 09:51:33,000 INFO  JobClient -     Reduce output records=3

2009-05-11 09:51:33,000 INFO  JobClient -     Map output bytes=72545

2009-05-11 09:51:33,000 INFO  JobClient -     Map input bytes=78

2009-05-11 09:51:33,000 INFO  JobClient -     Combine input records=0

2009-05-11 09:51:33,000 INFO  JobClient -     Map output records=3

2009-05-11 09:51:33,000 INFO  JobClient -     Reduce input records=3

2009-05-11 09:51:47,750 INFO  Fetcher - Fetcher: done

1.3.4   parse方法

描述：解析下载页面内容

1.3.5   update方法

描述：添加子链接到爬取数据库

2009-05-11 10:04:20,890 INFO  CrawlDb - CrawlDb update: starting

2009-05-11 10:04:22,500 INFO  CrawlDb - CrawlDb update: db: 20090508/crawldb

2009-05-11 10:05:53,593 INFO  CrawlDb - CrawlDb update: segments: [20090508/segments/20090511094102]

2009-05-11 10:06:06,031 INFO  CrawlDb - CrawlDb update: additions allowed: true

2009-05-11 10:06:07,296 INFO  CrawlDb - CrawlDb update: URL normalizing: true

2009-05-11 10:06:09,031 INFO  CrawlDb - CrawlDb update: URL filtering: true

2009-05-11 10:07:05,125 INFO  CrawlDb - CrawlDb update: Merging segment data into db.

2009-05-11 10:08:11,031 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:09:00,187 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:09:00,375 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:10:03,531 INFO  FileInputFormat - Total input paths to process : 3

2009-05-11 10:16:25,125 INFO  FileInputFormat - Total input paths to process : 3

2009-05-11 10:16:25,203 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:16:25,203 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:16:25,343 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:16:25,343 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:16:25,343 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:25,750 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 10:16:25,796 WARN  RegexURLNormalizer - can't find rules for scope 'crawldb', using default

2009-05-11 10:16:25,796 INFO  MapTask - Starting flush of map output

2009-05-11 10:16:25,984 INFO  MapTask - Finished spill 0

2009-05-11 10:16:26,000 INFO  TaskRunner - Task:attempt_local_0006_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:16:26,000 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/part-00000/data:0+143

2009-05-11 10:16:26,000 INFO  TaskRunner - Task 'attempt_local_0006_m_000000_0' done.

2009-05-11 10:16:26,031 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:16:26,031 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:16:26,140 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:16:26,140 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:16:26,156 INFO  CodecPool - Got brand-new decompressor

2009-05-11 10:16:26,171 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:26,687 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 10:16:26,718 WARN  RegexURLNormalizer - can't find rules for scope 'crawldb', using default

2009-05-11 10:16:26,734 INFO  MapTask - Starting flush of map output

2009-05-11 10:16:26,750 INFO  MapTask - Finished spill 0

2009-05-11 10:16:26,750 INFO  TaskRunner - Task:attempt_local_0006_m_000002_0 is done. And is in the process of commiting

2009-05-11 10:16:26,750 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/crawl_parse/part-00000:0+4026

2009-05-11 10:16:26,750 INFO  TaskRunner - Task 'attempt_local_0006_m_000002_0' done.

2009-05-11 10:16:26,781 INFO  LocalJobRunner -

2009-05-11 10:16:26,781 INFO  Merger - Merging 3 sorted segments

2009-05-11 10:16:26,781 INFO  Merger - Down to the last merge-pass, with 3 segments left of total size: 3706 bytes

2009-05-11 10:16:26,781 INFO  LocalJobRunner -

2009-05-11 10:16:26,875 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:27,031 INFO  FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule

2009-05-11 10:16:27,031 INFO  AbstractFetchSchedule - defaultInterval=2592000

2009-05-11 10:16:27,031 INFO  AbstractFetchSchedule - maxInterval=7776000

2009-05-11 10:16:27,046 INFO  TaskRunner - Task:attempt_local_0006_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:16:27,046 INFO  LocalJobRunner -

2009-05-11 10:16:27,046 INFO  TaskRunner - Task attempt_local_0006_r_000000_0 is allowed to commit now

2009-05-11 10:16:27,062 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0006_r_000000_0' to file:/D:/work/workspace/nutch_crawl/20090508/crawldb/132216774

2009-05-11 10:16:27,062 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:16:27,062 INFO  TaskRunner - Task 'attempt_local_0006_r_000000_0' done.

2009-05-11 10:17:43,984 INFO  JobClient - Running job: job_local_0006

2009-05-11 10:18:33,671 INFO  JobClient - Job complete: job_local_0006

2009-05-11 10:18:35,906 INFO  JobClient - Counters: 11

2009-05-11 10:18:35,906 INFO  JobClient -   File Systems

2009-05-11 10:18:35,906 INFO  JobClient -     Local bytes read=936164

2009-05-11 10:18:35,906 INFO  JobClient -     Local bytes written=1678861

2009-05-11 10:18:35,906 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce input groups=57

2009-05-11 10:18:35,906 INFO  JobClient -     Combine output records=0

2009-05-11 10:18:35,906 INFO  JobClient -     Map input records=63

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce output records=57

2009-05-11 10:18:35,906 INFO  JobClient -     Map output bytes=3574

2009-05-11 10:18:35,906 INFO  JobClient -     Map input bytes=4079

2009-05-11 10:18:35,906 INFO  JobClient -     Combine input records=0

2009-05-11 10:18:35,906 INFO  JobClient -     Map output records=63

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce input records=63

2009-05-11 10:19:48,078 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:22:51,437 INFO  CrawlDb - CrawlDb update: done

1.3.6   invert方法

描述：分析链接关系,生成反向链接

执行完后生成linkdb目录.

2009-05-11 10:04:20,890 INFO  CrawlDb - CrawlDb update: starting

2009-05-11 10:04:22,500 INFO  CrawlDb - CrawlDb update: db: 20090508/crawldb

2009-05-11 10:05:53,593 INFO  CrawlDb - CrawlDb update: segments: [20090508/segments/20090511094102]

2009-05-11 10:06:06,031 INFO  CrawlDb - CrawlDb update: additions allowed: true

2009-05-11 10:06:07,296 INFO  CrawlDb - CrawlDb update: URL normalizing: true

2009-05-11 10:06:09,031 INFO  CrawlDb - CrawlDb update: URL filtering: true

2009-05-11 10:07:05,125 INFO  CrawlDb - CrawlDb update: Merging segment data into db.

2009-05-11 10:08:11,031 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:09:00,187 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:09:00,375 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:10:03,531 INFO  FileInputFormat - Total input paths to process : 3

2009-05-11 10:16:25,125 INFO  FileInputFormat - Total input paths to process : 3

2009-05-11 10:16:25,203 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:16:25,203 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:16:25,343 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:16:25,343 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:16:25,343 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:25,750 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 10:16:25,796 WARN  RegexURLNormalizer - can't find rules for scope 'crawldb', using default

2009-05-11 10:16:25,796 INFO  MapTask - Starting flush of map output

2009-05-11 10:16:25,984 INFO  MapTask - Finished spill 0

2009-05-11 10:16:26,000 INFO  TaskRunner - Task:attempt_local_0006_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:16:26,000 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/part-00000/data:0+143

2009-05-11 10:16:26,000 INFO  TaskRunner - Task 'attempt_local_0006_m_000000_0' done.

2009-05-11 10:16:26,031 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:16:26,031 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:16:26,140 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:16:26,140 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:16:26,156 INFO  CodecPool - Got brand-new decompressor

2009-05-11 10:16:26,171 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:26,343 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 10:16:26,359 WARN  RegexURLNormalizer - can't find rules for scope 'crawldb', using default

2009-05-11 10:16:26,359 INFO  MapTask - Starting flush of map output

2009-05-11 10:16:26,359 INFO  MapTask - Finished spill 0

2009-05-11 10:16:26,375 INFO  TaskRunner - Task:attempt_local_0006_m_000001_0 is done. And is in the process of commiting

2009-05-11 10:16:26,375 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/crawl_fetch/part-00000/data:0+254

2009-05-11 10:16:26,375 INFO  TaskRunner - Task 'attempt_local_0006_m_000001_0' done.

2009-05-11 10:16:26,406 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:16:26,406 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:16:26,515 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:16:26,515 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:16:26,531 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:26,687 INFO  Configuration - found resource crawl-urlfilter.txt at file:/D:/work/workspace/nutch_crawl/bin/crawl-urlfilter.txt

2009-05-11 10:16:26,718 WARN  RegexURLNormalizer - can't find rules for scope 'crawldb', using default

2009-05-11 10:16:26,734 INFO  MapTask - Starting flush of map output

2009-05-11 10:16:26,750 INFO  MapTask - Finished spill 0

2009-05-11 10:16:26,750 INFO  TaskRunner - Task:attempt_local_0006_m_000002_0 is done. And is in the process of commiting

2009-05-11 10:16:26,750 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/crawl_parse/part-00000:0+4026

2009-05-11 10:16:26,750 INFO  TaskRunner - Task 'attempt_local_0006_m_000002_0' done.

2009-05-11 10:16:26,781 INFO  LocalJobRunner -

2009-05-11 10:16:26,781 INFO  Merger - Merging 3 sorted segments

2009-05-11 10:16:26,781 INFO  Merger - Down to the last merge-pass, with 3 segments left of total size: 3706 bytes

2009-05-11 10:16:26,781 INFO  LocalJobRunner -

2009-05-11 10:16:26,875 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:16:27,031 INFO  FetchScheduleFactory - Using FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule

2009-05-11 10:16:27,031 INFO  AbstractFetchSchedule - defaultInterval=2592000

2009-05-11 10:16:27,031 INFO  AbstractFetchSchedule - maxInterval=7776000

2009-05-11 10:16:27,046 INFO  TaskRunner - Task:attempt_local_0006_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:16:27,046 INFO  LocalJobRunner -

2009-05-11 10:16:27,046 INFO  TaskRunner - Task attempt_local_0006_r_000000_0 is allowed to commit now

2009-05-11 10:16:27,062 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0006_r_000000_0' to file:/D:/work/workspace/nutch_crawl/20090508/crawldb/132216774

2009-05-11 10:16:27,062 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:16:27,062 INFO  TaskRunner - Task 'attempt_local_0006_r_000000_0' done.

2009-05-11 10:17:43,984 INFO  JobClient - Running job: job_local_0006

2009-05-11 10:18:33,671 INFO  JobClient - Job complete: job_local_0006

2009-05-11 10:18:35,906 INFO  JobClient - Counters: 11

2009-05-11 10:18:35,906 INFO  JobClient -   File Systems

2009-05-11 10:18:35,906 INFO  JobClient -     Local bytes read=936164

2009-05-11 10:18:35,906 INFO  JobClient -     Local bytes written=1678861

2009-05-11 10:18:35,906 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce input groups=57

2009-05-11 10:18:35,906 INFO  JobClient -     Combine output records=0

2009-05-11 10:18:35,906 INFO  JobClient -     Map input records=63

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce output records=57

2009-05-11 10:18:35,906 INFO  JobClient -     Map output bytes=3574

2009-05-11 10:18:35,906 INFO  JobClient -     Map input bytes=4079

2009-05-11 10:18:35,906 INFO  JobClient -     Combine input records=0

2009-05-11 10:18:35,906 INFO  JobClient -     Map output records=63

2009-05-11 10:18:35,906 INFO  JobClient -     Reduce input records=63

2009-05-11 10:19:48,078 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:22:51,437 INFO  CrawlDb - CrawlDb update: done

2009-05-11 10:26:31,250 INFO  LinkDb - LinkDb: starting

2009-05-11 10:26:31,250 INFO  LinkDb - LinkDb: linkdb: 20090508/linkdb

2009-05-11 10:26:31,250 INFO  LinkDb - LinkDb: URL normalize: true

2009-05-11 10:26:31,250 INFO  LinkDb - LinkDb: URL filter: true

2009-05-11 10:26:31,281 INFO  LinkDb - LinkDb: adding segment: file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102

2009-05-11 10:26:31,281 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:26:31,296 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:26:31,453 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:26:31,484 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:26:32,078 INFO  JobClient - Running job: job_local_0007

2009-05-11 10:26:32,078 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:26:32,125 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:26:32,125 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:26:32,234 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:26:32,234 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:26:32,250 INFO  MapTask - Starting flush of map output

2009-05-11 10:26:32,437 INFO  MapTask - Finished spill 0

2009-05-11 10:26:32,453 INFO  TaskRunner - Task:attempt_local_0007_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:26:32,453 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/parse_data/part-00000/data:0+1382

2009-05-11 10:26:32,453 INFO  TaskRunner - Task 'attempt_local_0007_m_000000_0' done.

2009-05-11 10:26:32,468 INFO  LocalJobRunner -

2009-05-11 10:26:32,468 INFO  Merger - Merging 1 sorted segments

2009-05-11 10:26:32,468 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 3264 bytes

2009-05-11 10:26:32,468 INFO  LocalJobRunner -

2009-05-11 10:26:32,562 INFO  TaskRunner - Task:attempt_local_0007_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:26:32,562 INFO  LocalJobRunner -

2009-05-11 10:26:32,562 INFO  TaskRunner - Task attempt_local_0007_r_000000_0 is allowed to commit now

2009-05-11 10:26:32,578 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0007_r_000000_0' to file:/D:/work/workspace/nutch_crawl/linkdb-1900012851

2009-05-11 10:26:32,578 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:26:32,578 INFO  TaskRunner - Task 'attempt_local_0007_r_000000_0' done.

2009-05-11 10:26:33,078 INFO  JobClient - Job complete: job_local_0007

2009-05-11 10:26:33,078 INFO  JobClient - Counters: 11

2009-05-11 10:26:33,078 INFO  JobClient -   File Systems

2009-05-11 10:26:33,078 INFO  JobClient -     Local bytes read=535968

2009-05-11 10:26:33,078 INFO  JobClient -     Local bytes written=965231

2009-05-11 10:26:33,078 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:26:33,078 INFO  JobClient -     Reduce input groups=56

2009-05-11 10:26:33,078 INFO  JobClient -     Combine output records=56

2009-05-11 10:26:33,078 INFO  JobClient -     Map input records=1

2009-05-11 10:26:33,078 INFO  JobClient -     Reduce output records=56

2009-05-11 10:26:33,078 INFO  JobClient -     Map output bytes=3384

2009-05-11 10:26:33,078 INFO  JobClient -     Map input bytes=1254

2009-05-11 10:26:33,078 INFO  JobClient -     Combine input records=60

2009-05-11 10:26:33,078 INFO  JobClient -     Map output records=60

2009-05-11 10:26:33,078 INFO  JobClient -     Reduce input records=56

2009-05-11 10:26:33,078 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:26:33,125 INFO  LinkDb - LinkDb: done

1.3.7   index方法

描述：创建页面内容索引

生成indexes目录.

Nutch1.0源码分析-----抓取部分_第1张图片

2009-05-11 10:31:22,250 INFO  Indexer - Indexer: starting

2009-05-11 10:31:45,078 INFO  IndexerMapReduce - IndexerMapReduce: crawldb: 20090508/crawldb

2009-05-11 10:31:45,078 INFO  IndexerMapReduce - IndexerMapReduce: linkdb: 20090508/linkdb

2009-05-11 10:31:45,078 INFO  IndexerMapReduce - IndexerMapReduces: adding segment: file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102

2009-05-11 10:32:30,359 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:32:34,109 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:32:34,296 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:32:34,421 INFO  FileInputFormat - Total input paths to process : 6

2009-05-11 10:32:35,078 INFO  FileInputFormat - Total input paths to process : 6

2009-05-11 10:32:35,140 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:32:35,140 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:32:35,250 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:32:35,250 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:32:35,265 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:35,937 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:35,937 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:35,953 INFO  MapTask - Starting flush of map output

2009-05-11 10:32:35,968 INFO  MapTask - Finished spill 0

2009-05-11 10:32:35,968 INFO  TaskRunner - Task:attempt_local_0008_m_000001_0 is done. And is in the process of commiting

2009-05-11 10:32:35,968 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/crawl_parse/part-00000:0+4026

2009-05-11 10:32:35,968 INFO  TaskRunner - Task 'attempt_local_0008_m_000001_0' done.

2009-05-11 10:32:36,000 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:32:36,000 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:32:36,125 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:32:36,125 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:32:36,125 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:36,281 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:36,281 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:36,281 INFO  MapTask - Starting flush of map output

2009-05-11 10:32:36,296 INFO  MapTask - Finished spill 0

2009-05-11 10:32:36,312 INFO  TaskRunner - Task:attempt_local_0008_m_000002_0 is done. And is in the process of commiting

2009-05-11 10:32:36,312 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/parse_data/part-00000/data:0+1382

2009-05-11 10:32:36,312 INFO  TaskRunner - Task 'attempt_local_0008_m_000002_0' done.

2009-05-11 10:32:36,343 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:32:36,343 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:32:36,453 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:32:36,453 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:32:36,453 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:36,609 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:36,609 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:36,625 INFO  MapTask - Starting flush of map output

2009-05-11 10:32:36,625 INFO  MapTask - Finished spill 0

2009-05-11 10:32:36,640 INFO  TaskRunner - Task:attempt_local_0008_m_000003_0 is done. And is in the process of commiting

2009-05-11 10:32:36,640 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/parse_text/part-00000/data:0+738

2009-05-11 10:32:36,640 INFO  TaskRunner - Task 'attempt_local_0008_m_000003_0' done.

2009-05-11 10:32:36,671 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:32:36,671 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:32:36,781 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:32:36,781 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:32:36,796 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:36,937 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:36,953 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:36,953 INFO  MapTask - Starting flush of map output

2009-05-11 10:32:36,968 INFO  MapTask - Finished spill 0

2009-05-11 10:32:36,968 INFO  TaskRunner - Task:attempt_local_0008_m_000004_0 is done. And is in the process of commiting

2009-05-11 10:32:36,968 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/part-00000/data:0+3772

2009-05-11 10:32:36,968 INFO  TaskRunner - Task 'attempt_local_0008_m_000004_0' done.

2009-05-11 10:32:37,000 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:32:37,000 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:32:37,109 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:32:37,109 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:32:37,125 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:37,281 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:37,281 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:37,296 INFO  MapTask - Starting flush of map output

2009-05-11 10:32:37,296 INFO  MapTask - Finished spill 0

2009-05-11 10:32:37,312 INFO  TaskRunner - Task:attempt_local_0008_m_000005_0 is done. And is in the process of commiting

2009-05-11 10:32:37,312 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/linkdb/current/part-00000/data:0+4215

2009-05-11 10:32:37,312 INFO  TaskRunner - Task 'attempt_local_0008_m_000005_0' done.

2009-05-11 10:32:37,343 INFO  LocalJobRunner -

2009-05-11 10:32:37,359 INFO  Merger - Merging 6 sorted segments

2009-05-11 10:32:37,359 INFO  Merger - Down to the last merge-pass, with 6 segments left of total size: 13876 bytes

2009-05-11 10:32:37,359 INFO  LocalJobRunner -

2009-05-11 10:32:37,359 INFO  PluginRepository - Plugins: looking in: D:/work/workspace/nutch_crawl/bin/plugins

省略插件加载日志…

2009-05-11 10:32:37,515 INFO  IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter

2009-05-11 10:32:37,515 INFO  IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter

2009-05-11 10:32:37,546 INFO  Configuration - found resource common-terms.utf8 at file:/D:/work/workspace/nutch_crawl/bin/common-terms.utf8

2009-05-11 10:32:38,500 INFO  TaskRunner - Task:attempt_local_0008_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:32:38,500 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:32:38,500 INFO  TaskRunner - Task 'attempt_local_0008_r_000000_0' done.

2009-05-11 10:33:19,703 INFO  JobClient - Running job: job_local_0008

2009-05-11 10:33:50,156 INFO  JobClient - Job complete: job_local_0008

2009-05-11 10:33:52,562 INFO  JobClient - Counters: 11

2009-05-11 10:33:52,562 INFO  JobClient -   File Systems

2009-05-11 10:33:52,562 INFO  JobClient -     Local bytes read=2150441

2009-05-11 10:33:52,562 INFO  JobClient -     Local bytes written=3845733

2009-05-11 10:33:52,562 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:33:52,562 INFO  JobClient -     Reduce input groups=58

2009-05-11 10:33:52,562 INFO  JobClient -     Combine output records=0

2009-05-11 10:33:52,562 INFO  JobClient -     Map input records=177

2009-05-11 10:33:52,562 INFO  JobClient -     Reduce output records=1

2009-05-11 10:33:52,562 INFO  JobClient -     Map output bytes=13506

2009-05-11 10:33:52,562 INFO  JobClient -     Map input bytes=13661

2009-05-11 10:33:52,562 INFO  JobClient -     Combine input records=0

2009-05-11 10:33:52,562 INFO  JobClient -     Map output records=177

2009-05-11 10:33:52,562 INFO  JobClient -     Reduce input records=177

2009-05-11 10:33:57,656 INFO  Indexer - Indexer: done

1.3.8   dedup方法

描述：删除重复数据

2009-05-11 10:38:53,671 INFO  DeleteDuplicates - Dedup: starting

2009-05-11 10:39:32,890 INFO  DeleteDuplicates - Dedup: adding indexes in: 20090508/indexes

2009-05-11 10:39:57,265 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:40:09,015 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:40:09,218 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:40:51,890 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:42:56,203 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:42:56,265 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:42:56,265 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:42:56,390 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:42:56,390 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:42:56,515 INFO  MapTask - Starting flush of map output

2009-05-11 10:42:56,718 INFO  MapTask - Finished spill 0

2009-05-11 10:42:56,718 INFO  TaskRunner - Task:attempt_local_0009_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:42:56,718 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/20090508/indexes/part-00000

2009-05-11 10:42:56,718 INFO  TaskRunner - Task 'attempt_local_0009_m_000000_0' done.

2009-05-11 10:42:56,734 INFO  LocalJobRunner -

2009-05-11 10:42:56,734 INFO  Merger - Merging 1 sorted segments

2009-05-11 10:42:56,734 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 141 bytes

2009-05-11 10:42:56,734 INFO  LocalJobRunner -

2009-05-11 10:42:56,781 INFO  TaskRunner - Task:attempt_local_0009_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:42:56,781 INFO  LocalJobRunner -

2009-05-11 10:42:56,781 INFO  TaskRunner - Task attempt_local_0009_r_000000_0 is allowed to commit now

2009-05-11 10:42:56,796 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0009_r_000000_0' to file:/D:/work/workspace/nutch_crawl/dedup-urls-1843604809

2009-05-11 10:42:56,796 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:42:56,796 INFO  TaskRunner - Task 'attempt_local_0009_r_000000_0' done.

2009-05-11 10:43:06,515 INFO  JobClient - Running job: job_local_0009

2009-05-11 10:43:14,500 INFO  JobClient - Job complete: job_local_0009

2009-05-11 10:43:16,296 INFO  JobClient - Counters: 11

2009-05-11 10:43:16,296 INFO  JobClient -   File Systems

2009-05-11 10:43:16,296 INFO  JobClient -     Local bytes read=710951

2009-05-11 10:43:16,296 INFO  JobClient -     Local bytes written=1220879

2009-05-11 10:43:16,296 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:43:16,296 INFO  JobClient -     Reduce input groups=1

2009-05-11 10:43:16,296 INFO  JobClient -     Combine output records=0

2009-05-11 10:43:16,296 INFO  JobClient -     Map input records=1

2009-05-11 10:43:16,296 INFO  JobClient -     Reduce output records=1

2009-05-11 10:43:16,296 INFO  JobClient -     Map output bytes=137

2009-05-11 10:43:16,296 INFO  JobClient -     Map input bytes=2147483647

2009-05-11 10:43:16,296 INFO  JobClient -     Combine input records=0

2009-05-11 10:43:16,296 INFO  JobClient -     Map output records=1

2009-05-11 10:43:16,296 INFO  JobClient -     Reduce input records=1

2009-05-11 10:44:37,734 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:44:45,953 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:44:46,140 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:44:48,781 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:45:46,546 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:45:46,609 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:45:46,609 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:45:46,718 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:45:46,718 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:45:46,734 INFO  MapTask - Starting flush of map output

2009-05-11 10:45:46,953 INFO  MapTask - Finished spill 0

2009-05-11 10:45:46,953 INFO  TaskRunner - Task:attempt_local_0010_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:45:46,953 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/dedup-urls-1843604809/part-00000:0+247

2009-05-11 10:45:46,953 INFO  TaskRunner - Task 'attempt_local_0010_m_000000_0' done.

2009-05-11 10:45:46,968 INFO  LocalJobRunner -

2009-05-11 10:45:46,968 INFO  Merger - Merging 1 sorted segments

2009-05-11 10:45:46,968 INFO  Merger - Down to the last merge-pass, with 1 segments left of total size: 137 bytes

2009-05-11 10:45:46,968 INFO  LocalJobRunner -

2009-05-11 10:45:47,015 INFO  TaskRunner - Task:attempt_local_0010_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:45:47,015 INFO  LocalJobRunner -

2009-05-11 10:45:47,015 INFO  TaskRunner - Task attempt_local_0010_r_000000_0 is allowed to commit now

2009-05-11 10:45:47,015 INFO  FileOutputCommitter - Saved output of task 'attempt_local_0010_r_000000_0' to file:/D:/work/workspace/nutch_crawl/dedup-hash-291931517

2009-05-11 10:45:47,015 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:45:47,015 INFO  TaskRunner - Task 'attempt_local_0010_r_000000_0' done.

2009-05-11 10:45:52,187 INFO  JobClient - Running job: job_local_0010

2009-05-11 10:46:03,984 INFO  JobClient - Job complete: job_local_0010

2009-05-11 10:46:06,359 INFO  JobClient - Counters: 11

2009-05-11 10:46:06,359 INFO  JobClient -   File Systems

2009-05-11 10:46:06,359 INFO  JobClient -     Local bytes read=764171

2009-05-11 10:46:06,359 INFO  JobClient -     Local bytes written=1327019

2009-05-11 10:46:06,359 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:46:06,359 INFO  JobClient -     Reduce input groups=1

2009-05-11 10:46:06,359 INFO  JobClient -     Combine output records=0

2009-05-11 10:46:06,359 INFO  JobClient -     Map input records=1

2009-05-11 10:46:06,359 INFO  JobClient -     Reduce output records=0

2009-05-11 10:46:06,359 INFO  JobClient -     Map output bytes=133

2009-05-11 10:46:06,359 INFO  JobClient -     Map input bytes=141

2009-05-11 10:46:06,359 INFO  JobClient -     Combine input records=0

2009-05-11 10:46:06,359 INFO  JobClient -     Map output records=1

2009-05-11 10:46:06,359 INFO  JobClient -     Reduce input records=1

2009-05-11 10:47:19,953 INFO  JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

2009-05-11 10:47:19,953 WARN  JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

2009-05-11 10:47:20,140 WARN  JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

2009-05-11 10:47:20,156 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:47:20,765 INFO  JobClient - Running job: job_local_0011

2009-05-11 10:47:20,765 INFO  FileInputFormat - Total input paths to process : 1

2009-05-11 10:47:20,796 INFO  MapTask - numReduceTasks: 1

2009-05-11 10:47:20,796 INFO  MapTask - io.sort.mb = 100

2009-05-11 10:47:20,921 INFO  MapTask - data buffer = 79691776/99614720

2009-05-11 10:47:20,921 INFO  MapTask - record buffer = 262144/327680

2009-05-11 10:47:20,937 INFO  MapTask - Starting flush of map output

2009-05-11 10:47:21,140 INFO  MapTask - Index: (0, 2, 6)

2009-05-11 10:47:21,140 INFO  TaskRunner - Task:attempt_local_0011_m_000000_0 is done. And is in the process of commiting

2009-05-11 10:47:21,140 INFO  LocalJobRunner - file:/D:/work/workspace/nutch_crawl/dedup-hash-291931517/part-00000:0+103

2009-05-11 10:47:21,140 INFO  TaskRunner - Task 'attempt_local_0011_m_000000_0' done.

2009-05-11 10:47:21,156 INFO  LocalJobRunner -

2009-05-11 10:47:21,156 INFO  Merger - Merging 1 sorted segments

2009-05-11 10:47:21,156 INFO  Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes

2009-05-11 10:47:21,156 INFO  LocalJobRunner -

2009-05-11 10:47:21,171 INFO  TaskRunner - Task:attempt_local_0011_r_000000_0 is done. And is in the process of commiting

2009-05-11 10:47:21,171 INFO  LocalJobRunner - reduce > reduce

2009-05-11 10:47:21,171 INFO  TaskRunner - Task 'attempt_local_0011_r_000000_0' done.

2009-05-11 10:47:21,765 INFO  JobClient - Job complete: job_local_0011

2009-05-11 10:47:21,765 INFO  JobClient - Counters: 11

2009-05-11 10:47:21,765 INFO  JobClient -   File Systems

2009-05-11 10:47:21,765 INFO  JobClient -     Local bytes read=816128

2009-05-11 10:47:21,765 INFO  JobClient -     Local bytes written=1430954

2009-05-11 10:47:21,765 INFO  JobClient -   Map-Reduce Framework

2009-05-11 10:47:21,765 INFO  JobClient -     Reduce input groups=0

2009-05-11 10:47:21,765 INFO  JobClient -     Combine output records=0

2009-05-11 10:47:21,765 INFO  JobClient -     Map input records=0

2009-05-11 10:47:21,765 INFO  JobClient -     Reduce output records=0

2009-05-11 10:47:21,765 INFO  JobClient -     Map output bytes=0

2009-05-11 10:47:21,765 INFO  JobClient -     Map input bytes=0

2009-05-11 10:47:21,765 INFO  JobClient -     Combine input records=0

2009-05-11 10:47:21,765 INFO  JobClient -     Map output records=0

2009-05-11 10:47:21,765 INFO  JobClient -     Reduce input records=0

2009-05-11 10:47:44,031 INFO  DeleteDuplicates - Dedup: done

1.3.9   merge方法

描述：合并索引文件

首先在tmp/hadoop-Administrator/mapred/local/crawl生成一个临时文件夹20090511094057,用indexes里的数据生成索引添加到20090511094057下的merge-output目录,

fs.completeLocalOutput方法把临时目录的索引写到新生成的index目录下.


2009-05-11 10:53:56,156 INFO  IndexMerger - merging indexes to: 20090508/index

2009-05-11 10:58:50,906 INFO  IndexMerger - Adding file:/D:/work/workspace/nutch_crawl/20090508/indexes/part-00000

2009-05-11 11:04:36,562 INFO  IndexMerger - done merging

 

你可能感兴趣的:(nutch)

  • Java：爬虫框架 dingcho Javajava爬虫
    一、ApacheNutch2【参考地址】Nutch是一个开源Java实现的搜索引擎。它提供了我们运行自己的搜索引擎所需的全部工具。包括全文搜索和Web爬虫。Nutch致力于让每个人能很容易,同时花费很少就可以配置世界一流的Web搜索引擎.为了完成这一宏伟的目标,Nutch必须能够做到:每个月取几十亿网页为这些网页维护一个索引对索引文件进行每秒上千次的搜索提供高质量的搜索结果简单来说Nutch支持分
  • Python爬虫实战 weixin_34007879 爬虫jsonjava
    引言网络爬虫是抓取互联网信息的利器，成熟的开源爬虫框架主要集中于两种语言Java和Python。主流的开源爬虫框架包括：1.分布式爬虫框架：Nutch2.Java单机爬虫框架：Crawler4j,WebMagic,WebCollector、Heritrix3.python单机爬虫框架：scrapy、pyspiderNutch是专为搜索引擎设计的的分布式开源框架，上手难度高，开发复杂，基本无法满足快
  • 深入浅出hdfs-hadoop基本介绍 大数据之家 hdfshadoop大数据
    一、Hadoop基本介绍hadoop最开始是起源于ApacheNutch项目，这个是由DougCutting开发的开源网络搜索引擎，这个项目刚开始的目标是为了更好的做搜索引擎，后来Google发表了三篇未来持续影响大数据领域的三架马车论文：GoogleFileSystem、BigTable、Mapreduce开始掀起来了大数据的浪潮，paper原文可以参考我的这篇文章CSDN。这三篇论文介绍了如何
  • Hadoop简介：开启大数据处理之门 乌龙饼干 hadoop大数据分布式
    随着信息技术的飞速发展，数据呈现爆炸式增长，传统的数据处理方式已无法满足日益增长的数据需求。在此背景下，Hadoop作为一种分布式系统基础架构，应运而生，为大数据处理打开了新的大门。一、Hadoop的起源与概念Hadoop最初由DougCutting创建，作为ApacheLucene的子项目Nutch的一部分。随着项目的不断发展，Hadoop逐渐独立出来，成为Apache软件基金会下的一个开源项目
  • 专为初学者设计：Nutch库Java下载器入门指南 亿牛云爬虫专家 java代理IP爬虫代理java开发语言Nutch下载器爬虫代理代理IP多线程
    概述:Nutch是一款开源的Java爬虫框架，用于抓取、解析、提取和存储网页数据。基于Hadoop的分布式系统，Nutch支持大规模网络爬取，并提供各种插件，包括链接分析、语言检测和内容过滤等功能。本文旨在介绍如何使用Nutch库编写简单的Java下载器，即能从指定URL下载网页内容的程序。目标是帮助初学者了解Nutch库的基本用法，并展示如何通过代理IP技术和多线程技术提升下载效率。假设读者已安
  • 在CentOS7上安装Hadoop分布式系统 栗子艾李子 hadooplinuxhdfs分布式
    项目背景：Hadoop原来是ApacheLucene下的一个子项目，它最初是从Nutch项目中分离出来的专门负责分布式存储以及分布式运算的项目。简单地说来，Hadoop是一个可以更容易开发和运行处理大规模数据的软件平台。Hadoop由分布式存储HDFS和分布式计算MapReduce两部分组成。HDFS是一个master/slave的结构，就通常的部署来说，在master上只运行一个Namenode
  • 大数据技术之Hadoop入门一 在远方的你等我
    1.从Hadoop框架讨论大数据生态名字起源该项目的创建者，DougCutting解释Hadoop的得名：“这个名字是我孩子给一个棕黄色的大象玩具命名的项目起源Hadoop由ApacheSoftwareFoundation公司于2005年秋天作为Lucene的子项目Nutch的一部分正式引入。它受到最先由GoogleLab开发的Map/Reduce和GoogleFileSystem(GFS)的启发
  • openpyxl3.0官方文档（14）—— 甜甜圈图 Sinchard
    甜甜圈图表与饼图类似，只是它们使用了一个环而不是一个圆，还可以绘制出若干系列的数据作为中心环。fromopenpyxlimportWorkbookfromopenpyxl.chartimport(DoughnutChart,Reference,Series,)fromopenpyxl.chart.seriesimportDataPointdata=[['Pie',2014,2015],['Plai
  • kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） weixin_34185320 运维操作系统系统架构
    李克华云计算高级群:292870151195907286交流：Hadoop、NoSQL、分布式、lucene、solr、nutchkafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转）问题导读：1.zookeeper在kafka的作用是什么？2.kafka中几乎不允许对消息进行“随机读写”的原因是什么？3.kafka集群consumer和producer状态信息是如何保存的？4.par
  • 大数据之 Hadoop 小裕哥略帅 大数据hadoopjava
    hadoop主要解决：海量数据的存储和海量数据的分析计算hadoop发展历史Google是hadoop的思想之源（Google在大数据方面的三篇论文）2006年3月，Map-reduce和NutchDistributedFileSystem(NDFS)分别被纳入到Hadoop项目，Hadoop正式诞生。MapReduce对海量数据处理map函数进行数据的提取、排序，实现mapper，四个形参（输入
  • 自己动手写搜索引擎系列【目录】 luyee2010 自己动手写搜索引擎自己动手写搜索引擎
    第1章遍历搜索引擎技术11.130分钟实现的搜索引擎11.1.1准备工作环境（10分钟）11.1.2编写代码（15分钟）31.1.3发布运行（5分钟）51.2Google神话91.3体验搜索引擎91.4搜索语法101.5你也可以做搜索引擎131.6搜索引擎基本技术141.6.1网络蜘蛛141.6.2全文索引结构141.6.3Lucene全文检索引擎151.6.4Nutch网络搜索软件161.6.5
  • ElasticSearch（ES）——概述/API 平平无奇小码农 笔记大数据elasticsearch数据库搜索引擎
    文章目录一、ElasticSearch基础1.1简介1.2使用场景1.3ES与其他数据存储进行比较1.4ES的特点1.5Lucene、Nutch、ElasticSearch关系二、基本概念ES概念和MySQL关系对比三、安装ES3.1解压、改名3.2修改配置文件3.3教学环境启动优化分发3.4修改hadoop163、hadoop164的节点名3.5单台启动测试，解决问题四、安装kibana4.1解
  • asp html5 ajax,ASP.NET AJAX Chart (HTML5) - RadControls for Web Forms | Telerik UI for ASP.NET AJAX weixin_39942191 asphtml5ajax
    AnyEssentialChartTypeQuicklyaddmeaningtodatawiththemostcommonlyusedASP.NETchartingtypes:PieorDonutcharttovisualizeeachpieceofdataaspartofawholeLineorAreatomonitortrendsBar,ColumnorRadartocomparesevera
  • 安装关系型数据库MySQL和大数据处理框架Hadoop weixin_30621919 数据库嵌入式大数据
    这个作业的要求来自于：https://edu.cnblogs.com/campus/gzcc/GZCC-16SE2/homework/3161。1.简述Hadoop平台的起源、发展历史与应用现状。列举发展过程中重要的事件、主要版本、主要厂商；国内外Hadoop应用的典型案例。（1）Hadoop的介绍：Hadoop最早起源于Nutch，Nutch的设计目标是构建一个大型的全网搜索引擎，包括网页抓取、
  • ChatGPT4 完成数据分析结构分析，动态饼图可视化 阿里数据专家 ChatGPT实战案例ChatGPT数据分析信息可视化数据挖掘excel人工智能AIGCchatgpt
    对于数据分析中的结构占比分析，以下几种图表类型是比较常见和合适的：1.**饼图（PieChart）**：饼图是一种表现部分与整体关系的图表，各部分占整体的比例在图中以圆形的切片形式体现。它适用于表示不同类别之间的比较，以及每个类别占总数的百分比。2.**环图（DoughnutChart）**：环图是饼图的变种，有一个空心中心。它也是显示类别之间占比关系的一种有效的方式。3.**堆叠柱状图/堆叠条形
  • hadoop yuanjianqiang_0925 hadoopspark
    hadoop主要解决：海量数据的存储和海量数据的分析计算hadoop发展历史Google是hadoop的思想之源（Google在大数据方面的三篇论文）2006年3月，Map-reduce和NutchDistributedFileSystem(NDFS)分别被纳入到Hadoop项目，Hadoop正式诞生。MapReduce对海量数据处理map函数进行数据的提取、排序，实现mapper，四个形参（
  • Ubuntu环境下Hadoop1.2.1, HBase0.94.25, nutch2.2.1各个配置文件一览 weixin_30491641 大数据javaruntime
    /×××××××××××××××××××××××××××××××××××××××××/Author：xxx0624HomePage：http://www.cnblogs.com/xxx0624//×××××××××××××××××××××××××××××××××××××××××/Hadoop伪分布式配置过程：Hadoop：1.2.1Hbase：0.94.25nutch：2.2.1Java：1.8.
  • ElasticSearch详细教程-基础加实战 工藤-新二 实时数仓大数据实时项目elasticsearch实时大数据spark
    文章目录第1章ElasticSearch基础1.1简介1.2使用场景1.3ES与其他数据存储进行比较1.4ElasticSearch的特点1.4.1天然分片，天然集群1.4.2天然索引1.5Lucene、Nutch、ElasticSearch关系第2章ElasticSearch的安装2.1上传安装包2.2将ES解压到/opt/module目录下2.3在/opt/module目录下对ES重命名2.4
  • jvm命令和可视化工具 调优 weixin_30834783 java操作系统开发工具
    李克华云计算高级群:292870151195907286交流：Hadoop、NoSQL、分布式、lucene、solr、nutch虚拟机：系统虚拟机程序虚拟机系统虚拟机有：VMWarevisureBox程序虚拟机：JVMJVM：1.类加载子系统（类加载器）2.方法区3.java堆4.直接内存5.java栈6.本地方法栈7.垃圾回收系统8.PC寄存器9.执行引擎堆：存储问题栈：程序运行方法去：辅助堆
  • linux服务器忘记ssh密码_【Linux】配置linux服务器之间ssh不用密码访问 weixin_40008033 linux服务器忘记ssh密码
    如果想在A这太机器上可以不需要密码就ssh到B、C两台机器上，可以采用如下的方法：(1)在A机器上：ssh-keygen-trsaGeneratingpublic/privatersakeypair.Enterfileinwhichtosavethekey(/nutch/home/.ssh/id_rsa):不输入任何东西，直接回车Enterpassphrase(emptyfornopassphra
  • Hadoop分布式文件系统 杀神lwz hadoop大数据分布式
    一、HadoopHadoop之父DougCuttingHadoop的发音[hædu:p]，Cutting儿子对玩具小象的昵称1、Hadoop发展简史2002年10月，DougCutting和MikeCafarella创建了开源网页爬虫项目Nutch。2003年10月，Google发表GoogleFileSystem论文。2004年7月，DougCutting和MikeCafarella在Nutch
  • java 爬虫框架nutch_网络爬虫（2）-- Java爬虫框架 鲍鱼王 java爬虫框架nutch
    NutchNutch属于分布式爬虫，爬虫使用分布式，主要是解决两个问题：1)海量URL管理；2)网速。如果要做搜索引擎，Nutch1.x是一个非常好的选择。Nutch1.x和solr或者es配合，就可以构成一套非常强大的搜索引擎，否则尽量不要选择Nutch作为爬虫。用Nutch进行爬虫的二次开发，爬虫的编写和调试所需的时间，往往是单机爬虫所需的十倍时间不止。HeritrixHeritrix是个“A
  • nutch爬取网站数据详细步骤 Echoooo_o
    环境：hadoop2.7.7+hbase0.98+nutch2.3+solr4.9大致步骤思想：hadoop提供底层数据存储hbase在其之上建立非关系型数据库nutch将爬的数据存到hbase上并建立索引到solr展示首先采用简单命令：#$1$2...$n表示命令后跟的第n个参数#存放待注入种子的路径SEEDDIR="$1"#存放爬取数据（URL状态信息、爬取数据、解析数据）文件夹的路径CRAW
  • nutch，hbase记录 feihuadao
    hbase表操作优化http://blog.pureisle.net/archives/1930.htmlHow-to:UseHBaseBulkLoading,andWhyhttp://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/nutch2.2分析http://blog.csdn.net/itufo/a
  • Hadoop 凤舞飘伶 Gohadoop
    Hadoop是Google的集群系统的开源实现，Google集群系统:GFS(GoogleFileSystem)、MapReduce、BigTable。Hadoop主要由HDFS(HadoopDistributedFileSystemHadoop分布式文件系统)、MapReduce和HBase组成Hadoop的初衷是为解决Nutch的海量数据爬取和存储的需要。Hadoop于2005年秋天作为Luc
  • Hadoop之父：Doug Cutting Mr_Elliot
    hadoop生活中，可能所有人都间接用过他的作品，他是Lucene、Nutch、Hadoop等项目的发起人。是他，把高深莫测的搜索技术形成产品，贡献给普罗大众；还是他，打造了目前在云计算和大数据领域里如日中天的Hadoop。他是某种意义上的盗火者，他就是DougCutting。DougCutting从实习生做起1985年，Cutting毕业于美国斯坦福大学。他并不是一开始就决心投身IT行业的，在大
  • Hadoop-2.6.5完整安装配置过程 syp_net 系统开发hadoopmapreduce搜索引擎
    记录Hadoop-2.6.5完整安装配置过程一、Hadoop是什么？二、Hadoop-2.6.5安装配置1.修改主机名2.下载并解压JDK3.配置环境变量4.修改Hadoop中5个主要配置文件5.启动Hadoop6.HadoopWeb端口测试三、总结一、Hadoop是什么？Hadoop系统最初的源头来自于ApacheLucene项目下的搜索引擎子项目Nutch，该项目的负责人是DougCuttin
  • Hadoop之HDFS简介 数新网络 hadoop大数据hdfs
    前言Hadoop是由Apache基金会开发的分布式系统基础框架，主要解决海量数据存储和海量数据分析问题。Hadoop起源于ApacheNutch项目，起始于2002年，在2006年被正式命名为Hadoop。Hadoop有3大核心组件，分别是HDFS、MapReduce和YARN，本次我们重点介绍HDFS。一、HDFS简介HDFS全称HadoopDistributedFileSystem，是一个分布
  • hadoop原理和细节 truezqx
    一、Hadoop概述Hadoop是Google的集群系统开源实现Google的集群系统：GFS、MapReduce、BigTableHadoop的集群系统：HDFS、MapReduce、HBaseHadoop设计的初衷是为了解决Nutch的海量数据存储和处理的需求，可以解决大数据场景下的数据存储和处理的问题。传统数据：GB、TB级别的数据、数据增长不快、主要为结构化的数据、统计和报表大数据：TB、
  • 听阿里P7工程师只分七步讲解HDFS搭建 Python大数据工程师
    前言HADOOP产生背景（1）HADOOP最早起源于Nutch。Nutch的设计目标是构建一个大型的全网搜索引擎，包括网页抓取、索引、查询等功能，但随着抓取网页数量的增加，遇到了严重的可扩展性问题——如何解决数十亿网页的存储和索引问题。（2）2003年、2004年谷歌发表的两篇论文为该问题提供了可行的解决方案。——分布式文件系统（GFS），可用于处理海量网页的存储——分布式计算框架MAPREDUC
  • 插入表主键冲突做更新 a-john
    有以下场景： 用户下了一个订单，订单内的内容较多，且来自多表，首次下单的时候，内容可能会不全（部分内容不是必须，出现有些表根本就没有没有该订单的值）。在以后更改订单时，有些内容会更改，有些内容会新增。 问题： 如果在sql语句中执行update操作，在没有数据的表中会出错。如果在逻辑代码中先做查询，查询结果有做更新，没有做插入，这样会将代码复杂化。 解决： mysql中提供了一个sql语
  • Android xml资源文件中@、@android:type、@*、？、@+含义和区别 Cb123456 @+@?@*
    一.@代表引用资源 1.引用自定义资源。格式：@[package:]type/name android：text="@string/hello"   2.引用系统资源。格式：@android:type/name     android:textColor="@android:color/opaque_red"
  • 数据结构的基本介绍 天子之骄 数据结构散列表树、图线性结构价格标签
    数据结构的基本介绍 数据结构就是数据的组织形式，用一种提前设计好的框架去存取数据，以便更方便，高效的对数据进行增删查改。正确选择合适的数据结构，对软件程序的高效执行的影响作用不亚于算法的设计。此外，在计算机系统中数据结构的作用也是非同小可。例如常常在编程语言中听到的栈，堆等，就是经典的数据结构。   经典的数据结构大致如下：   一：线性数据结构 (1)：列表 a
  • 通过二维码开放平台的API快速生成二维码 一炮送你回车库 api
     现在很多网站都有通过扫二维码用手机连接的功能，联图网(http://www.liantu.com/pingtai/)的二维码开放平台开放了一个生成二维码图片的Api,挺方便使用的。闲着无聊，写了个前台快速生成二维码的方法。        html代码如下:(二维码将生成在这div下) ? 1  &nbs
  • ImageIO读取一张图片改变大小 3213213333332132 javaIOimageBufferedImage
    package com.demo; import java.awt.image.BufferedImage; import java.io.File; import java.io.IOException; import javax.imageio.ImageIO; /** * @Description 读取一张图片改变大小 * @author FuJianyon
  • myeclipse集成svn（一针见血） 7454103 eclipseSVNMyEclipse
                                     &n
  • 装箱与拆箱----autoboxing和unboxing darkranger J2SE
    4.2　自动装箱和拆箱 基本数据(Primitive)类型的自动装箱(autoboxing)、拆箱(unboxing)是自J2SE 5.0开始提供的功能。虽然为您打包基本数据类型提供了方便，但提供方便的同时表示隐藏了细节，建议在能够区分基本数据类型与对象的差别时再使用。 4.2.1　autoboxing和unboxing 在Java中，所有要处理的东西几乎都是对象(Object)
  • ajax传统的方式制作ajax aijuans Ajax
    //这是前台的代码 <%@ page language="java" import="java.util.*" pageEncoding="UTF-8"%> <% String path = request.getContextPath(); String basePath = request.getScheme()+
  • 只用jre的eclipse是怎么编译java源文件的？ avords javaeclipsejdktomcat
    eclipse只需要jre就可以运行开发java程序了，也能自动 编译java源代码，但是jre不是java的运行环境么，难道jre中也带有编译工具？ 还是eclipse自己实现的？谁能给解释一下呢问题补充：假设系统中没有安装jdk or jre，只在eclipse的目录中有一个jre，那么eclipse会采用该jre，问题是eclipse照样可以编译java源文件，为什么呢？ &nb
  • 前端模块化 bee1314 模块化
    背景： 前端JavaScript模块化，其实已经不是什么新鲜事了。但是很多的项目还没有真正的使用起来，还处于刀耕火种的野蛮生长阶段。   JavaScript一直缺乏有效的包管理机制，造成了大量的全局变量，大量的方法冲突。我们多么渴望有天能像Java（import），Python (import)，Ruby(require)那样写代码。在没有包管理机制的年代，我们是怎么避免所
  • 处理百万级以上的数据处理 bijian1013 oraclesql数据库大数据查询
    一.处理百万级以上的数据提高查询速度的方法：        1.应尽量避免在 where 子句中使用!=或<>操作符，否则将引擎放弃使用索引而进行全表扫描。         2.对查询进行优化，应尽量避免全表扫描，首先应考虑在 where 及 o
  • mac 卸载 java 1.7 或更高版本 征客丶 javaOS
    卸载 java 1.7 或更高 sudo rm -rf /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin 成功执行此命令后，还可以执行 java 与 javac 命令 sudo rm -rf /Library/PreferencePanes/JavaControlPanel.prefPane 成功执行此命令后，还可以执行 java
  • 【Spark六十一】Spark Streaming结合Flume、Kafka进行日志分析 bit1129 Stream
    第一步，Flume和Kakfa对接，Flume抓取日志，写到Kafka中 第二部，Spark Streaming读取Kafka中的数据，进行实时分析   本文首先使用Kakfa自带的消息处理（脚本）来获取消息，走通Flume和Kafka的对接 1. Flume配置 1. 下载Flume和Kafka集成的插件，下载地址：https://github.com/beyondj2ee/f
  • Erlang vs TNSDL bookjovi erlang
          TNSDL是Nokia内部用于开发电信交换软件的私有语言，是在SDL语言的基础上加以修改而成，TNSDL需翻译成C语言得以编译执行，TNSDL语言中实现了异步并行的特点，当然要完整实现异步并行还需要运行时动态库的支持，异步并行类似于Erlang的process（轻量级进程），TNSDL中则称之为hand，Erlang是基于vm(beam)开发，
  • 非常希望有一个预防疲劳的java软件, 预防过劳死和眼睛疲劳,大家一起努力搞一个 ljy325 企业应用
    　非常希望有一个预防疲劳的java软件，我看新闻和网站，国防科技大学的科学家累死了，太疲劳，老是加班，不休息，经常吃药，吃药根本就没用，根本原因是疲劳过度。我以前做java,那会公司垃圾，老想赶快学习到东西跳槽离开，搞得超负荷，不明理。深圳做软件开发经常累死人，总有不明理的人，有个软件提醒限制很好，可以挽救很多人的生命。 相关新闻： （1）IT行业成五大疾病重灾区：过劳死平均37.9岁
  • 读《研磨设计模式》-代码笔记-原型模式 bylijinnan java设计模式
    声明： 本文只为方便我个人查阅和理解，详细的分析以及源代码请移步 原作者的博客http://chjavach.iteye.com/ /** * Effective Java 建议使用copy constructor or copy factory来代替clone()方法： * 1.public Product copy(Product p){} * 2.publi
  • 配置管理---svn工具之权限配置 chenyu19891124 SVN
    今天花了大半天的功夫，终于弄懂svn权限配置。下面是今天收获的战绩。 安装完svn后就是在svn中建立版本库，比如我本地的是版本库路径是C:\Repositories\pepos。pepos是我的版本库。在pepos的目录结构 pepos    component    webapps 在conf里面的auth里赋予的权限配置为 [groups]
  • 浅谈程序员的数学修养 comsci 设计模式编程算法面试招聘
                                     浅谈程序员的数学修养
  • 批量执行 bulk collect与forall用法 daizj oraclesqlbulk collectforall
    BULK COLLECT 子句会批量检索结果，即一次性将结果集绑定到一个集合变量中，并从SQL引擎发送到PL/SQL引擎。通常可以在SELECT INTO、 FETCH INTO以及RETURNING INTO子句中使用BULK COLLECT。本文将逐一描述BULK COLLECT在这几种情形下的用法。     有关FORALL语句的用法请参考：批量SQL之 F
  • Linux下使用rsync最快速删除海量文件的方法 dongwei_6688 OS
    1、先安装rsync：yum install rsync 2、建立一个空的文件夹：mkdir /tmp/test 3、用rsync删除目标目录：rsync --delete-before -a -H -v --progress --stats /tmp/test/ log/这样我们要删除的log目录就会被清空了，删除的速度会非常快。rsync实际上用的是替换原理，处理数十万个文件也是秒删。
  • Yii CModel中rules验证规格 dcj3sjt126com rulesyiivalidate
    Yii cValidator主要用法分析：  yii验证rulesit 分类： Yii yii的rules验证 cValidator主要属性 attributes ,builtInValidators,enableClientValidation,message,on,safe,skipOnError  
  • 基于vagrant的redis主从实验 dcj3sjt126com vagrant
    平台: Mac 工具: Vagrant 系统: Centos6.5 实验目的: Redis主从   实现思路 制作一个基于sentos6.5, 已经安装好reids的box, 添加一个脚本配置从机, 然后作为后面主机从机的基础box   制作sentos6.5+redis的box   mkdir vagrant_redis cd vagrant_
  • Memcached(二)、Centos安装Memcached服务器 frank1234 centosmemcached
    一、安装gcc rpm和yum安装memcached服务器连接没有找到，所以我使用的是make的方式安装，由于make依赖于gcc，所以要先安装gcc 开始安装，命令如下，[color=red][b]顺序一定不能出错[/b][/color]： 建议可以先切换到root用户，不然可能会遇到权限问题：su root 输入密码...... rpm -ivh kernel-head
  • Remove Duplicates from Sorted List hcx2013 remove
    Given a sorted linked list, delete all duplicates such that each element appear only once. For example,Given 1->1->2, return 1->2.Given 1->1->2->3->3, return&
  • Spring4新特性——JSR310日期时间API的支持 jinnianshilongnian spring4
    Spring4新特性——泛型限定式依赖注入 Spring4新特性——核心容器的其他改进 Spring4新特性——Web开发的增强 Spring4新特性——集成Bean Validation 1.1(JSR-349)到SpringMVC  Spring4新特性——Groovy Bean定义DSL Spring4新特性——更好的Java泛型操作API  Spring4新
  • 浅谈enum与单例设计模式 247687009 java单例
    在JDK1.5之前的单例实现方式有两种(懒汉式和饿汉式并无设计上的区别故看做一种)，两者同是私有构 造器，导出静态成员变量，以便调用者访问。 第一种 package singleton; public class Singleton { //导出全局成员 public final static Singleton INSTANCE = new S
  • 使用switch条件语句需要注意的几点 openwrt cbreakswitch
    1. 当满足条件的case中没有break，程序将依次执行其后的每种条件（包括default）直到遇到break跳出 int main() { int n = 1; switch(n) { case 1: printf("--1--\n"); default: printf("defa
  • 配置Spring Mybatis JUnit测试环境的应用上下文 schnell18 springmybatisJUnit
    Spring-test模块中的应用上下文和web及spring boot的有很大差异。主要试下来差异有： 单元测试的app context不支持从外部properties文件注入属性 @Value注解不能解析带通配符的路径字符串 解决第一个问题可以配置一个PropertyPlaceholderConfigurer的bean。 第二个问题的具体实例是：    
  • Java 定时任务总结一 tuoni javaspringtimerquartztimertask
     Java定时任务总结  一.从技术上分类大概分为以下三种方式：  1.Java自带的java.util.Timer类，这个类允许你调度一个java.util.TimerTask任务;   说明：    java.util.Timer定时器，实际上是个线程，定时执行TimerTask类 &
  • 一种防止用户生成内容站点出现商业广告以及非法有害等垃圾信息的方法 yangshangchuan rank相似度计算文本相似度词袋模型余弦相似度
    本文描述了一种在ITEYE博客频道上面出现的新型的商业广告形式及其应对方法，对于其他的用户生成内容站点类型也具有同样的适用性。   最近在ITEYE博客频道上面出现了一种新型的商业广告形式，方法如下：     1、注册多个账号（一般10个以上）。     2、从多个账号中选择一个账号，发表1-2篇博文
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他