Nutch1.0 日志分析（转）

Hadoop集群创建文件

[nutch@gc01vm13 /]$ cd ./home/nutch/nutchinstall/nutch-1.0/ [nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -ls Found 1 items drwxr-xr-x   - nutch supergroup          0 2010-06-09 20:10 /user/nutch/zklin [nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -mkdir crawldatatest   //mkdir 的地址相对于/user/nutch [nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -ls Found 2 items drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:40 /user/nutch/crawldatatest drwxr-xr-x   - nutch supergroup          0 2010-06-09 20:10 /user/nutch/zklin

 

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -mkdir urls [nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -ls Found 3 items drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:40 /user/nutch/crawldatatest drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:45 /user/nutch/urls drwxr-xr-x   - nutch supergroup          0 2010-06-09 20:10 /user/nutch/zklin

 

[nutch@gc01vm13 nutchinstall]$ mkdir urls   //先在本地创建urls文件夹 [nutch@gc01vm13 nutchinstall]$ cd ./urls/ [nutch@gc01vm13 urls]$ ls [nutch@gc01vm13 urls]$ vim urls1           //写url入口地址，urls1 [nutch@gc01vm13 urls]$ cd .. [nutch@gc01vm13 nutchinstall]$ cd ./nutch-1.0/ [nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -copyFromLocal /home/nutch/nutchinstall/urls/urls1 urls   //从本地拷贝到集群  集群是相对路径

 

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -lsr  //在集群上查看urls1 drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:40 /user/nutch/crawldatatest drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:46 /user/nutch/urls -rw-r--r--   2 nutch supergroup         31 2010-06-11 00:46 /user/nutch/urls/urls1 drwxr-xr-x   - nutch supergroup          0 2010-06-09 20:10 /user/nutch/zklin

 

[nutch@gc01vm13 nutch-1.0]$ bin/nutch crawl urls1 -dir crawldatatest -depth 3 -topN 10   Crawl需要用绝对路径，相对路径报错   crawl started in: crawldatatest   rootUrlDir = urls1 threads = 10 depth = 3 topN = 10 Injector: starting Injector: crawlDb: crawldatatest/crawldb Injector: urlDir: urls1 Injector: Converting injected urls to crawl db entries. Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://gc01vm13:9000/user/nutch/urls1          at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)          at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)          at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:797)          at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)          at org.apache.nutch.crawl.Injector.inject(Injector.java:160)          at org.apache.nutch.crawl.Crawl.main(Crawl.java:113)

 

Crawl需要用绝对路径，相对路径报错

 

日志分析

 

[nutch@gc01vm13 nutch-1.0]$ bin/nutch crawl /user/nutch/urls/urls1 -dir crawldatatest -depth 3 -topN 10    // crawldatatest是爬虫后的数据的存放位置，相对路径，和nutch-site.xml的Search.dir一致 crawl started in: crawldatatest         //表明网络蜘蛛的名称 rootUrlDir = /user/nutch/urls/urls1     //待下载数据的列表文件或列表 threads = 10 depth = 3 topN = 10

 

Injector: starting                    //注入下载列表 Injector: crawlDb: crawldatatest/crawldb Injector: urlDir: /user/nutch/urls/urls1 Injector: Converting injected urls to crawl db entries.  //根据注入的列表生成待下载的地址库 Injector: Merging injected urls into crawl db.   //执行Merge Injector: done

 

Generator: Selecting best-scoring urls due for fetch.   //判断网页重要性，决定下载顺序 Generator: starting Generator: segment: crawldatatest/segments/20100611004927  //生成下载结果存储的数据段 Generator: filtering: true Generator: topN: 10 Generator: Partitioning selected urls by host, for politeness.   //将url下载列表按Hadoop中的配置文件slaves中定义的datanode来分配。按host分配。 Generator: done.

 

Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property. Fetcher: starting Fetcher: segment: crawldatatest/segments/20100611004927  //下载指定网页内容到segment中去 Fetcher: done

 

CrawlDb update: starting   //下载完毕后，更新下载数据库，增加新的下载 CrawlDb update: db: crawldatatest/crawldb CrawlDb update: segments: [crawldatatest/segments/20100611004927] CrawlDb update: additions allowed: true CrawlDb update: URL normalizing: true CrawlDb update: URL filtering: true CrawlDb update: Merging segment data into db. CrawlDb update: done

 

//循环执行下载 第二次

Generator: Selecting best-scoring urls due for fetch. Generator: starting Generator: segment: crawldatatest/segments/20100611005051 Generator: filtering: true Generator: topN: 10 Generator: Partitioning selected urls by host, for politeness. Generator: done.

 

Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property. Fetcher: starting Fetcher: segment: crawldatatest/segments/20100611005051 Fetcher: done

 

CrawlDb update: starting CrawlDb update: db: crawldatatest/crawldb CrawlDb update: segments: [crawldatatest/segments/20100611005051] CrawlDb update: additions allowed: true CrawlDb update: URL normalizing: true CrawlDb update: URL filtering: true CrawlDb update: Merging segment data into db. CrawlDb update: done

 

//循环下载，第三次

Generator: Selecting best-scoring urls due for fetch. Generator: starting Generator: segment: crawldatatest/segments/20100611005212 Generator: filtering: true Generator: topN: 10 Generator: Partitioning selected urls by host, for politeness. Generator: done.

 

Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property. Fetcher: starting Fetcher: segment: crawldatatest/segments/20100611005212 Fetcher: done

 

CrawlDb update: starting CrawlDb update: db: crawldatatest/crawldb CrawlDb update: segments: [crawldatatest/segments/20100611005212] CrawlDb update: additions allowed: true CrawlDb update: URL normalizing: true CrawlDb update: URL filtering: true CrawlDb update: Merging segment data into db. CrawlDb update: done

 

//总共循环depth次，Nutch的局域网模式采用了广度优先策略，把二级页面抓取完毕以后，进行三级页面抓取

LinkDb: starting           //进行网页链接关系分析 LinkDb: linkdb: crawldatatest/linkdb LinkDb: URL normalize: true    //规范化 LinkDb: URL filter: true        //根据Crawl-urlfiter.txt来过滤 LinkDb: adding segment: hdfs://gc01vm13:9000/user/nutch/crawldatatest/segments/20100611004927 LinkDb: adding segment: hdfs://gc01vm13:9000/user/nutch/crawldatatest/segments/20100611005051 LinkDb: adding segment: hdfs://gc01vm13:9000/user/nutch/crawldatatest/segments/20100611005212 LinkDb: done    //链接分析完毕

 

 

Indexer: starting   //开始创建索引 Indexer: done Dedup: starting   //网页去重 Dedup: adding indexes in: crawldatatest/indexes Dedup: done merging indexes to: crawldatatest/index    //索引合并 Adding hdfs://gc01vm13:9000/user/nutch/crawldatatest/indexes/part-00000 done merging crawl finished: crawldatatest   //结束 [nutch@gc01vm13 nutch-1.0]$

 

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -copyToLocal crawldatatest /home/nutch/nutchinstall/ [nutch@gc01vm13 nutch-1.0]$ cd .. [nutch@gc01vm13 nutchinstall]$ ls confbak  crawldatatest  filesystem  hadoopscheduler  hadooptmp  nutch-1.0  urls

copyToLocal用的是相对路径

 

生出数据分析

bin/nutch crawl /user/nutch/urls/urls1 -dir crawldatatest -depth 3 -topN 10

从集群拷贝到本地

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -copyToLocal crawldatatest /home/nutch/nutchinstall/ [nutch@gc01vm13 nutch-1.0]$ cd .. [nutch@gc01vm13 nutchinstall]$ ls confbak  crawldatatest  filesystem  hadoopscheduler  hadooptmp  nutch-1.0  urls

  

 

抓取程序在集群用户根目录（/user/nutch）下面建立目录，

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -ls Found 3 items drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:55 /user/nutch/crawldatatest drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:46 /user/nutch/urls drwxr-xr-x   - nutch supergroup          0 2010-06-09 20:10 /user/nutch/zklin

 

在crawldatatest目录下生成了crawldb,segments,index,indexs,linkdb五个文件

[nutch@gc01vm13 nutch-1.0]$ bin/hadoop fs -ls /user/nutch/crawldatatest Found 5 items drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:53 /user/nutch/crawldatatest/crawldb drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:55 /user/nutch/crawldatatest/index drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:54 /user/nutch/crawldatatest/indexes drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:53 /user/nutch/crawldatatest/linkdb drwxr-xr-x   - nutch supergroup          0 2010-06-11 00:52 /user/nutch/crawldatatest/segments

 

1) crawldb目录下面存放下载的URL,以及下载的日期，用来页面更新检查时间。

2) linkdb目录存放URL的关联关系，是下载完成后分析时创建的，通过这个关联关系可以实现类似google的pagerank功能。

3) segments目录存储抓取的页面，下面子目录的个数与获取页面的层数有关系，我指定-depth是3层，这个目录下就有3层。

  里面有6个子目录

  content,下载页面的内容；

  crawl_fetch,下载URL的状态内容；

  crawl_generate,待下载的URL的集合，在generate任务生成时和下载过程中持续分析出来；

  crawl_parse,存放用来更新crawldb的外部链接库；

  parse_data,存放每个URL解析出来的外部链接和元数据；

  parse_text,存放每个解析过的URL的文本内容；

4) index目录存放符合lucene格式的索引目录，是indexs里所有的索引内容合并后的完整内容,看了一下这里的索引文件和用 lucenedemo做出来的文件名称都不一样，待进一步研究；

5 )indexs目录存放每次下载的索引目录，存放part-0000；