nutch爬取内容分析和爬取流程命令实现

1、 nutch的存储文件夹data下面各个文件夹和文件里面的内容究竟是什么？

  crawl             one-step crawler for intranets (DEPRECATED - USE CRAWL SCRIPT INSTEAD)
  readdb            read / dump crawl db
  mergedb           merge crawldb-s, with optional filtering
  readlinkdb        read / dump link db
  inject            inject new urls into the database
  generate          generate new segments to fetch from crawl db
  freegen           generate new segments to fetch from text files
  fetch             fetch a segment's pages
  parse             parse a segment's pages
  readseg           read / dump segment data
  mergesegs         merge several segments, with optional filtering and slicing
  updatedb          update crawl db from segments after fetching
  invertlinks       create a linkdb from parsed segments
  mergelinkdb       merge linkdb-s, with optional filtering
  solrindex         run the solr indexer on parsed segments and linkdb
  solrdedup         remove duplicates from solr
  solrclean         remove HTTP 301 and 404 documents from solr
  parsechecker      check the parser for a given url
  indexchecker      check the indexing filters for a given url
  domainstats       calculate domain statistics from crawldb
  webgraph          generate a web graph from existing segments
  linkrank          run a link analysis program on the generated web graph
  scoreupdater      updates the crawldb with linkrank scores
  nodedumper        dumps the web graph's node scores
  plugin            load a plugin and run one of its classes main()
  junit             runs the given JUnit test
 or
  CLASSNAME         run the class named CLASSNAME

这次我们将要使用的命令是readdb readseg readlinkdb来查看目录下的相关内容信息

crawldb

bin/nutch | grep read

bin/nutch  readdb  data/crawldb  -stats

bin/nutch  readdb  data/crawldb  -dump  data/crawldb/crawldb_dump

bin/nutch  readdb  data/crawldb  -url  http://4008209999.tianyaclub.com/

bin/nutch  readdb  data/crawldb  -topN  10  data/crawldb/crawldb_topN

bin/nutch readdb data/crawldb  -topN 10  data/crawldb/crawldb_topN_m 1

segments

crawl_generate：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nocontent -nofetch -noparse -noparsedata  –noparsetext

crawl_fetch：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nocontent -nogenerate -noparse -noparsedata  –noparsetext

content：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nofetch  -nogenerate -noparse -noparsedata  –noparsetext

crawl_parse：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nofetch  -nogenerate -nocontent –noparsedata  –noparsetext

parse_data：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nofetch  -nogenerate -nocontent -noparse  –noparsetext

parse_text：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump -nofetch  -nogenerate -nocontent -noparse  -noparsedata

全部：

bin/nutch readseg -dump data/segments/20130325042858 data/segments/20130325042858_dump

segments

bin/nutch readseg -list -dir data/segments

bin/nutch readseg -list data/segments/20130325043023

bin/nutch readseg -get  data/segments/20130325042858 http://blog.tianya.cn/

linkdb

bin/nutch readlinkdb data/linkdb -url http://4008209999.tianyaclub.com/

bin/nutch readlinkdb data/linkdb -dump data/linkdb_dump

2.nutch爬取流程的命令实现

第一步 引入

bin/nutch inject

Usage: Injector <crawldb> <url_dir>

第一个是crawldb的生成目录  第二个为初始的目标url的目录

 第二步 generate 生成抓取列表

第三步 fetch 抓取

第四步 parse 解析抓取结果

第五步 updatedb 更新抓取列表

如果想进行多轮抓取 执行2-5步即可

最后抓取结束 执行 invertlinks 生成linkdb