键盘上的孤舞者

Nutch一次爬取运行结果

test@admin:~/programs/nutch-1.2$ ./bin/nutch crawl urls/seed.txt -dir localweb -depth 5 -threads 4 -topN 50
crawl started in: localweb
rootUrlDir = urls/seed.txt
threads = 4
depth = 5
indexer=lucene
topN = 50
Injector: starting at 2014-05-23 14:20:11
Injector: crawlDb: localweb/crawldb
Injector: urlDir: urls/seed.txt
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: finished at 2014-05-23 14:20:15, elapsed: 00:00:03
Generator: starting at 2014-05-23 14:20:15
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 50
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: localweb/segments/20140523142018
Generator: finished at 2014-05-23 14:20:19, elapsed: 00:00:04
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2014-05-23 14:20:19
Fetcher: segment: localweb/segments/20140523142018
Fetcher: threads: 4
QueueFeeder finished: total 50 records + hit by time limit :0
fetching http://www.cafepress.com/nutch/
fetching http://www.baidu.com/
fetching http://www.apache.org/dist/nutch/2.2.1/CHANGES-2.2.1.txt
fetching http://xw.qq.com/simple/s/finance/index.htm
fetching http://issues.apache.org/jira/browse/NUTCH
fetch of http://www.cafepress.com/nutch/ failed with: java.net.SocketException: Connection reset
fetching http://forrest.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=44
fetching http://house60.3g.qq.com/g/welcome.jsp
fetching http://www.oschina.net/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=42
fetching http://www.elasticsearch.org/
fetch of http://www.oschina.net/ failed with: Http code=403, url=http://www.oschina.net/
fetching http://xw.qq.com/a/ent/20140418023432/ENT20140418023432AE
fetching http://mat1.gtimg.com/
fetching http://wiki.apache.org/nutch/
fetching http://nutch.apache.org/credits.html
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=37
fetching http://xw.qq.com/simple/s/house/index.htm
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=36
fetching http://search.maven.org/
fetching http://code.google.com/p/crawler-commons/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=34
fetching http://s.apache.org/oHY
fetching http://xw.qq.com/simple/s/games/index.htm
fetching http://find.searchhub.org/p:nutch
fetching http://nutch.apache.org/skin/fontsize.js
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=30
fetching http://qgame.3g.qq.com/
Error parsing: http://nutch.apache.org/skin/fontsize.js: failed(2,0): Can't retrieve Tika parser for mime-type application/javascript
fetching http://www.apache.org/dyn/closer.cgi/nutch/
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=28
fetching http://xw.qq.com/simple/s/fashion/index.htm
fetching http://nutch.apache.org/nightly.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=26
fetching http://s.apache.org/PGa
fetching http://xw.qq.com/a/ent/20140523003834/ENT2014052300383403
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=24
fetching http://nutch.apache.org/old_downloads.html
fetching http://xw.qq.com/a/ent/20131128004407/ENT201311280044077O
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=22
fetching http://nutch.apache.org/skin/breadcrumbs.js
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=21
fetching http://xw.qq.com/a/ent/20140523001154/ENT2014052300115404
fetching http://www.apache.org/security/
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=19
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=19
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=19
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=19
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=19
fetching http://www.apache.org/foundation/thanks.html
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
fetch of http://nutch.apache.org/skin/breadcrumbs.js failed with: java.net.SocketTimeoutException: connect timed out
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=18
fetching http://www.apache.org/foundation/sponsorship.html
fetching http://nutch.apache.org/menu_1.3
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=16
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=16
fetching http://nutch.apache.org/skin/getMenu.js
Error parsing: http://nutch.apache.org/skin/getMenu.js: failed(2,0): Can't retrieve Tika parser for mime-type application/javascript
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=15
fetching http://nutch.apache.org/index.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=14
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=14
fetching http://www.apache.org/dist/nutch/1.8/CHANGES.txt
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=13
fetching http://nutch.apache.org/apidocs-1.8/index.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=12
fetching http://nutch.apache.org/sonar.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=11
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=11
fetching http://nutch.apache.org/apidocs-2.2.1/index.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=10
fetching http://nutch.apache.org/bot.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=9
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=9
fetching http://nutch.apache.org/issue_tracking.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=8
fetching http://nutch.apache.org/faq.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=7
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=7
fetching http://nutch.apache.org/version_control.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=6
fetching http://nutch.apache.org/menu_1.4
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=5
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=5
fetching http://nutch.apache.org/tutorial.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=4
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826062347
  now = 1400826061417
  0. http://nutch.apache.org/menu_1.2
  1. http://nutch.apache.org/skin/getBlank.js
  2. http://nutch.apache.org/index.pdf
  3. http://nutch.apache.org/menu_selected_1.1
fetching http://nutch.apache.org/menu_1.2
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=3
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826062347
  now = 1400826062418
  0. http://nutch.apache.org/skin/getBlank.js
  1. http://nutch.apache.org/index.pdf
  2. http://nutch.apache.org/menu_selected_1.1
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=3
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826063932
  now = 1400826063420
  0. http://nutch.apache.org/skin/getBlank.js
  1. http://nutch.apache.org/index.pdf
  2. http://nutch.apache.org/menu_selected_1.1
fetching http://nutch.apache.org/skin/getBlank.js
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=2
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826063932
  now = 1400826064425
  0. http://nutch.apache.org/index.pdf
  1. http://nutch.apache.org/menu_selected_1.1
Error parsing: http://nutch.apache.org/skin/getBlank.js: failed(2,0): Can't retrieve Tika parser for mime-type application/javascript
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=2
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826065480
  now = 1400826065427
  0. http://nutch.apache.org/index.pdf
  1. http://nutch.apache.org/menu_selected_1.1
fetching http://nutch.apache.org/index.pdf
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=1
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826065480
  now = 1400826066430
  0. http://nutch.apache.org/menu_selected_1.1
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=1
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826065480
  now = 1400826067431
  0. http://nutch.apache.org/menu_selected_1.1
Error parsing: http://nutch.apache.org/index.pdf: failed(2,0): expected='endstream' actual='' org.apache.pdfbox.io.PushBackInputStream@65d43d1e
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826068954
  now = 1400826068433
  0. http://nutch.apache.org/menu_selected_1.1
fetching http://nutch.apache.org/menu_selected_1.1
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2014-05-23 14:21:12, elapsed: 00:00:52
CrawlDb update: starting at 2014-05-23 14:21:12
CrawlDb update: db: localweb/crawldb
CrawlDb update: segments: [localweb/segments/20140523142018]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2014-05-23 14:21:14, elapsed: 00:00:02
Generator: starting at 2014-05-23 14:21:14
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 50
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: localweb/segments/20140523142116
Generator: finished at 2014-05-23 14:21:18, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2014-05-23 14:21:18
Fetcher: segment: localweb/segments/20140523142116
Fetcher: threads: 4
QueueFeeder finished: total 50 records + hit by time limit :0
fetching http://isdspeed.qq.com/
fetching http://nutch.apache.org/apidocs-2.2.1/overview-summary.html
fetching http://pnewsapp.tc.qq.com/newsapp_ls/0/17978428_150110/0
fetching http://pingjs.qq.com/ping.js
fetching http://lucene.apache.org/hadoop
fetching http://m.v.qq.com/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=44
fetching http://xw.qq.com/a/ent/20140523013234/ENT2014052301323407
fetching http://imgcache.qq.com/
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=42
fetching http://nutch.apache.org/apidocs-1.8/overview-summary.html
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=41
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=41
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=41
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=41
fetching http://xw.qq.com/a/fashion/20140523005888/FAS2014052300588802
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=40
fetching http://xw.qq.com/a/finance/20140510015766/FIN2014051001576635
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=39
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=39
fetching http://xw.qq.com/a/fashion/20140523018320/FAS2014052301832001
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=38
fetching http://xw.qq.com/a/news/20140523016639/NEW2014052301663902
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=37
fetching http://xw.qq.com/a/house/20140523005159/HOS2014052300515906
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=36
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=36
fetching http://xw.qq.com/m/house/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=35
fetching http://xw.qq.com/a/auto/20140522044474/AUT2014052204447404
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=34
fetching http://xw.qq.com/a/digi_tech/20140523022831/DIG2014052302283101
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=33
fetching http://xw.qq.com/m/photo/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=32
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=32
fetching http://xw.qq.com/a/finance/20140516032679/FIN201405160326790Q
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=31
fetching http://xw.qq.com/m/auto/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=30
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=30
fetching http://xw.qq.com/a/auto/20140523006942/AUT2014052300694202
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=29
fetching http://xw.qq.com/m/news/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=28
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=28
fetching http://xw.qq.com/a/auto/20140523007567/AUT2014052300756702
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=27
fetching http://xw.qq.com/a/digi_tech/20140523013949/DIG2014052301394901
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=26
fetching http://xw.qq.com/a/finance/20140523009913/FIN201405230099130A
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=25
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=25
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=25
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=25
fetching http://xw.qq.com/c/ent/20140523003834/ENT2014052300383403
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=24
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=24
fetching http://xw.qq.com/a/news/20140523022774/NEW201405230227740H
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=23
fetching http://xw.qq.com/m/digi/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=22
fetching http://xw.qq.com/a/news/20140523007240/MIL2014052300724002
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=21
fetching http://xw.qq.com/c/news/20140523022986/NEW2014052302298605
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=20
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=20
fetching http://xw.qq.com/a/finance/20140521060120/FIN201405210601200T
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=19
fetching http://xw.qq.com/a/news/20140523005827/MIL2014052300582702
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=18
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=18
fetching http://xw.qq.com/a/house/20140523005305/HOS2014052300530501
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=17
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=17
fetching http://xw.qq.com/a/news/20140523024720/NEW2014052302472004
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=16
fetching http://xw.qq.com/a/news/20140523008280/MIL2014052300828002
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=15
fetching http://xw.qq.com/c/zt/201312100083161/SPO201312100083161R
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=14
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=14
fetching http://xw.qq.com/m/fashion/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=13
fetching http://xw.qq.com/a/news/20140523017079/MIL2014052301707904
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=12
fetching http://xw.qq.com/a/digi_tech/20140521009545/DIG201405210095451E
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=11
fetching http://xw.qq.com/a/auto/20140523007267/AUT2014052300726702
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=10
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=10
fetching http://xw.qq.com/a/finance/20140523008356/FIN2014052300835605
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=9
fetching http://xw.qq.com/m/astro/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=8
fetching http://xw.qq.com/a/house/20140523014070/HOS2014052301407005
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=7
fetching http://xw.qq.com/a/news/20140523022986/NEW2014052302298605
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=6
fetching http://xw.qq.com/m/games/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=5
fetching http://xw.qq.com/a/digi_tech/20140523019030/DIG2014052301903001
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=4
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826148455
  now = 1400826147670
  0. http://xw.qq.com/a/house/20140523005188/HOS2014052300518801
  1. http://xw.qq.com/m/finance/
  2. http://xw.qq.com/m/tech/
  3. http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
fetching http://xw.qq.com/a/house/20140523005188/HOS2014052300518801
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=3
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826148455
  now = 1400826148672
  0. http://xw.qq.com/m/finance/
  1. http://xw.qq.com/m/tech/
  2. http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=3
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826149689
  now = 1400826149675
  0. http://xw.qq.com/m/finance/
  1. http://xw.qq.com/m/tech/
  2. http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
fetching http://xw.qq.com/m/finance/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=2
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826150958
  now = 1400826150677
  0. http://xw.qq.com/m/tech/
  1. http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
fetching http://xw.qq.com/m/tech/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826152184
  now = 1400826151680
  0. http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
fetching http://xw.qq.com/a/news/20140523017822/MIL2014052301782203
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2014-05-23 14:22:34, elapsed: 00:01:16
CrawlDb update: starting at 2014-05-23 14:22:34
CrawlDb update: db: localweb/crawldb
CrawlDb update: segments: [localweb/segments/20140523142116]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2014-05-23 14:22:36, elapsed: 00:00:02
Generator: starting at 2014-05-23 14:22:36
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 50
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: localweb/segments/20140523142238
Generator: finished at 2014-05-23 14:22:39, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2014-05-23 14:22:39
Fetcher: segment: localweb/segments/20140523142238
Fetcher: threads: 4
QueueFeeder finished: total 50 records + hit by time limit :0
fetching http://btrace.qq.com/
fetching http://pingfore.qq.com/
fetching http://www.xw.qq.com/
fetching http://qt.gtimg.cn/
fetch of http://pingfore.qq.com/ failed with: java.io.EOFException
fetching http://jsqmt.qq.com/
fetch of http://btrace.qq.com/ failed with: Http code=403, url=http://btrace.qq.com/
fetching http://pnewsapp.tc.qq.com/newsapp_ls/0/17977385_640330/0
fetching http://sns.video.qq.com/
fetching http://xw.qq.com/a/sports/20131218015367/SPO20131218015367SL
fetching http://i.gtimg.cn/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=41
fetch of http://www.xw.qq.com/ failed with: Http code=500, url=http://www.xw.qq.com/
fetching http://openapi.inews.qq.com/
fetching http://vliveachy.tc.qq.com/
fetching http://jqmt.qq.com/
fetch of http://vliveachy.tc.qq.com/ failed with: Http code=405, url=http://vliveachy.tc.qq.com/
fetching http://mat1.gtimg.com/www/mobi/js/simple.article.js
fetching http://trace.qq.com/
fetching http://tajs.qq.com/stats
fetching http://ui.ptlogin2.qq.com/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=34
fetching http://vv.video.qq.com/
fetching http://live.qq.com/
fetching http://img.gtimg.cn/
fetch of http://img.gtimg.cn/ failed with: Http code=403, url=http://img.gtimg.cn/
fetching http://xw.qq.com/a/sports/20140522050024/SPO2014052205002408
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=30
fetching http://api.t.qq.com/
fetching http://vpic.video.qq.com/
fetching http://check.ptlogin2.qq.com/
fetch of http://check.ptlogin2.qq.com/ failed with: Http code=403, url=http://check.ptlogin2.qq.com/
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=27
fetching http://xw.qq.com/a/sports/20140523020282/SPO2014052302028205
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=26
fetching http://xw.qq.com/c/sports/20140522048392/SPO2014052204839205
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=25
fetching http://xw.qq.com/a/news/20140523008556/NEW2014052300855606
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=24
fetching http://xw.qq.com/c/sports/20140523023486/SPO2014052302348601
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=23
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=23
fetching http://xw.qq.com/a/news/20140523024872/NEW2014052302487201
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=22
fetching http://xw.qq.com/c/ent/20140522045677/ENT2014052204567701
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=21
fetching http://pnewsapp.tc.qq.com/newsapp_ls/0/17976606_640330/0
fetching http://xw.qq.com/c/ent/20140523001109/ENT2014052300110906
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=19
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=19
fetching http://xw.qq.com/a/news/20140523007960/NEW2014052300796006
fetch of http://vpic.video.qq.com/ failed with: java.net.SocketTimeoutException: Read timed out
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=18
fetching http://xw.qq.com/c/zt/20131218015367/SPO20131218015367SL
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=17
fetching http://xw.qq.com/a/sports/20140523023486/SPO2014052302348601
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=16
fetching http://xw.qq.com/c/sports/20140522050024/SPO2014052205002408
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=15
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=15
fetching http://xw.qq.com/c/zt/201405100152772/SPO201405100152772X
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=14
fetching http://xw.qq.com/simple/s/index/qq.com
-activeThreads=4, spinWaiting=1, fetchQueues.totalSize=13
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=13
fetching http://xw.qq.com/c/sports/20140523009111/SPO2014052300911102
fetch of http://jqmt.qq.com/ failed with: java.net.SocketTimeoutException: Read timed out
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=12
fetching http://xw.qq.com/a/news/20140523024749/NEW2014052302474905
fetching http://pnewsapp.tc.qq.com/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=10
fetching http://xw.qq.com/a/news/20140523013685/NEW2014052301368505
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=9
fetching http://xw.qq.com/c/ent/20140522048291/ENT2014052204829104
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=8
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=8
fetching http://xw.qq.com/c/ent/20140523000009/ENT2014052300000904
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=7
fetching http://xw.qq.com/c/sports/20140523024606/SPO2014052302460603
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
fetching http://xw.qq.com/a/sports/20131210008316/SPO201312100083161R
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=5
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=5
fetching http://xw.qq.com/c/mil/20140523005827/MIL2014052300582702
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=4
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826191797
  now = 1400826191299
  0. http://xw.qq.com/a/fashion/20140522044069/FAS201405220440690L
  1. http://xw.qq.com/c/zt/20140418023432/ENT20140418023432AE
  2. http://xw.qq.com/a/fashion/20140507013249/FAS201405070132490V
  3. http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
fetching http://xw.qq.com/a/fashion/20140522044069/FAS201405220440690L
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=3
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826193100
  now = 1400826192303
  0. http://xw.qq.com/c/zt/20140418023432/ENT20140418023432AE
  1. http://xw.qq.com/a/fashion/20140507013249/FAS201405070132490V
  2. http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
fetching http://xw.qq.com/c/zt/20140418023432/ENT20140418023432AE
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=2
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826193100
  now = 1400826193305
  0. http://xw.qq.com/a/fashion/20140507013249/FAS201405070132490V
  1. http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=2
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826194379
  now = 1400826194307
  0. http://xw.qq.com/a/fashion/20140507013249/FAS201405070132490V
  1. http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
fetching http://xw.qq.com/a/fashion/20140507013249/FAS201405070132490V
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=1
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826195692
  now = 1400826195308
  0. http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
fetching http://xw.qq.com/c/zt/201311280044077/ENT201311280044077O
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
fetch of http://pnewsapp.tc.qq.com/ failed with: java.net.SocketTimeoutException: Read timed out
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2014-05-23 14:23:25, elapsed: 00:00:45
CrawlDb update: starting at 2014-05-23 14:23:25
CrawlDb update: db: localweb/crawldb
CrawlDb update: segments: [localweb/segments/20140523142238]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2014-05-23 14:23:27, elapsed: 00:00:02
Generator: starting at 2014-05-23 14:23:27
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 50
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: localweb/segments/20140523142330
Generator: finished at 2014-05-23 14:23:31, elapsed: 00:00:04
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2014-05-23 14:23:31
Fetcher: segment: localweb/segments/20140523142330
Fetcher: threads: 4
QueueFeeder finished: total 50 records + hit by time limit :0
fetching http://www.qq.com/mobile/appios.htm
fetching http://nutch.apache.org/menu_1.1
fetching http://pnewsapp.tc.qq.com/newsapp_ls/0/17954361_150110/0
fetching http://lucene.apache.org/
fetching http://xw.qq.com/c/news/20140523007960/NEW2014052300796006
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=45
fetch of http://nutch.apache.org/menu_1.1 failed with: java.net.SocketException: Connection reset
fetching http://people.apache.org/committer-index.html
fetching http://s.apache.org/1zE
fetching http://v.qq.com/index.html
fetching http://ubook.qq.com/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=41
fetching http://xw.qq.com/iphone/m/finance/finance.htm
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=40
fetching http://xw.qq.com/c/news/20140523024720/NEW2014052302472004
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=39
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=39
fetching http://xw.qq.com/iphone/m/view/52292e0c763fd027c6eba6b8f494d2eb.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=38
fetching http://xw.qq.com/iphone/m/caijingguancha/dabd8d2ce74e782c65a973ef76fd540b.html
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=37
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=37
fetching http://xw.qq.com/iphone/m/tmtdecode/a3fb4fbf9a6f9cf09166aa9c20cbc1ad.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=36
fetching http://xw.qq.com/c/finance/20140523017080/FIN2014052301708006
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=35
fetching http://xw.qq.com/c/finance/20140523012251/FIN2014052301225102
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=34
fetching http://pnewsapp.tc.qq.com/newsapp_ls/0/17965405_150110/0
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=33
fetching http://xw.qq.com/c/yc/820140523/tmtdecode
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=32
fetching http://xw.qq.com/c/finance/20140523008356/FIN2014052300835605
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=31
fetching http://xw.qq.com/c/news/20140523024749/NEW2014052302474905
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=30
fetching http://xw.qq.com/c/zt/2014051001576635/FIN2014051001576635
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=29
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=29
fetching http://xw.qq.com/c/zt/201405210601200/FIN201405210601200T
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=28
fetching http://xw.qq.com/c/news/20140523013685/NEW2014052301368505
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=27
fetching http://xw.qq.com/c/zt/201403030155684/SPO20140303015568J4
-activeThreads=4, spinWaiting=2, fetchQueues.totalSize=26
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=26
fetching http://xw.qq.com/c/yc/520140523/guiquan
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=25
fetching http://xw.qq.com/m/shehui/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=24
fetching http://xw.qq.com/c/zhibo/10000300/FIN201405230099130A
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=23
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=23
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=23
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=23
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=23
fetching http://xw.qq.com/c/news/20140523026482/NEW2014052302648202
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=22
fetching http://xw.qq.com/iphone/m/dy/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=21
fetching http://xw.qq.com/c/ent/20140523001154/ENT2014052300115404
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=20
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=20
fetching http://xw.qq.com/c/yc/220140523/view
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=19
fetching http://xw.qq.com/c/ent/20140523013234/ENT2014052301323407
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=18
fetching http://xw.qq.com/c/sports/20140523020282/SPO2014052302028205
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=17
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=17
fetching http://xw.qq.com/c/finance/20140523009486/FIN2014052300948601
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=16
fetching http://xw.qq.com/c/news/20140523016639/NEW2014052301663902
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=15
fetching http://xw.qq.com/c/sports/20140523006605/SPO2014052300660502
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=14
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=14
fetching http://xw.qq.com/c/zt/201405160326790/FIN201405160326790Q
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=13
fetching http://xw.qq.com/c/news/20140523008556/NEW2014052300855606
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=12
fetching http://xw.qq.com/iphone/m/guiquan/7d771e0e8f3633ab54856925ecdefc5d.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=11
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=11
fetching http://xw.qq.com/c/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=10
fetching http://xw.qq.com/c/finance/20140516032679/FIN201405160326790Q
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=9
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=9
fetching http://xw.qq.com/c/ent/20140523000360/ENT2014052300036003
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=8
fetching http://xw.qq.com/iphone/m/sports/nba/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=7
fetching http://xw.qq.com/m/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=6
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=6
fetching http://xw.qq.com/c/yc/1120140523/caijingguancha
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=5
fetching http://xw.qq.com/c/ent/20140523007189/ENT2014052300718904
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=4
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826265915
  now = 1400826265267
  0. http://xw.qq.com/c/news/20140523024872/NEW2014052302487201
  1. http://xw.qq.com/c/news/20140523022774/NEW201405230227740H
  2. http://xw.qq.com/c/news/20140523026425/NEW2014052302642501
  3. http://xw.qq.com/m/shijiebei/
fetching http://xw.qq.com/c/news/20140523024872/NEW2014052302487201
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=3
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826267201
  now = 1400826266269
  0. http://xw.qq.com/c/news/20140523022774/NEW201405230227740H
  1. http://xw.qq.com/c/news/20140523026425/NEW2014052302642501
  2. http://xw.qq.com/m/shijiebei/
fetching http://xw.qq.com/c/news/20140523022774/NEW201405230227740H
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=2
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826267201
  now = 1400826267269
  0. http://xw.qq.com/c/news/20140523026425/NEW2014052302642501
  1. http://xw.qq.com/m/shijiebei/
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=2
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826268467
  now = 1400826268271
  0. http://xw.qq.com/c/news/20140523026425/NEW2014052302642501
  1. http://xw.qq.com/m/shijiebei/
fetching http://xw.qq.com/c/news/20140523026425/NEW2014052302642501
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://xw.qq.com
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826269716
  now = 1400826269272
  0. http://xw.qq.com/m/shijiebei/
fetching http://xw.qq.com/m/shijiebei/
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2014-05-23 14:24:32, elapsed: 00:01:00
CrawlDb update: starting at 2014-05-23 14:24:32
CrawlDb update: db: localweb/crawldb
CrawlDb update: segments: [localweb/segments/20140523142330]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2014-05-23 14:24:34, elapsed: 00:00:02
Generator: starting at 2014-05-23 14:24:34
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 50
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: localweb/segments/20140523142436
Generator: finished at 2014-05-23 14:24:37, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.
Fetcher: starting at 2014-05-23 14:24:37
Fetcher: segment: localweb/segments/20140523142436
Fetcher: threads: 4
QueueFeeder finished: total 50 records + hit by time limit :0
fetching http://search-lucene.com/nutch
fetching http://lenya.apache.org/
fetching http://gump.apache.org/
fetching http://zookeeper.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=46
fetching http://repo1.maven.org/maven2/org/apache/gora/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=45
fetching http://gora.apache.org/downloads.html
fetching http://cassandra.apache.org/
fetching http://hadoop.apache.org/releases.html
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=42
fetching http://www.isi.edu/~koehn/europarl/
fetching http://lucene.apache.org/solr/books.html
fetching http://hive.apache.org/
fetching http://openmeetings.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=38
fetching http://s.apache.org/LPB
fetching http://giraph.apache.org/
fetching http://projects.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=35
fetching http://mahout.apache.org/
fetching http://www.eu.apachecon.com/c/aceu2009/
fetch of http://www.eu.apachecon.com/c/aceu2009/ failed with: java.net.UnknownHostException: www.eu.apachecon.com
fetching http://accumulo.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=32
fetching http://avro.apache.org/
fetching http://www.apache.org/licenses/LICENSE-2.0
fetching http://sqt.gtimg.cn/
fetching http://people.apache.org/committers-by-project.html
fetching http://imgcache.qq.com/ihome.qzone.qq.com
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=27
fetching http://incubator.apache.org/
fetching http://i.gtimg.cn/ihome.qzone.qq.com
fetching http://hbase.apache.org/
fetching http://ace.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=23
fetching http://mat1.gtimg.com/www/mobi/js/vote.v1.0.js
fetching http://cocoon.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=21
fetching http://wiki.apache.org/nutch/FrontPage
fetching http://nutch.apache.org/apidocs-2.2.1/allclasses-frame.html
fetching http://abdera.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=18
fetching http://tika.apache.org/download.html
fetching http://mat1.gtimg.com/www/mobi/js/template.min.js
fetching http://pig.apache.org/
fetching http://url.cn/JVoGlc
fetching http://activemq.apache.org/
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=13
fetching http://ant.apache.org/
fetching http://www.apache.org/dyn/closer.cgi
fetching http://mat1.gtimg.com/www/mobi/js/zepto.min.js
-activeThreads=4, spinWaiting=0, fetchQueues.totalSize=10
fetching http://nutch.apache.org/apidocs-1.8/overview-frame.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=9
fetching http://mat1.gtimg.com/www/mobi/js/article.v2.js
fetching http://nutch.apache.org/apidocs-2.2.1/overview-frame.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=7
fetching http://mat1.gtimg.com/www/mobi/js/QMobiFoot.js
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=6
fetching http://nutch.apache.org/apidocs-1.8/allclasses-frame.html
fetching http://mat1.gtimg.com/www/mobi/js/touch.min.js
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=4
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826294822
  now = 1400826294043
  0. http://www.apache.org/foundation/
  1. http://www.apache.org/licenses/LICENSE-2.0.html
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826293413
  now = 1400826294045
  0. http://nutch.apache.org/menu_selected_1.2
  1. http://nutch.apache.org/menu_selected_1.3
fetching http://www.apache.org/foundation/
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=3
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 1
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826294822
  now = 1400826295046
  0. http://www.apache.org/licenses/LICENSE-2.0.html
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826295239
  now = 1400826295047
  0. http://nutch.apache.org/menu_selected_1.2
  1. http://nutch.apache.org/menu_selected_1.3
fetching http://nutch.apache.org/menu_selected_1.2
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=2
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826299973
  now = 1400826296050
  0. http://www.apache.org/licenses/LICENSE-2.0.html
* queue: http://nutch.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 1000
  minCrawlDelay = 0
  nextFetchTime = 1400826296808
  now = 1400826296051
  0. http://nutch.apache.org/menu_selected_1.3
fetching http://nutch.apache.org/menu_selected_1.3
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=1
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826299973
  now = 1400826297052
  0. http://www.apache.org/licenses/LICENSE-2.0.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826299973
  now = 1400826298054
  0. http://www.apache.org/licenses/LICENSE-2.0.html
-activeThreads=4, spinWaiting=4, fetchQueues.totalSize=1
* queue: http://www.apache.org
  maxThreads = 1
  inProgress = 0
  crawlDelay = 4000
  minCrawlDelay = 0
  nextFetchTime = 1400826299973
  now = 1400826299055
  0. http://www.apache.org/licenses/LICENSE-2.0.html
fetching http://www.apache.org/licenses/LICENSE-2.0.html
-activeThreads=4, spinWaiting=3, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=3
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2014-05-23 14:25:02, elapsed: 00:00:25
CrawlDb update: starting at 2014-05-23 14:25:02
CrawlDb update: db: localweb/crawldb
CrawlDb update: segments: [localweb/segments/20140523142436]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: finished at 2014-05-23 14:25:06, elapsed: 00:00:03
LinkDb: starting at 2014-05-23 14:25:06
LinkDb: linkdb: localweb/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523135555
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523140148
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523140159
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523135939
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523142330
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523135546
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523142436
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523140007
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523142116
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523135609
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523140136
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523142238
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523135953
LinkDb: adding segment: file:/home/test/programs/nutch-1.2/localweb/segments/20140523142018
LinkDb: merging with existing linkdb: localweb/linkdb
LinkDb: finished at 2014-05-23 14:25:12, elapsed: 00:00:06
Deleting old indexes: localweb/indexes
Deleting old merged index: localweb/index
Indexer: starting at 2014-05-23 14:25:12
Indexer: finished at 2014-05-23 14:25:25, elapsed: 00:00:12
Dedup: starting at 2014-05-23 14:25:25
Dedup: adding indexes in: localweb/indexes
Dedup: finished at 2014-05-23 14:25:28, elapsed: 00:00:03
IndexMerger: starting at 2014-05-23 14:25:28
IndexMerger: merging indexes to: localweb/index
Adding file:/home/test/programs/nutch-1.2/localweb/indexes/part-00000
IndexMerger: finished at 2014-05-23 14:25:30, elapsed: 00:00:01
crawl finished: localweb

你可能感兴趣的:(nutch)

Java：爬虫框架 dingcho Java java 爬虫
一、ApacheNutch2【参考地址】Nutch是一个开源Java实现的搜索引擎。它提供了我们运行自己的搜索引擎所需的全部工具。包括全文搜索和Web爬虫。Nutch致力于让每个人能很容易,同时花费很少就可以配置世界一流的Web搜索引擎.为了完成这一宏伟的目标,Nutch必须能够做到:每个月取几十亿网页为这些网页维护一个索引对索引文件进行每秒上千次的搜索提供高质量的搜索结果简单来说Nutch支持分
Python爬虫实战 weixin_34007879 爬虫 json java
引言网络爬虫是抓取互联网信息的利器，成熟的开源爬虫框架主要集中于两种语言Java和Python。主流的开源爬虫框架包括：1.分布式爬虫框架：Nutch2.Java单机爬虫框架：Crawler4j,WebMagic,WebCollector、Heritrix3.python单机爬虫框架：scrapy、pyspiderNutch是专为搜索引擎设计的的分布式开源框架，上手难度高，开发复杂，基本无法满足快
深入浅出hdfs-hadoop基本介绍大数据之家 hdfs hadoop 大数据
一、Hadoop基本介绍hadoop最开始是起源于ApacheNutch项目，这个是由DougCutting开发的开源网络搜索引擎，这个项目刚开始的目标是为了更好的做搜索引擎，后来Google发表了三篇未来持续影响大数据领域的三架马车论文：GoogleFileSystem、BigTable、Mapreduce开始掀起来了大数据的浪潮，paper原文可以参考我的这篇文章CSDN。这三篇论文介绍了如何
Hadoop简介：开启大数据处理之门乌龙饼干 hadoop 大数据分布式
随着信息技术的飞速发展，数据呈现爆炸式增长，传统的数据处理方式已无法满足日益增长的数据需求。在此背景下，Hadoop作为一种分布式系统基础架构，应运而生，为大数据处理打开了新的大门。一、Hadoop的起源与概念Hadoop最初由DougCutting创建，作为ApacheLucene的子项目Nutch的一部分。随着项目的不断发展，Hadoop逐渐独立出来，成为Apache软件基金会下的一个开源项目
专为初学者设计：Nutch库Java下载器入门指南亿牛云爬虫专家 java 代理IP 爬虫代理 java 开发语言 Nutch 下载器爬虫代理代理IP 多线程
概述:Nutch是一款开源的Java爬虫框架，用于抓取、解析、提取和存储网页数据。基于Hadoop的分布式系统，Nutch支持大规模网络爬取，并提供各种插件，包括链接分析、语言检测和内容过滤等功能。本文旨在介绍如何使用Nutch库编写简单的Java下载器，即能从指定URL下载网页内容的程序。目标是帮助初学者了解Nutch库的基本用法，并展示如何通过代理IP技术和多线程技术提升下载效率。假设读者已安
在CentOS7上安装Hadoop分布式系统栗子艾李子 hadoop linux hdfs 分布式
项目背景：Hadoop原来是ApacheLucene下的一个子项目，它最初是从Nutch项目中分离出来的专门负责分布式存储以及分布式运算的项目。简单地说来，Hadoop是一个可以更容易开发和运行处理大规模数据的软件平台。Hadoop由分布式存储HDFS和分布式计算MapReduce两部分组成。HDFS是一个master/slave的结构，就通常的部署来说，在master上只运行一个Namenode
大数据技术之Hadoop入门一在远方的你等我
1.从Hadoop框架讨论大数据生态名字起源该项目的创建者，DougCutting解释Hadoop的得名：“这个名字是我孩子给一个棕黄色的大象玩具命名的项目起源Hadoop由ApacheSoftwareFoundation公司于2005年秋天作为Lucene的子项目Nutch的一部分正式引入。它受到最先由GoogleLab开发的Map/Reduce和GoogleFileSystem(GFS)的启发
openpyxl3.0官方文档（14）—— 甜甜圈图 Sinchard
甜甜圈图表与饼图类似，只是它们使用了一个环而不是一个圆，还可以绘制出若干系列的数据作为中心环。fromopenpyxlimportWorkbookfromopenpyxl.chartimport(DoughnutChart,Reference,Series,)fromopenpyxl.chart.seriesimportDataPointdata=[['Pie',2014,2015],['Plai
kafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转） weixin_34185320 运维操作系统系统架构
李克华云计算高级群:292870151195907286交流：Hadoop、NoSQL、分布式、lucene、solr、nutchkafka入门：简介、使用场景、设计原理、主要配置及集群搭建（转）问题导读：1.zookeeper在kafka的作用是什么？2.kafka中几乎不允许对消息进行“随机读写”的原因是什么？3.kafka集群consumer和producer状态信息是如何保存的？4.par
大数据之 Hadoop 小裕哥略帅大数据 hadoop java
hadoop主要解决：海量数据的存储和海量数据的分析计算hadoop发展历史Google是hadoop的思想之源（Google在大数据方面的三篇论文）2006年3月，Map-reduce和NutchDistributedFileSystem(NDFS)分别被纳入到Hadoop项目，Hadoop正式诞生。MapReduce对海量数据处理map函数进行数据的提取、排序，实现mapper，四个形参（输入
自己动手写搜索引擎系列【目录】 luyee2010 自己动手写搜索引擎自己动手写搜索引擎
第1章遍历搜索引擎技术11.130分钟实现的搜索引擎11.1.1准备工作环境（10分钟）11.1.2编写代码（15分钟）31.1.3发布运行（5分钟）51.2Google神话91.3体验搜索引擎91.4搜索语法101.5你也可以做搜索引擎131.6搜索引擎基本技术141.6.1网络蜘蛛141.6.2全文索引结构141.6.3Lucene全文检索引擎151.6.4Nutch网络搜索软件161.6.5
ElasticSearch（ES）——概述/API 平平无奇小码农笔记大数据 elasticsearch 数据库搜索引擎
文章目录一、ElasticSearch基础1.1简介1.2使用场景1.3ES与其他数据存储进行比较1.4ES的特点1.5Lucene、Nutch、ElasticSearch关系二、基本概念ES概念和MySQL关系对比三、安装ES3.1解压、改名3.2修改配置文件3.3教学环境启动优化分发3.4修改hadoop163、hadoop164的节点名3.5单台启动测试，解决问题四、安装kibana4.1解
asp html5 ajax,ASP.NET AJAX Chart (HTML5) - RadControls for Web Forms | Telerik UI for ASP.NET AJAX weixin_39942191 asp html5 ajax
AnyEssentialChartTypeQuicklyaddmeaningtodatawiththemostcommonlyusedASP.NETchartingtypes:PieorDonutcharttovisualizeeachpieceofdataaspartofawholeLineorAreatomonitortrendsBar,ColumnorRadartocomparesevera
安装关系型数据库MySQL和大数据处理框架Hadoop weixin_30621919 数据库嵌入式大数据
这个作业的要求来自于：https://edu.cnblogs.com/campus/gzcc/GZCC-16SE2/homework/3161。1.简述Hadoop平台的起源、发展历史与应用现状。列举发展过程中重要的事件、主要版本、主要厂商；国内外Hadoop应用的典型案例。（1）Hadoop的介绍：Hadoop最早起源于Nutch，Nutch的设计目标是构建一个大型的全网搜索引擎，包括网页抓取、
ChatGPT4 完成数据分析结构分析，动态饼图可视化阿里数据专家 ChatGPT实战案例 ChatGPT 数据分析信息可视化数据挖掘 excel 人工智能 AIGC chatgpt
对于数据分析中的结构占比分析，以下几种图表类型是比较常见和合适的：1.**饼图（PieChart）**：饼图是一种表现部分与整体关系的图表，各部分占整体的比例在图中以圆形的切片形式体现。它适用于表示不同类别之间的比较，以及每个类别占总数的百分比。2.**环图（DoughnutChart）**：环图是饼图的变种，有一个空心中心。它也是显示类别之间占比关系的一种有效的方式。3.**堆叠柱状图/堆叠条形
hadoop yuanjianqiang_0925 hadoop spark
hadoop主要解决：海量数据的存储和海量数据的分析计算hadoop发展历史Google是hadoop的思想之源（Google在大数据方面的三篇论文）2006年3月，Map-reduce和NutchDistributedFileSystem(NDFS)分别被纳入到Hadoop项目，Hadoop正式诞生。MapReduce对海量数据处理map函数进行数据的提取、排序，实现mapper，四个形参（
Ubuntu环境下Hadoop1.2.1, HBase0.94.25, nutch2.2.1各个配置文件一览 weixin_30491641 大数据 java runtime
/×××××××××××××××××××××××××××××××××××××××××/Author：xxx0624HomePage：http://www.cnblogs.com/xxx0624//×××××××××××××××××××××××××××××××××××××××××/Hadoop伪分布式配置过程：Hadoop：1.2.1Hbase：0.94.25nutch：2.2.1Java：1.8.
ElasticSearch详细教程-基础加实战工藤-新二实时数仓大数据实时项目 elasticsearch 实时大数据 spark
文章目录第1章ElasticSearch基础1.1简介1.2使用场景1.3ES与其他数据存储进行比较1.4ElasticSearch的特点1.4.1天然分片，天然集群1.4.2天然索引1.5Lucene、Nutch、ElasticSearch关系第2章ElasticSearch的安装2.1上传安装包2.2将ES解压到/opt/module目录下2.3在/opt/module目录下对ES重命名2.4
jvm命令和可视化工具调优 weixin_30834783 java 操作系统开发工具
李克华云计算高级群:292870151195907286交流：Hadoop、NoSQL、分布式、lucene、solr、nutch虚拟机：系统虚拟机程序虚拟机系统虚拟机有：VMWarevisureBox程序虚拟机：JVMJVM：1.类加载子系统（类加载器）2.方法区3.java堆4.直接内存5.java栈6.本地方法栈7.垃圾回收系统8.PC寄存器9.执行引擎堆：存储问题栈：程序运行方法去：辅助堆
linux服务器忘记ssh密码_【Linux】配置linux服务器之间ssh不用密码访问 weixin_40008033 linux服务器忘记ssh密码
如果想在A这太机器上可以不需要密码就ssh到B、C两台机器上，可以采用如下的方法：(1)在A机器上：ssh-keygen-trsaGeneratingpublic/privatersakeypair.Enterfileinwhichtosavethekey(/nutch/home/.ssh/id_rsa):不输入任何东西，直接回车Enterpassphrase(emptyfornopassphra
Hadoop分布式文件系统杀神lwz hadoop 大数据分布式
一、HadoopHadoop之父DougCuttingHadoop的发音[hædu:p]，Cutting儿子对玩具小象的昵称1、Hadoop发展简史2002年10月，DougCutting和MikeCafarella创建了开源网页爬虫项目Nutch。2003年10月，Google发表GoogleFileSystem论文。2004年7月，DougCutting和MikeCafarella在Nutch
java 爬虫框架nutch_网络爬虫（2）-- Java爬虫框架鲍鱼王 java 爬虫框架nutch
NutchNutch属于分布式爬虫，爬虫使用分布式，主要是解决两个问题：1)海量URL管理；2)网速。如果要做搜索引擎，Nutch1.x是一个非常好的选择。Nutch1.x和solr或者es配合，就可以构成一套非常强大的搜索引擎，否则尽量不要选择Nutch作为爬虫。用Nutch进行爬虫的二次开发，爬虫的编写和调试所需的时间，往往是单机爬虫所需的十倍时间不止。HeritrixHeritrix是个“A
nutch爬取网站数据详细步骤 Echoooo_o
环境：hadoop2.7.7+hbase0.98+nutch2.3+solr4.9大致步骤思想：hadoop提供底层数据存储hbase在其之上建立非关系型数据库nutch将爬的数据存到hbase上并建立索引到solr展示首先采用简单命令：#$1$2...$n表示命令后跟的第n个参数#存放待注入种子的路径SEEDDIR="$1"#存放爬取数据（URL状态信息、爬取数据、解析数据）文件夹的路径CRAW
nutch，hbase记录 feihuadao
hbase表操作优化http://blog.pureisle.net/archives/1930.htmlHow-to:UseHBaseBulkLoading,andWhyhttp://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/nutch2.2分析http://blog.csdn.net/itufo/a
Hadoop 凤舞飘伶 Go hadoop
Hadoop是Google的集群系统的开源实现，Google集群系统:GFS(GoogleFileSystem)、MapReduce、BigTable。Hadoop主要由HDFS(HadoopDistributedFileSystemHadoop分布式文件系统)、MapReduce和HBase组成Hadoop的初衷是为解决Nutch的海量数据爬取和存储的需要。Hadoop于2005年秋天作为Luc
Hadoop之父：Doug Cutting Mr_Elliot
hadoop生活中，可能所有人都间接用过他的作品，他是Lucene、Nutch、Hadoop等项目的发起人。是他，把高深莫测的搜索技术形成产品，贡献给普罗大众；还是他，打造了目前在云计算和大数据领域里如日中天的Hadoop。他是某种意义上的盗火者，他就是DougCutting。DougCutting从实习生做起1985年，Cutting毕业于美国斯坦福大学。他并不是一开始就决心投身IT行业的，在大
Hadoop-2.6.5完整安装配置过程 syp_net 系统开发 hadoop mapreduce 搜索引擎
记录Hadoop-2.6.5完整安装配置过程一、Hadoop是什么？二、Hadoop-2.6.5安装配置1.修改主机名2.下载并解压JDK3.配置环境变量4.修改Hadoop中5个主要配置文件5.启动Hadoop6.HadoopWeb端口测试三、总结一、Hadoop是什么？Hadoop系统最初的源头来自于ApacheLucene项目下的搜索引擎子项目Nutch，该项目的负责人是DougCuttin
Hadoop之HDFS简介数新网络 hadoop 大数据 hdfs
前言Hadoop是由Apache基金会开发的分布式系统基础框架，主要解决海量数据存储和海量数据分析问题。Hadoop起源于ApacheNutch项目，起始于2002年，在2006年被正式命名为Hadoop。Hadoop有3大核心组件，分别是HDFS、MapReduce和YARN，本次我们重点介绍HDFS。一、HDFS简介HDFS全称HadoopDistributedFileSystem，是一个分布
hadoop原理和细节 truezqx
一、Hadoop概述Hadoop是Google的集群系统开源实现Google的集群系统：GFS、MapReduce、BigTableHadoop的集群系统：HDFS、MapReduce、HBaseHadoop设计的初衷是为了解决Nutch的海量数据存储和处理的需求，可以解决大数据场景下的数据存储和处理的问题。传统数据：GB、TB级别的数据、数据增长不快、主要为结构化的数据、统计和报表大数据：TB、
听阿里P7工程师只分七步讲解HDFS搭建 Python大数据工程师
前言HADOOP产生背景（1）HADOOP最早起源于Nutch。Nutch的设计目标是构建一个大型的全网搜索引擎，包括网页抓取、索引、查询等功能，但随着抓取网页数量的增加，遇到了严重的可扩展性问题——如何解决数十亿网页的存储和索引问题。（2）2003年、2004年谷歌发表的两篇论文为该问题提供了可行的解决方案。——分布式文件系统（GFS），可用于处理海量网页的存储——分布式计算框架MAPREDUC
如何用ruby来写hadoop的mapreduce并生成jar包 wudixiaotie mapreduce
ruby来写hadoop的mapreduce，我用的方法是rubydoop。怎么配置环境呢： 1.安装rvm：不说了网上有 2.安装ruby：由于我以前是做ruby的，所以习惯性的先安装了ruby，起码调试起来比jruby快多了。 3.安装jruby： rvm install jruby然后等待安
java编程思想 -- 访问控制权限百合不是茶 java 访问控制权限单例模式
访问权限是java中一个比较中要的知识点,它规定者什么方法可以访问,什么不可以访问一:包访问权限; 自定义包: package com.wj.control; //包 public class Demo { //定义一个无参的方法 public void DemoPackage(){ System.out.println("调用
[生物与医学]请审慎食用小龙虾 comsci 生物
现在的餐馆里面出售的小龙虾,有一些是在野外捕捉的,这些小龙虾身体里面可能带有某些病毒和细菌,人食用以后可能会导致一些疾病,严重的甚至会死亡..... 所以,参加聚餐的时候,最好不要点小龙虾...就吃养殖的猪肉,牛肉,羊肉和鱼,等动物蛋白质
org.apache.jasper.JasperException: Unable to compile class for JSP: 商人shang maven 2.2 jdk1.8
环境： jdk1.8 maven tomcat7-maven-plugin 2.0 原因： tomcat7-maven-plugin 2.0 不知吃 jdk 1.8，换成 tomcat7-maven-plugin 2.2就行，即 <plugin>
你的垃圾你处理掉了吗?GC oloz GC
前序:本人菜鸟，此文研究学习来自网络，各位牛牛多指教　 1.垃圾收集算法的核心思想　　Java语言建立了垃圾收集机制，用以跟踪正在使用的对象和发现并回收不再使用(引用)的对象。该机制可以有效防范动态内存分配中可能发生的两个危险：因内存垃圾过多而引发的内存耗尽，以及不恰当的内存释放所造成的内存非法引用。　　垃圾收集算法的核心思想是：对虚拟机可用内存空间，即堆空间中的对象进行识别
shiro 和 SESSSION 杨白白 shiro
shiro 在web项目里默认使用的是web容器提供的session，也就是说shiro使用的session是web容器产生的，并不是自己产生的，在用于非web环境时可用其他来源代替。在web工程启动的时候它就和容器绑定在了一起，这是通过web.xml里面的shiroFilter实现的。通过session.getSession()方法会在浏览器cokkice产生JESSIONID，当关闭浏览器，此
移动互联网终端淘宝客如何实现盈利小桔子移動客戶端淘客淘寶App
2012年淘宝联盟平台为站长和淘宝客带来的分成收入突破30亿元，同比增长100%。而来自移动端的分成达1亿元，其中美丽说、蘑菇街、果库、口袋购物等App运营商分成近5000万元。可以看出，虽然目前阶段PC端对于淘客而言仍旧是盈利的大头，但移动端已经呈现出爆发之势。而且这个势头将随着智能终端(手机，平板)的加速普及而更加迅猛
wordpress小工具制作 aichenglong wordpress 小工具
wordpress 使用侧边栏的小工具，很方便调整页面结构小工具的制作过程 1 在自己的主题文件中新建一个文件夹(如widget)，在文件夹中创建一个php(AWP_posts-category.php) 小工具是一个类,想侧边栏一样，还得使用代码注册，他才可以再后台使用，基本的代码一层不变 <?php class AWP_Post_Category extends WP_Wi
JS微信分享 AILIKES js
// 所有功能必须包含在 WeixinApi.ready 中进行 WeixinApi.ready(function(Api) { // 微信分享的数据 var wxData = { &nb
封装探讨百合不是茶 JAVA面向对象封装
//封装属性方法将某些东西包装在一起，通过创建对象或使用静态的方法来调用，称为封装；封装其实就是有选择性地公开或隐藏某些信息，它解决了数据的安全性问题，增加代码的可读性和可维护性在 Aname类中申明三个属性，将其封装在一个类中：通过对象来调用例如 1： //属性将其设为私有姓名 name 可以公开
jquery radio/checkbox change事件不能触发的问题 bijian1013 JavaScript jquery
我想让radio来控制当前我选择的是机动车还是特种车，如下所示： <html> <head> <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"><
AngularJS中安全性措施 bijian1013 JavaScript AngularJS 安全性 XSRF JSON漏洞
在使用web应用中，安全性是应该首要考虑的一个问题。AngularJS提供了一些辅助机制，用来防护来自两个常见攻击方向的网络攻击。一.JSON漏洞当使用一个GET请求获取JSON数组信息的时候（尤其是当这一信息非常敏感，
[Maven学习笔记九]Maven发布web项目 bit1129 maven
基于Maven的web项目的标准项目结构 user-project user-core user-service user-web src
【Hive七】Hive用户自定义聚合函数(UDAF) bit1129 hive
用户自定义聚合函数，用户提供的多个入参通过聚合计算(求和、求最大值、求最小值)得到一个聚合计算结果的函数。问题：UDF也可以提供输入多个参数然后输出一个结果的运算，比如加法运算add(3，5)，add这个UDF需要实现UDF的evaluate方法,那么UDF和UDAF的实质分别究竟是什么？ Double evaluate(Double a, Double b)
通过 nginx-lua 给 Nginx 增加 OAuth 支持 ronin47
前言：我们使用Nginx的Lua中间件建立了OAuth2认证和授权层。如果你也有此打算，阅读下面的文档，实现自动化并获得收益。SeatGeek 在过去几年中取得了发展，我们已经积累了不少针对各种任务的不同管理接口。我们通常为新的展示需求创建新模块，比如我们自己的博客、图表等。我们还定期开发内部工具来处理诸如部署、可视化操作及事件处理等事务。在处理这些事务中，我们使用了几个不同的接口来认证： &n
利用tomcat-redis-session-manager做session同步时自定义类对象属性保存不上的解决方法 bsr1983 session
在利用tomcat-redis-session-manager做session同步时，遇到了在session保存一个自定义对象时，修改该对象中的某个属性，session未进行序列化，属性没有被存储到redis中。在 tomcat-redis-session-manager的github上有如下说明： Session Change Tracking As noted in the &qu
《代码大全》表驱动法-Table Driven Approach-1 bylijinnan java 算法
关于Table Driven Approach的一篇非常好的文章： http://www.codeproject.com/Articles/42732/Table-driven-Approach package com.ljn.base; import java.util.Random; public class TableDriven { public
Sybase封锁原理 chicony Sybase
昨天在操作Sybase IQ12.7时意外操作造成了数据库表锁定，不能删除被锁定表数据也不能往其中写入数据。由于着急往该表抽入数据，因此立马着手解决该表的解锁问题。无奈此前没有接触过Sybase IQ12.7这套数据库产品，加之当时已属于下班时间无法求助于支持人员支持，因此只有借助搜索引擎强大的
java异常处理机制 CrazyMizzz java
java异常关键字有以下几个，分别为 try catch final throw throws 他们的定义分别为 try： Opening exception-handling statement. catch： Captures the exception. finally： Runs its code before terminating
hive 数据插入DML语法汇总 daizj hive DML 数据插入
Hive的数据插入DML语法汇总1、Loading files into tables语法：1) LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]解释：1)、上面命令执行环境为hive客户端环境下： hive>l
工厂设计模式 dcj3sjt126com 设计模式
使用设计模式是促进最佳实践和良好设计的好办法。设计模式可以提供针对常见的编程问题的灵活的解决方案。工厂模式工厂模式（Factory）允许你在代码执行时实例化对象。它之所以被称为工厂模式是因为它负责“生产”对象。工厂方法的参数是你要生成的对象对应的类名称。 Example #1 调用工厂方法（带参数） <?phpclass Example{
mysql字符串查找函数 dcj3sjt126com mysql
FIND_IN_SET(str,strlist) 假如字符串str 在由N 子链组成的字符串列表strlist 中，则返回值的范围在1到 N 之间。一个字符串列表就是一个由一些被‘,’符号分开的自链组成的字符串。如果第一个参数是一个常数字符串，而第二个是type SET列，则 FIND_IN_SET() 函数被优化，使用比特计算。如果str不在strlist 或st
jvm内存管理 easterfly jvm
一、JVM堆内存的划分分为年轻代和年老代。年轻代又分为三部分：一个eden,两个survivor。工作过程是这样的：e区空间满了后，执行minor gc，存活下来的对象放入s0, 对s0仍会进行minor gc，存活下来的的对象放入s1中，对s1同样执行minor gc，依旧存活的对象就放入年老代中；年老代满了之后会执行major gc，这个是stop the word模式，执行
CentOS-6.3安装配置JDK-8 gengzg centos
JAVA_HOME=/usr/java/jdk1.8.0_45 JRE_HOME=/usr/java/jdk1.8.0_45/jre PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib export JAVA_HOME
【转】关于web路径的获取方法 huangyc1210 Web 路径
假定你的web application 名称为news,你在浏览器中输入请求路径： http://localhost:8080/news/main/list.jsp 则执行下面向行代码后打印出如下结果： 1、 System.out.println(request.getContextPath()); //可返回站点的根路径。也就是项
php里获取第一个中文首字母并排序远去的渡口数据结构 PHP
很久没来更新博客了，还是觉得工作需要多总结的好。今天来更新一个自己认为比较有成就的问题吧。最近在做储值结算，需求里结算首页需要按门店的首字母A-Z排序。我的数据结构原本是这样的： Array ( [0] => Array ( [sid] => 2885842 [recetcstoredpay] =&g
java内部类 hm4123660 java 内部类匿名内部类成员内部类方法内部类
　在Java中，可以将一个类定义在另一个类里面或者一个方法里面，这样的类称为内部类。内部类仍然是一个独立的类，在编译之后内部类会被编译成独立的.class文件，但是前面冠以外部类的类名和$符号。内部类可以间接解决多继承问题,可以使用内部类继承一个类，外部类继承一个类，实现多继承。 &nb
Caused by: java.lang.IncompatibleClassChangeError: class org.hibernate.cfg.Exten zhb8015
maven pom.xml关于hibernate的配置和异常信息如下，查了好多资料，问题还是没有解决。只知道是包冲突，就是不知道是哪个包....遇到这个问题的分享下是怎么解决的。。 maven pom: <dependency> <groupId>org.hibernate</groupId> <ar
Spark 性能相关参数配置详解－任务调度篇 Stark_Summer spark cache cpu 任务调度 yarn
随着Spark的逐渐成熟完善, 越来越多的可配置参数被添加到Spark中来, 本文试图通过阐述这其中部分参数的工作原理和配置思路, 和大家一起探讨一下如何根据实际场合对Spark进行配置优化。由于篇幅较长，所以在这里分篇组织，如果要看最新完整的网页版内容，可以戳这里：http://spark-config.readthedocs.org/，主要是便
css3滤镜 wangkeheng html css
经常看到一些网站的底部有一些灰色的图标，鼠标移入的时候会变亮，开始以为是js操作src或者bg呢，搜索了一下，发现了一个更好的方法：通过css3的滤镜方法。 html代码： <a href='' class='icon'><img src='utv.jpg' /></a> css代码： .icon{-webkit-filter: graysc