Heritrix学习及遇到问题汇总(四)

1.
message:Value of illegal type: 'org.archive.crawler.settings.ModuleType', 'org.archive.crawler.framework.Frontier' was expected.: Value of illegal type: 'org.archive.crawler.settings.ModuleType', 'org.archive.crawler.framework.Frontier' was expected.
Exception:No associated exception.

2.
message:On crawl: question Unable to setup crawl modules
exception:java.lang.ClassCastException: org.archive.crawler.settings.ModuleType cannot be cast to org.archive.crawler.framework.Frontier
Stacktrace: java.lang.ClassCastException: org.archive.crawler.settings.ModuleType cannot be cast to org.archive.crawler.framework.Frontier
at org.archive.crawler.framework.CrawlController.setupCrawlModules(CrawlController.java:675)
at org.archive.crawler.framework.CrawlController.initialize(CrawlController.java:381)
at org.archive.crawler.admin.CrawlJob.setupForCrawlStart(CrawlJob.java:853)
at org.archive.crawler.admin.CrawlJobHandler.startNextJobInternal(CrawlJobHandler.java:1144)
at org.archive.crawler.admin.CrawlJobHandler$3.run(CrawlJobHandler.java:1127)
at java.lang.Thread.run(Thread.java:619)

3.
message:Wrong document type 'crawl-order' in 'file:/c:/heritrix/jobs/question-20141005032127804/order.xml', line: 1, column: 160
exception:No associated exception.

解决方案:一般都是由于处理器链没有正确设置而导致
比如,在应该是Prefetcher的地方,设置成了Writer。这样就会导致错误
请严格按照以下方式来设置:
1. frontier
org.archive.crawler.frontier.BdbFrontier
2. scope
org.archive.crawler.scope.BroadScope
3. Prefetcher
org.archive.crawler.prefetch.Preselector
org.archive.crawler.prefetch.PreconditionEnforcer
4. Fetcher
org.archive.crawler.fetcher.FetchDNS
org.archive.crawler.fetcher.FetchHTTP
5. Extractor
org.archive.crawler.extractor.ExtractorHTTP
org.archive.crawler.extractor.ExtractorHTML
(这里可以按自己的需要多添几个,比如ExtractorSWF、ExtractorJS什么的,但是前两个是必不可少的)
6. Writer
可以是MirrorWriter或ARCWriter,一般建议使用MirrorWriter
7. PostProcessor
org.archive.crawler.postprocessor.CrawlStateUpdater
org.archive.crawler.postprocessor.LinksScoper
org.archive.crawler.postprocessor.FrontierScheduler
(FrontierScheduler可以自行扩展,按书上的方法)

你可能感兴趣的:(Heritrix)