Nutch2.x+Hadoop 2.5.2+Hbase0.94.26(续2)

转载地址：  http://www.cnblogs.com/mactech/p/4239163.html

1.执行bin/nutch generate -topN 5 -crawlId tieba的时候，出现以下错误

java.lang.Exception: java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to org.apache.gora.persistency.Persistent

        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)

        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)

Caused by: java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to org.apache.gora.persistency.Persistent

        at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(PersistentDeserializer.java:71)

        at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(PersistentDeserializer.java:35)

        at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146)

        at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121)

        at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302)

        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170)

        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)

        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)

        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

初步怀疑是avrò的版本不匹配，把avrò从1.7.7降级到1.7.6问题依然存在。然后发现执行nutch的时候，classpath里面调用的都是hadoop 2.5.2的jar，而在hadoop-2.5.2/share/hadoop/common/lib/下，avro的版本是1.7.4，把1.7.7版本替换进去，问题解决

2.执行bin/nutch fetch 1421804965-1372033824 -crawlId tieba -threads 50，其中1421804965-1372033824为在hbase shell中执行 get 'tieba_webpage','com.baidu.tieba:http/' 所得f:bid timestamp=1421804970851, value=1421804965-1372033824

此时报错，No agents listed in 'http.agent.name' property

修改nutch-default.properties中的 <name>http.agent.name</name>部分，添加任意字符串