Nutch2 + Solr 6: solrdedup causes ClassCastException

阅读更多

Error running:

  /mnt/nutch/nutch/runtime/local/bin/nutch solrdedup -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D mapred.reduce.tasks.speculative.execution=false -D mapred.map.tasks.speculative.execution=false -D mapred.compress.map.output=true http://localhost:8983/solr/nutch

Failed with exit value 1.

 

hadoop.log :

java.lang.Exception: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String

        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)

        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)

Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String

        at org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrRecordReader.nextKeyValue(SolrDeleteDuplicates.java:233)

        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)

        at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)

        at 

 

这个报错在solr5,6都存在。参考的解决方案见这里:

http://lucene.472066.n3.nabble.com/Nutch-2-Solr-5-solrdedup-causes-ClassCastException-td4301149.html#a4302739

 

1)将solr-6.6.0/server/solr/configsets目录下的data_driven_schema_configs复制一份到原目录下,并重命名为nutch

2)$NUTCH_HOME/conf下的schema.xml复制到server/solr/configsets/nutch/conf下。

3)将server/solr/configsets/nutch/conf/schema.xml文件中的enablePositionIncrements删掉。

4)执行命令创建core:

 

solr create -c nutch -d nutch

   解释:[-c name] [-d confdir]

 

   这里的nutch分别是core的名字和上面指定的conf的名字

成功了会看到这样的输入:

Zhuos-MacBook-Pro:solr-6.6.0 jo$ solr create -c nutch -d nutch

 

Copying configuration to new core instance directory:

/Users/jo/soft/solr-5.5.4/server/solr/nutch

 

Creating new core 'nutch' using command:

http://localhost:8983/solr/admin/cores?action=CREATE&name=nutch&instanceDir=nutch

 

{

  "responseHeader":{

    "status":0,

    "QTime":107},

  "core":"nutch"}

 

 

 

如果报错:ERROR: Error CREATEing SolrCore 'nutch': Unable to create core [nutch] Caused by: fieldType 'tdates' not found in the schema

这里可以是tdates或者其他的..可以对比下server/solr/configsets/nutch/conf目录下的schema.xml和managed-schema两个文件。tdates应该是在managed-schema文件中用到了,却没有在schema.xml中定义。因此将managed-schema中tdates的定义拷贝到schema.xml中。例如下面这段我是从managed-schema中找到的,然后拷贝到了schema.xml中:

    
    
    
    

 

这下应该没什么问题了。

 

 

你可能感兴趣的:(Nutch2 + Solr 6: solrdedup causes ClassCastException)