nutch-2.1、mysql整合

这次主要介绍下nutch2.1和mysql和elasticsearch的整合,是在单机上运行,并不是分布式部署。
1、下载nutch2.1
    nutch下载地址:http://labs.mop.com/apache-mirror/nutch/2.1/apache-nutch-2.1-src.tar.gz
    
2、配置nutch使用mysql作为数据存储,修改nutch根目录/ivy/ivy.xml文件
    将这行的注释取消<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>
    修改nutch根目录/conf/gora.properties文件,把默认存储配置改成如下 
# MySQL properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://host:3306/nutch? useUnicode=true&characterEncoding=utf8&autoReconnect=true&zeroDateTimeBehavior=convertToNull
gora.sqlstore.jdbc.user=root
gora.sqlstore.jdbc.password=root
3、修改conf的nutch-site.xml
<? xml version = "1.0" ?>
<? xml-stylesheet type = "text/xsl" href = "configuration.xsl" ?>
 
< configuration >
< property >
< name >http.agent.name</ name >
< value >My Spider</ value >
</ property >
 
< property >
< name >http.accept.language</ name >
< value >ja-jp,zh-cn,en-us,en-gb,en;q=0.7,*;q=0.3</ value >
</ property >
 
< property >
< name >parser.character.encoding.default</ name >
< value >utf-8</ value >
< description >The character encoding to fall back to when no other information
is available</ description >
</ property >
 
< property >
< name >storage.data.store.class</ name >
< value >org.apache.gora.sql.store.SqlStore</ value >
</ property >
 
< property >
< name >plugin.includes</ name >
< value >protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</ value >
</ property >
 
</ configuration >
4、使用ant编译源码。
     nutch根目录下输入ant
5、设置待抓取的网站
    cd nutch根目录/runtime/local 
    创建urls目录
    mkdir -p urls 
    创建初始爬取网站列表
    echo 'http://www.iie.ac.cn/' > urls/seed.txt
 
6、创建数据库和表
CREATE DATABASE nutch DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
 
CREATE TABLE `webpage` (`id` varchar (767) CHARACTER SET latin1 NOT NULL ,
`headers` blob,
`text` mediumtext DEFAULT NULL ,
`status` int (11) DEFAULT NULL ,
`markers` blob,
`parseStatus` blob,
`modifiedTime` bigint (20) DEFAULT NULL ,
`score` float DEFAULT NULL ,
`typ` varchar (32) CHARACTER SET latin1 DEFAULT NULL ,
`baseUrl` varchar (512) CHARACTER SET latin1 DEFAULT NULL ,
`content` mediumblob,
`title` varchar (2048) DEFAULT NULL ,
`reprUrl` varchar (512) CHARACTER SET latin1 DEFAULT NULL ,
`fetchInterval` int (11) DEFAULT NULL ,
`prevFetchTime` bigint (20) DEFAULT NULL ,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime` bigint (20) DEFAULT NULL ,
`retriesSinceFetch` int (11) DEFAULT NULL ,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 千万要注意:typ不能写成type,否则会报下面的错误:
2016-03-31 04:21:06,234 INFO  fetcher.FetcherJob - -activeThreads=0
2016-03-31 04:21:06,281 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2016-03-31 04:21:06,282 WARN  mapred.LocalJobRunner - job_local_0003
java.io.IOException: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
        at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
        at org.apache.gora.sql.store.SqlStore.close(SqlStore.java:185)
        at org.apache.gora.mapreduce.GoraRecordWriter.close(GoraRecordWriter.java:55)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:579)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
Caused by: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
        at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2016)
        at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1452)
        at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:328)
        ... 6 more


7、执行爬行操作:
./nutch crawl ../urls -depth 3
执行完在mysql中即可以查看到爬虫抓取的内容

你可能感兴趣的:(nutch-2.1、mysql整合)