nutch-2.1、mysql整合
这次主要介绍下nutch2.1和mysql和elasticsearch的整合，是在单机上运行，并不是分布式部署。
1、下载nutch2.1
nutch下载地址：http://labs.mop.com/apache-mirror/nutch/2.1/apache-nutch-2.1-src.tar.gz
2、配置nutch使用mysql作为数据存储，修改nutch根目录/ivy/ivy.xml文件
将这行的注释取消<dependency org=”mysql” name=”mysql-connector-java” rev=”5.1.18″ conf=”*->default”/>
修改nutch根目录/conf/gora.properties文件，把默认存储配置改成如下
# MySQL properties #
###############################
gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://host:3306/nutch? useUnicode=true&characterEncoding=utf8&autoReconnect=true&zeroDateTimeBehavior=convertToNull
gora.sqlstore.jdbc.user=root
gora.sqlstore.jdbc.password=root
|
3、修改conf的nutch-site.xml
<?
xml
version
=
"1.0"
?>
<?
xml-stylesheet
type
=
"text/xsl"
href
=
"configuration.xsl"
?>
<
configuration
>
<
property
>
<
name
>http.agent.name</
name
>
<
value
>My Spider</
value
>
</
property
>
<
property
>
<
name
>http.accept.language</
name
>
<
value
>ja-jp,zh-cn,en-us,en-gb,en;q=0.7,*;q=0.3</
value
>
</
property
>
<
property
>
<
name
>parser.character.encoding.default</
name
>
<
value
>utf-8</
value
>
<
description
>The character encoding to fall back to when no other information
is available</
description
>
</
property
>
<
property
>
<
name
>storage.data.store.class</
name
>
<
value
>org.apache.gora.sql.store.SqlStore</
value
>
</
property
>
<
property
>
<
name
>plugin.includes</
name
>
<
value
>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|urlnormalizer-(pass|regex|basic)|scoring-opic</
value
>
</
property
>
</
configuration
>
|
4、使用ant编译源码。
nutch根目录下输入ant
5、设置待抓取的网站
cd nutch根目录/runtime/local
创建urls目录
mkdir -p urls
创建初始爬取网站列表
echo 'http://www.iie.ac.cn/' > urls/seed.txt
6、创建数据库和表
CREATE
DATABASE
nutch
DEFAULT
CHARACTER
SET
utf8
DEFAULT
COLLATE
utf8_general_ci;
CREATE
TABLE
`webpage` (`id`
varchar
(767)
CHARACTER
SET
latin1
NOT
NULL
,
`headers` blob,
`text` mediumtext
DEFAULT
NULL
,
`status`
int
(11)
DEFAULT
NULL
,
`markers` blob,
`parseStatus` blob,
`modifiedTime`
bigint
(20)
DEFAULT
NULL
,
`score`
float
DEFAULT
NULL
,
`typ`
varchar
(32)
CHARACTER
SET
latin1
DEFAULT
NULL
,
`baseUrl`
varchar
(512)
CHARACTER
SET
latin1
DEFAULT
NULL
,
`content` mediumblob,
`title`
varchar
(2048)
DEFAULT
NULL
,
`reprUrl`
varchar
(512)
CHARACTER
SET
latin1
DEFAULT
NULL
,
`fetchInterval`
int
(11)
DEFAULT
NULL
,
`prevFetchTime`
bigint
(20)
DEFAULT
NULL
,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime`
bigint
(20)
DEFAULT
NULL
,
`retriesSinceFetch`
int
(11)
DEFAULT
NULL
,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY
KEY
(`id`)
) ENGINE=InnoDB
DEFAULT
CHARSET=utf8;
|
千万要注意：typ不能写成type，否则会报下面的错误：
2016-03-31 04:21:06,234 INFO fetcher.FetcherJob - -activeThreads=0
2016-03-31 04:21:06,281 WARN mapred.FileOutputCommitter - Output path is null in cleanup
2016-03-31 04:21:06,282 WARN mapred.LocalJobRunner - job_local_0003
java.io.IOException: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:340)
at org.apache.gora.sql.store.SqlStore.close(SqlStore.java:185)
at org.apache.gora.mapreduce.GoraRecordWriter.close(GoraRecordWriter.java:55)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:579)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)
Caused by: java.sql.BatchUpdateException: Unknown column 'typ' in 'field list'
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2016)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1452)
at org.apache.gora.sql.store.SqlStore.flush(SqlStore.java:328)
... 6 more
7、执行爬行操作：
./nutch
crawl ../urls -depth 3
|
执行完在mysql中即可以查看到爬虫抓取的内容