scrapy 中文的资料挺少的，写文章记录一下，以爬cnbeta新闻为例子，
抓取cnbeta的新闻标题+链接。
1.新建scrapy项目
scrapy startproject cnbeta
目录结构：
|
1
2
3
4
5
6
7
8
9
|
cnbeta
/
├──
cnbeta
│
├──
__init__
.
py
│
├──
items
.
py
│
├──
pipelines
.
py
│
├──
settings
.
py
│
└──
spiders
│
└──
__init__
.
py
└──
scrapy
.
cfg
|
2.定义数据结构
编辑cnbeta/items.py
|
1
2
3
4
5
6
7
8
9
10
11
12
|
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
from
scrapy
.
item
import
Item
,
Field
class
CnbetaItem
(
Item
)
:
# define the fields for your item here like:
# name = Field()
title
=
Field
(
)
url
=
Field
(
)
|
定义了两个字段，分别存储标题和链接
3.编写spider（爬虫）
编辑cnbeta/spiders/cb.py
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
from
scrapy
.
contrib
.
spiders
import
CrawlSpider
,
Rule
from
scrapy
.
contrib
.
linkextractors
.
sgml
import
SgmlLinkExtractor
from
scrapy
.
selector
import
Selector
from
cnbeta
.
items
import
CnbetaItem
class
CBSpider
(
CrawlSpider
)
:
name
=
'cnbeta'
allowed_domains
=
[
'cnbeta.com'
]
start_urls
=
[
'http://www.cnbeta.com'
]
rules
=
(
Rule
(
SgmlLinkExtractor
(
allow
=
(
'/articles/.*\.htm'
,
)
)
,
callback
=
'parse_page'
,
follow
=
True
)
,
)
def
parse_page
(
self
,
response
)
:
item
=
CnbetaItem
(
)
sel
=
Selector
(
response
)
item
[
'title'
]
=
sel
.
xpath
(
'//title/text()'
)
.
extract
(
)
item
[
'url'
]
=
response
.
url
return
item
|
rules指定了含有/articles/.*\.htm的链接都会被匹配.
4.运行爬虫
|
1
|
scrapy
crawl
cnbeta
-
o
result
.json
-
t
json
|
将结果输出到result.json -t json指定文件格式为json
5.结果
[
{
"url"
:
"http://www.cnbeta.com/articles/268661.htm"
,
"title"
:
[
"\u53ea\u97001\u5143\u4e0d\u62a2\u624d\u75af\uff01\u5c0f\u5ea6Wi-Fi\u5957\u88c5\u9707\u64bc\u4ef716\u65e5\u5f00\u62a2_Baidu \u767e\u5ea6_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268872.htm"
,
"title"
:
[
"\u8c37\u6b4c\u667a\u80fd\u5bb6\u5c45\u8ba1\u5212\u6216\u72af\u7684\u9519\uff1a\u5c01\u95edNest_Google / \u8c37\u6b4c_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268865.htm"
,
"title"
:
[
"\u4e2d\u56fd\u624b\u673a\u7f51\u6c11\u89c4\u6a21\u8fbe5\u4ebf \u5e74\u589e\u957f8009\u4e07\u4eba_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268869.htm"
,
"title"
:
[
"\u524d\u82f9\u679c\u9ad8\u7ea7\u526f\u603b\u88c1\u4e3a\u5927\u5b66\u4ee3\u8a00\uff1f\u539f\u662f\u56fe\u7247\u88ab\u76d7\u7528_cnBeta \u4eba\u7269_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268866.htm"
,
"title"
:
[
"\u6bd4\u7279\u5e01\u4ea4\u6613\u5e73\u53f0\u906d\u9047\u751f\u5b58\u5371\u673a_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268870.htm"
,
"title"
:
[
"\u76db\u5927\u6e38\u620f\u4f20\u5947\u88ab\u4fb5\u6743\u6848\u65b0\u8fdb\u5c55\uff1a\u8ffd\u52a0\u56db\u540d\u88ab\u544a_cnBeta \u89c6\u70b9\u89c2\u5bdf_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268867.htm"
,
"title"
:
[
"\u91d1\u878d\u65f6\u62a5\uff1a\u4e2d\u56fd\u8d2b\u56f0\u5730\u533a\u7f51\u8d2d\u589e\u901f\u8d85\u53d1\u8fbe\u5730\u533a_\u7535\u5b50\u5546\u52a1 - B2C / B2B_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268868.htm"
,
"title"
:
[
"\u79fb\u52a84G\u7248iPhone 5s/5c\u4e0a\u624b\uff1a\u901f\u5ea6\u6539\u53d8\u4f53\u9a8c_Apple iPhone_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268871.htm"
,
"title"
:
[
"[\u7ec4\u56fe]\u7247\u573a\u63a2\u79d8\u544a\u8bc9\u4f60\u7d22\u5c3c4K\u5f71\u7247\u662f\u600e\u4e48\u70bc\u6210\u7684_SONY \u7d22\u5c3c_cnBeta.COM"
]
}
,
{
"url"
:
"http://www.cnbeta.com/articles/268875.htm"
,
"title"
:
[
"USB 3.0\u548c\u534
源码下载：
cnbeta_spider.tar