Python爬虫系列之----Scrapy(四)一个简单的示例
一、创建一个简单的项目
注:以下使用的python3
在使用Scrapy之前先要创建一个Scrapy项目,可以通过startproject命令来实现,首先在CMD中进入用来储存新建爬虫项目的文件夹,比如我们要在"G:\Scrapy_work"目录中创建Scrapy爬虫,可以在CMD中进入该文件夹,然后执行命令:
scrapy startproject myfendo
然后会在该目录下多出个myfendo目录
目录结构大概是这样的:
我们也可以在创建项目的时候加上一些参数进行控制,我们可以通过scrapy startproject -h调出start-project的帮助信息:
二、定义Item
这里以:dmoz.org网站进行测试,Dmoz(The Open Directory Project/ODP)是一个人工编辑管理的目录集合，为 搜索引擎提供结果或数据。
目前这个网站已经关闭。。。
Item 是保存爬取到的数据的容器；其使用方法和python字典类似， 并且提供了额外保护机制来避免拼写错误导致的未定义字段错误。
首先根据需要从dmoz.org获取到的数据对item进行建模。 我们需要从dmoz.org中获取名字,url,以及网站的描述。 对此，在item中定义相应的字段。编辑 myfendo 目录中的 items.py 文件:
import scrapy
class MyfendoItem(scrapy.Item):
title=scrapy.Field()
link=scrapy.Field()
desc=scrapy.Field()三、编写第一个爬虫(Spider)
Spider是用户编写用于从单个网站(或者一些网站)爬取数据的类。
其包含了一个用于下载的初始URL，如何跟进网页中的链接以及如何分析页面中的内容， 提取生成 item 的方法。
为了创建一个Spider，您必须继承 scrapy.Spider 类， 且定义以下三个属性:
● name: 用于区别Spider。 该名字必须是唯一的，您不可以为不同的Spider设定相同的名字。
● start_urls: 包含了Spider在启动时进行爬取的url列表。 因此，第一个被获取到的页面将是其中之一。 后续的URL则从初始的URL获取到的数据中提取。
● parse() 是spider的一个方法。 被调用时，每个初始URL完成下载后生成的 Response 对象将会作为唯一的参数传递给该函数。 该方法负责解析返回的数据(response data)，提取数据(生成item)以及生成需要进一步处理的URL的 Request 对象。
以下为我们的第一个Spider代码，保存在 myfendo/spiders 目录下的 dmoz_spider.py 文件中,可以通过genspider命令先查看可以使用的爬虫模板
scrapy genspider -l
然后才创建一个爬虫模板:scrapy genspider [-t 模板名称] 文件名 域名
scrapy genspider -t basic myfendos dmoz.org
创建成功目录下会多出个文件,文件的内容为
# -*- coding: utf-8 -*-
import scrapy
class MyfendosSpider(scrapy.Spider):
name = "myfendos"
allowed_domains = ["dmoz.org"]
start_urls = (
'http://www.dmoz.org/',
)
def parse(self, response):
pass
修改成如下:
import scrapy
class MyfendosSpider(scrapy.Spider):
name = "myfendos"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
]
def parse(self, response):
filename = response.url.split("/")[-2]
with open(filename, 'wb') as f:
f.write(response.body)
四、运行爬虫
进入项目的根目录，执行下列命令启动spider:
G:\Scrapy_work\myfendo>scrapy crawl myfendos
2017-04-15 18:17:31 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: myfendo)
2017-04-15 18:17:31 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
2017-04-15 18:17:31 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2017-04-15 18:17:31 [twisted] CRITICAL: Unhandled error in Deferred:
2017-04-15 18:17:31 [twisted] CRITICAL:
Traceback (most recent call last):
File "f:\software\python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 72, in crawl
self.engine = self._create_engine()
File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "f:\software\python36\lib\site-packages\scrapy\core\engine.py", line 68, in __init__
self.downloader = downloader_cls(crawler)
File "f:\software\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "f:\software\python36\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "f:\software\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 978, in _gcd_import
File "", line 961, in _find_and_load
File "", line 950, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 678, in exec_module
File "", line 205, in _call_with_frames_removed
File "f:\software\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 23, in
from scrapy.xlib.tx import ResponseFailed
File "f:\software\python36\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in
from twisted.web import client
File "f:\software\python36\lib\site-packages\twisted\web\client.py", line 42, in
from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
File "f:\software\python36\lib\site-packages\twisted\internet\endpoints.py", line 37, in
from twisted.internet.stdio import StandardIO, PipeAddress
File "f:\software\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in
from twisted.internet import _win32stdio
File "f:\software\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in
import win32api
ModuleNotFoundError: No module named 'win32api' 报错提示找不到:"win32api"去这个网站:http://www.lfd.uci.edu/~gohlke/pythonlibs/#pywin32 下载pywin32-221-cp36-cp36m-win_amd64.exe
下载下来之后,进行安装
安装成功,运行爬虫又报错。。
G:\Scrapy_work\myfendo>scrapy crawl myfendos
2017-04-15 18:38:24 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: myfendo)
2017-04-15 18:38:24 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
2017-04-15 18:38:24 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
Unhandled error in Deferred:
2017-04-15 18:38:24 [twisted] CRITICAL: Unhandled error in Deferred:
2017-04-15 18:38:24 [twisted] CRITICAL:
Traceback (most recent call last):
File "f:\software\python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 72, in crawl
self.engine = self._create_engine()
File "f:\software\python36\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "f:\software\python36\lib\site-packages\scrapy\core\engine.py", line 68, in __init__
self.downloader = downloader_cls(crawler)
File "f:\software\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "f:\software\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "f:\software\python36\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "f:\software\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 978, in _gcd_import
File "", line 961, in _find_and_load
File "", line 950, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 678, in exec_module
File "", line 205, in _call_with_frames_removed
File "f:\software\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 23, in
from scrapy.xlib.tx import ResponseFailed
File "f:\software\python36\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in
from twisted.web import client
File "f:\software\python36\lib\site-packages\twisted\web\client.py", line 42, in
from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
File "f:\software\python36\lib\site-packages\twisted\internet\endpoints.py", line 37, in
from twisted.internet.stdio import StandardIO, PipeAddress
File "f:\software\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in
from twisted.internet import _win32stdio
File "f:\software\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in
import win32api
ImportError: DLL load failed: 找不到指定的模块。 解决方案：将F:\Software\Python36\Lib\site-packages\pywin32_system32目录创建及以下2个DLL文件复制到system32中：
然后再次执行命令:
scrapy crawl myfendos执行成功,不过返回403,由于这个网站目前已经关闭,所以返回的是403!!
G:\Scrapy_work\myfendo>scrapy crawl myfendos
2017-04-15 19:57:17 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: myfendo)
2017-04-15 19:57:17 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
2017-04-15 19:57:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-04-15 19:57:18 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-04-15 19:57:18 [scrapy.core.engine] INFO: Spider opened
2017-04-15 19:57:18 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-04-15 19:57:18 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-04-15 19:57:19 [scrapy.core.engine] DEBUG: Crawled (403) (referer: None)
2017-04-15 19:57:20 [scrapy.core.engine] DEBUG: Crawled (403)