执行 scrapy bench
命令时 出现错误。(之前安装了pywin32库)
Traceback (most recent call last):
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\twisted\internet\defer.py", line 1260, in _inlineCallbacks
result = g.send(result)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\crawler.py", line 72, in crawl
self.engine = self._create_engine()
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\crawler.py", line 97, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\core\engine.py", line 68, in __init__
self.downloader = downloader_cls(crawler)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\core\downloader\__init__.py", line 88, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "c:\users\aobo\appdata\local\programs\python\python35\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 662, in exec_module
File "", line 222, in _call_with_frames_removed
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 23, in
from scrapy.xlib.tx import ResponseFailed
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\scrapy\xlib\tx\__init__.py", line 3, in
from twisted.web import client
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\twisted\web\client.py", line 42, in
from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\twisted\internet\endpoints.py", line 36, in
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\twisted\internet\stdio.py", line 30, in
from twisted.internet import _win32stdio
File "c:\users\aobo\appdata\local\programs\python\python35\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in
import win32api
ImportError: DLL load failed: 找不到指定的模块。
参考网站:
http://blog.csdn.net/mtt_sky/article/details/50445938
http://blog.sina.com.cn/s/blog_5a81b7990101l225.html
C:\Users\AOBO\AppData\Local\Programs\Python\Python35\Lib\site-packages\pywin32_system32
里面的所有的文件复制到:C:\Windows\System32
现在,问题解决。无需重新打开DOS窗口,直接执行:scrapy bench
。
输出正常:
D:\BaiduYunDownload\first>scrapy bench
2016-11-23 13:56:45 [scrapy] INFO: Scrapy 1.2.1 started (bot: first)
2016-11-23 13:56:45 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'first.spiders', 'CLOSESPIDER_TIMEOUT': 10, 'LOGSTATS_INTERVAL': 1, 'LOG_LEVEL': 'INFO', 'BOT_NAME': 'first', 'SPIDER_MODULES': ['first.spiders']}
2016-11-23 13:56:47 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.closespider.CloseSpider',
'scrapy.extensions.logstats.LogStats']
2016-11-23 13:56:48 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-11-23 13:56:48 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-11-23 13:56:48 [scrapy] INFO: Enabled item pipelines:
['first.pipelines.FirstPipeline']
2016-11-23 13:56:48 [scrapy] INFO: Spider opened
2016-11-23 13:56:48 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:49 [scrapy] INFO: Crawled 69 pages (at 4140 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:50 [scrapy] INFO: Crawled 141 pages (at 4320 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:51 [scrapy] INFO: Crawled 205 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:52 [scrapy] INFO: Crawled 269 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:53 [scrapy] INFO: Crawled 325 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:54 [scrapy] INFO: Crawled 373 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:55 [scrapy] INFO: Crawled 429 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:56 [scrapy] INFO: Crawled 477 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:57 [scrapy] INFO: Crawled 533 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:58 [scrapy] INFO: Crawled 581 pages (at 2880 pages/min), scraped 0 items (at 0 items/min)
2016-11-23 13:56:58 [scrapy] INFO: Closing spider (closespider_timeout)
2016-11-23 13:56:59 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 265444,
'downloader/request_count': 597,
'downloader/request_method_count/GET': 597,
'downloader/response_bytes': 1833261,
'downloader/response_count': 597,
'downloader/response_status_count/200': 597,
'finish_reason': 'closespider_timeout',
'finish_time': datetime.datetime(2016, 11, 23, 5, 56, 59, 266168),
'log_count/INFO': 17,
'request_depth_max': 20,
'response_received_count': 597,
'scheduler/dequeued': 597,
'scheduler/dequeued/memory': 597,
'scheduler/enqueued': 11938,
'scheduler/enqueued/memory': 11938,
'start_time': datetime.datetime(2016, 11, 23, 5, 56, 48, 450531)}
2016-11-23 13:56:59 [scrapy] INFO: Spider closed (closespider_timeout)
D:\BaiduYunDownload\first>