关于这个问题我弄了几天,呜呜,原来发现关键时还是很简单的
C:\Users\Hasee>cd python123demo
C:\Users\Hasee\python123demo>scrapy crawl demo
2022-09-29 11:34:50 [scrapy.utils.log] INFO: Scrapy 2.6.2 started (bot: python123demo)
2022-09-29 11:34:50 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 2.0.1, Twisted 22.8.0, Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.1.0 (OpenSSL 3.0.5 5 Jul 2022), cryptography 38.0.1, Platform Windows-10-10.0.18362-SP0
2022-09-29 11:34:50 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'python123demo',
'NEWSPIDER_MODULE': 'python123demo.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['python123demo.spiders']}
2022-09-29 11:34:50 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-09-29 11:34:50 [scrapy.extensions.telnet] INFO: Telnet Password: 56cb518663c48135
2022-09-29 11:34:50 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2022-09-29 11:34:50 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http"
Traceback (most recent call last):
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "d:\hasee\python37\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object
mod = import_module(module)
File "d:\hasee\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in
from scrapy.core.downloader.handlers.http11 import (
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in
METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
2022-09-29 11:34:50 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "https"
Traceback (most recent call last):
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "d:\hasee\python37\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object
mod = import_module(module)
File "d:\hasee\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in
from scrapy.core.downloader.handlers.http11 import (
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in
METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
2022-09-29 11:34:50 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.s3.S3DownloadHandler" for scheme "s3"
Traceback (most recent call last):
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "d:\hasee\python37\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object
mod = import_module(module)
File "d:\hasee\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\s3.py", line 1, in
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in
from scrapy.core.downloader.handlers.http11 import (
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in
METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
Unhandled error in Deferred:
2022-09-29 11:34:50 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 205, in crawl
return self._crawl(crawler, *args, **kwargs)
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 209, in _crawl
d = crawler.crawl(*args, **kwargs)
File "d:\hasee\python37\lib\site-packages\twisted\internet\defer.py", line 1946, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "d:\hasee\python37\lib\site-packages\twisted\internet\defer.py", line 1856, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status, _copy_context())
--- ---
File "d:\hasee\python37\lib\site-packages\twisted\internet\defer.py", line 1696, in _inlineCallbacks
result = context.run(gen.send, result)
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 101, in crawl
self.engine = self._create_engine()
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 115, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "d:\hasee\python37\lib\site-packages\scrapy\core\engine.py", line 83, in __init__
self.downloader = downloader_cls(crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\middleware.py", line 59, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\middleware.py", line 40, in from_settings
mwcls = load_object(clspath)
File "d:\hasee\python37\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object
mod = import_module(module)
File "d:\hasee\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "d:\hasee\python37\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 27, in
from scrapy.core.downloader.handlers.http11 import TunnelError
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in
METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)
builtins.AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
2022-09-29 11:34:50 [twisted] CRITICAL:
Traceback (most recent call last):
File "d:\hasee\python37\lib\site-packages\twisted\internet\defer.py", line 1696, in _inlineCallbacks
result = context.run(gen.send, result)
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 101, in crawl
self.engine = self._create_engine()
File "d:\hasee\python37\lib\site-packages\scrapy\crawler.py", line 115, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "d:\hasee\python37\lib\site-packages\scrapy\core\engine.py", line 83, in __init__
self.downloader = downloader_cls(crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\middleware.py", line 59, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "d:\hasee\python37\lib\site-packages\scrapy\middleware.py", line 40, in from_settings
mwcls = load_object(clspath)
File "d:\hasee\python37\lib\site-packages\scrapy\utils\misc.py", line 61, in load_object
mod = import_module(module)
File "d:\hasee\python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "d:\hasee\python37\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 27, in
from scrapy.core.downloader.handlers.http11 import TunnelError
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 23, in
from scrapy.core.downloader.contextfactory import load_context_factory_from_settings
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\contextfactory.py", line 11, in
from scrapy.core.downloader.tls import DEFAULT_CIPHERS, openssl_methods, ScrapyClientTLSOptions
File "d:\hasee\python37\lib\site-packages\scrapy\core\downloader\tls.py", line 23, in
METHOD_SSLv3: SSL.SSLv3_METHOD, # SSL 3 (NOT recommended)
AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD'
卸载scrapy 和 ssl 后,重新安装一遍解决了。
注意安装的顺序,先pyopenssl 后 scrapy
1.pip uninstall scrapy
2.pip uninstall pyopenssl
3.pip install pyopenssl
4.pip install scrapy
然后把要运行的爬虫从头来一遍就OK了