scrapy + cloudscraper 产生错误: CloudflareChallengeError Detected a Cloudflare version 2 challenge

错误示例

Traceback (most recent call last):
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.
2021-04-27 09:59:30 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.for.com/us/shop/cata>
Traceback (most recent call last):
StopIteration: <403 https://www.for.com/us/s>

原因没有加入cookie:

解决办法:

'COOKIES_ENABLED': True

深层原因:

需要购买作者的付费版本,详情需要加入作者的discord.

scrapy兼容cloudscraper

因使用cloudescraper需升级openssl版本教程

参考链接

你可能感兴趣的:(爬虫,scrapy,错误,docker,容器,运维)