Scrapy 爬虫框架 ——User Agent

如何设置Scrapy框架的User Agent 呢?

在settings.py中设置即可:

BOT_NAME = 'tecent'

SPIDER_MODULES = ['tecent.spiders']
NEWSPIDER_MODULE = 'tecent.spiders'

LOG_LEVEL="WARNING"
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/66.0'

 

你可能感兴趣的:(Scarpy)