Python爬虫 | 反爬机制：UA限制 （附User-Agent大全）

示例网站：搜狗搜索引擎

爬取搜索引擎首页代码：

import requests
url = 'https://www.sogou.com/'
response = requests.get(url=url).text
# 将网页保存到本地
with open('souguo.html','w',encoding='utf-8') as f:
    f.write(response)

但是当你爬取搜索内容页面的时候~

import requests
url = 'https://www.sogou.com/web?query=程序猿过家家CSDN'
response = requests.get(url=url).text
with open('cxygjjxsdn.html','w',encoding='utf-8') as f:
    f.write(response)

修改代码之后：

import requests
url = 'https://www.sogou.com/web?query=程序猿过家家CSDN'
response = requests.get(url=url)
# 修改字符编码为utf-8
response.encoding = 'utf-8'
page_test = response.text
# 将网页保存到本地
with open('cxygjjcsdn.html','w',encoding='utf-8') as f:
    f.write(page_test)

代码没有user-agent,被识别为反爬。
这里我们给他添加一个user-agent，从浏览器右键检查里获取，如果没有内容，刷新网页之后就会显示。

import requests
url = 'https://www.sogou.com/web?query=程序猿过家家CSDN'
headers = {
     
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3776.400 QQBrowser/10.6.4212.400'
}
response = requests.get(url=url,headers=headers)
response.encoding = 'utf-8'
page_test = response.text
# 将网页保存到本地
with open('cxygjjcsdn.html','w',encoding='utf-8') as f:
    f.write(page_test)

如果高频大量爬取，可多找几个浏览器的 User-Agent 随机使用，增加代码稳健性。

附： 电脑/手机浏览器 User Agent 列表大全

系统	浏览器	User-Agent字符串
Mac	Chrome	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36
Mac	Firefox	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:65.0) Gecko/20100101 Firefox/65.0
Mac	Safari	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15
Windows	Chrome	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4292.2 Safari/537.36
Windows	Edge	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
Windows	IE	Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
iOS	Chrome	Mozilla/5.0 (iPhone; CPU iPhone OS 7_0_4 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) CriOS/31.0.1650.18 Mobile/11B554a Safari/8536.25
iOS	Safari	Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4
Android	Chrome	Mozilla/5.0 (Linux; Android 4.2.1; M040 Build/JOP40D) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.59 Mobile Safari/537.36
Android	Webkit	Mozilla/5.0 (Linux; U; Android 4.4.4; zh-cn; M351 Build/KTU84P) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30