python爬虫请求头_Python制作爬虫请求头的设置方法

Python在设计爬虫时，经常由于网站会设置防爬虫的措施，这时，就要依据实际情况设置请求头和代理IP，本文就几种爬虫技术设置请求头的方法做个总结，抛砖引玉，希望对大家有帮助；

一、requests设置请求头:

import requests

url="http://www.targetweb.com"

headers={

'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',

'Cache-Control':'max-age=0',

'Connection':'keep-alive',

'Referer':'http://www.baidu.com/',

'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4882.400 QQBrowser/9.7.13059.400'}

res=requests.get(url,headers=headers)

#图片下载时要用到字节流，请求方式如下

#res=requests.get(url,stream=True,headers)

二、Selenium+Chrome请求头设置:

from selenium import webdriver

options = webdriver.ChromeOptions()

options.add_argument('lang=zh_CN.UTF-8')# 设置中文

options.add_argument('user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4882.400 QQBrowser/9.7.13059.400"')# 设置头部

browser = webdriver.Chrome(chrome_options=options)

url="http://www.targetweb.com"

browser.get(url)

browser.quit()

三、selenium+phantomjs请求头设置：

from selenium import webdriver

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

des_cap = dict(DesiredCapabilities.PHANTOMJS)

des_cap["phantomjs.page.settings.userAgent"] = ("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4882.400 QQBrowser/9.7.13059.400")

browser = webdriver.PhantomJS(desired_capabilities=des_cap)

url="http://www.targetweb.com"

browser.get(url)

browser.quit()

四、爬虫框架scrapy设置请求头：

在settings.py文件中添加如下：

DEFAULT_REQUEST_HEADERS = {

'accept': 'image/webp,*/*;q=0.8',

'accept-language': 'zh-CN,zh;q=0.8',

'referer': 'https://www.baidu.com/',

'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4882.400 QQBrowser/9.7.13059.400',}

五、Python异步Aiohttp请求头设置:

import aiohttp

url="http://www.targetweb.com"

headers={

'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',

'Cache-Control':'max-age=0',

'Connection':'keep-alive',

'Referer':'http://www.baidu.com/',

'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.4882.400 QQBrowser/9.7.13059.400'}

asyncwithaiohttp.ClientSession(headers=headers)assession:

asyncwithsession.get(url)asresp:

print(resp.status)

print(awaitresp.text())