scrapy 模拟登录方式
直接向目标url发起请求并携带cookie
像目标url发送post请求携带data(账号和密码)
通过selenium来模拟登录 (input标签 切换登录方式 找到用户名和密码的输入框 定位按钮)
本篇博文重点讲述直接向目标url发起请求并携带cookie方法。
以qq空间为例，获取https://user.qzone.qq.com/QQ号码网页源代码并进行解析。由于在获取该网页源代码前必须先进行登录，输入用户名和密码。而cookies是记录网页的登录信息，所以我们必须携带cookies信息。
刷新网站-Network-QQ号-找到cookies信息。
向目标url发起请求并携带cookie方法（一）
1、直接在settings.py文件里设置
- LOG_LEVEL = 'WARNING'
- ROBOTSTXT_OBEY = False
-COOKIES_ENABLED = False 记住！一定要打开COOKIES_ENABLED = False，否则无法获取响应。 - 打开DEFAULT_REQUEST_HEADERS，并将cookies以字典方式添加进来
DEFAULT_REQUEST_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36',
'cookie': 'RK=uMy93J/fTg; ptcz=**************************************5e6f05a2a8c4d2951; qz_screen=1280x720; pgv_pvid=4704727674; QZ_FE_WEBP_SUPPORT=1; _Q_w_s__QZN_TodoMsgCnt=1; zzpaneluin=; zzpanelkey=; qpsvr_localtk=0.7101978041494885; pgv_info=ssid=s2835616349; uin=o***********; skey=@fszYKYM5U; p_uin=o0********; pt4_token=TQPtF8ZsI5YhKRXdHCeU753ztK6DL6*mUHVinn06Q; p_skey=328z5QLGVe8-qxZwKtW25qNnIGNEQyqTSVkVXv3KxB8; Loading=Yes; cpu_performance_v8=4',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8',
'Accept-Language': 'en',
}
2、直接在spider主文件中直接编写
def parse(self, response):
res = response.text
with open('qzone.html','w',encoding='utf-8')as f:
f.write(res)
import scrapy
class QzSpider(scrapy.Spider):
name = 'qz'
allowed_domains = ['qq.com']
start_urls = ['https://user.qzone.qq.com/*********/']
def parse(self, response):
res = response.text
with open('qzone.html','w',encoding='utf-8')as f:
f.write(res)
3、打开qzone.html，通过谷歌浏览器打开，我们发现该网页不需要登录，直接打开。
向目标url发起请求并携带cookie方法（二）
1、我们也可以不需要经过settings.py文件进行初始化设置。我们打开spider主文件。self.parse()方法是对start_urls进行解析，所以我们要对start_urls加入cookies与UA等.那么对于类class CookSpider(scrapy.Spider):。我们查找继承类Spider，发现start_requests(self)方法相当于在spider主文件中代替settings.py文件进行初始化设置。
2、scrapy.Request()中cookie参数是一个字典。而网页中的cookies是一个字符串，所以我们必须将字符串通过字典倒推式进行转换。首先我们进行观察，我们将字符串以split(' : ')方式转换成列表，同时通过列表遍历结果传递给参数i,由于i相当于键值对，可以通过i.split('=')[0]:i.split('=')[1]:转换成字典。
cookies= {i.split('=')[0]:i.split('=')[1] for i in cook.split(' : ')}
3、scrapy.Request().
scrapy.Request()
init(self, url, callback=None, method='GET', headers=None, body=None,
cookies=None, meta=None, encoding='utf-8', priority=0,
dont_filter=False, errback=None, flags=None, cb_kwargs=None)
它携 cookies参数，所以：
yield scrapy.Request(
url = self.start_urls[0],
callback = self.parse,
cookies = cookies）
3、具体代码如下：
import scrapy
class CookSpider(scrapy.Spider):
name = 'cook'
allowed_domains = ['qq.com']
start_urls = ['https://user.qzone.qq.com/**********']
def start_requests(self):
cookie= 'RK=uMy93J/fTg; ptcz=44ac271d86936267c9755e44b***************5e6f05a2a8c4d2951; qz_screen=1280x720; pgv_pvid=4704727674; QZ_FE_WEBP_SUPPORT=1; __Q_w_s__QZN_TodoMsgCnt=1; zzpaneluin=; zzpanelkey=; _qpsvr_localtk=0.5254784523847567; pgv_info=ssid=s9175078990; uin=o************; skey=@keSzZIJQf; p_uin=o0***********; pt4_token=k9qvz-YCV5-kxS3X1dsrlwbfvyWQvWJGJRgDtRNttZM_; p_skey=5SKJsCWzK14EC1mfz9poDp8KQr-QSk9oniPmByEakxA_; Loading=Yes; x-stgw-ssl-info=87ea822f31b5ea36ac0669e3753b2520|0.139|-|25|.|I|TLSv1.2|ECDHE-RSA-AES128-GCM-SHA256|37000|h2|0; cpu_performance_v8=2'
cookies ={i.split('=')[0]:i.split('=')[1] for i in cookie.split('; ')}
yield scrapy.Request(
url = self.start_urls[0],
callback = self.parse,
cookies = cookies
)
def parse(self, response):
res = response.text
with open('qqzone.html','w',encoding='utf-8')as f:
f.write(res)
4、同理我们也可以在start_requests()中添加user-angent
import scrapy
class CookSpider(scrapy.Spider):
name = 'cook'
allowed_domains = ['qq.com']
start_urls = ['https://user.qzone.qq.com/**********']
def start_requests(self):
cookie= 'RK=uMy93J/fTg; ptcz=44ac271d86936******************************5e6f05a2a8c4d2951; qz_screen=1280x720; pgv_pvid=4704727674; QZ_FE_WEBP_SUPPORT=1; __Q_w_s__QZN_TodoMsgCnt=1; zzpaneluin=; zzpanelkey=; _qpsvr_localtk=0.5254784523847567; pgv_info=ssid=s9175078990; uin=o******************; skey=@keSzZIJQf; p_uin=o*************; pt4_token=k9qvz-YCV5-kxS3X1dsrlwbfvyWQvWJGJRgDtRNttZM_; p_skey=5SKJsCWzK14EC1mfz9poDp8KQr-QSk9oniPmByEakxA_; Loading=Yes; x-stgw-ssl-info=87ea822f31b5ea36ac0669e3753b2520|0.139|-|25|.|I|TLSv1.2|ECDHE-RSA-AES128-GCM-SHA256|37000|h2|0; cpu_performance_v8=2'
cookies ={i.split('=')[0]:i.split('=')[1] for i in cookie.split('; ')}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64)AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36'}
yield scrapy.Request(
url = self.start_urls[0],
callback = self.parse,
cookies = cookies,
headers = headers
)
def parse(self, response):
res = response.text
with open('qqzone.html','w',encoding='utf-8')as f:
f.write(res)