python scrapy 模拟登录(手动登录保存cookie)

先登录网页，获取cookie,然后转化为字典，保存在settings.py中的COOKIES池中，使用中间件用cookie登录。

1、cookie,转化为字典
def cookieChangeToDict(cookie):
'''
将cookie字符串转换成字典
:param cookie: 登录后的cookie
:return:字典
'''
cookieList = cookie.split(';')
cookieDict = {}
for cookie in cookieList:
name = cookie.split('=', maxsplit=1)[0].strip()
value = cookie.split('=', maxsplit=1)[1].strip()
cookieDict[name] = value
return cookieDict

if name == 'main':
cookie = """
你的cookie
"""
print(cookieChangeToDict(cookie))

把打印出的cookie放到settings.py中自定义的COOKIES=[]中

2、使用登录后的cookie发送请求
方式一：

可以重写Spider类的start_requests方法，附带Cookie值，发送POST请求

def start_requests(self):
    url= ''
    return [scrapy.FormRequest(url, cookies = self.cookies, callback = self.parse)]

方式2：使用中间件：

from scrapy import signals
from scrapy.downloadermiddlewares.cookies import CookiesMiddleware
import random

from renren.settings import COOKIES

class RandomCookieMiddleware(CookiesMiddleware):
'''
随机cookie池
'''
def process_request(self, request, spider):
cookie = random.choice(COOKIES)
request.cookies = cookie

在settings.py中设置：

ROBOTSTXT_OBEY = False

COOKIES_ENABLED = True

启用中间件

DOWNLOADER_MIDDLEWARES = {
'renren.middlewares.RandomCookieMiddleware': 543,
}

COOKIES池

COOKIES = [
]