爬虫 第二讲 urllib模块和requests模块

一、urllib模块

1.什么是urllib模块？

python的内置网络请求模块
为什么要学习这个模块？
1，有些比较老的爬虫项目用的就是这个技术
2.有的时候我们去爬取一些数据需要请求和urllib的配合使用
3.内置模块是标准库

示例1

# 保存'未来汽车'图片到本地
import requests

response = requests.get(
    'https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fphotocdn.sohu.com%2F20120823%2FImg351337268.jpg&refer=http%3A%2F%2Fphotocdn.sohu.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1621479722&t=5cda4533dad4056d5809bf5f2450a22f').content
with open('未来汽车.jpg', 'wb') as f:
    f.write(response)

示例2

# 保存'未来汽车'图片到本地
from urllib.request import urlretrieve

response = urlretrieve(
    'https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fphotocdn.sohu.com%2F20120823%2FImg351337268.jpg&refer=http%3A%2F%2Fphotocdn.sohu.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1621479722&t=5cda4533dad4056d5809bf5f2450a22f',
    '未来汽车.jpg')

1.urllib.request模块

python2：urllib2，urllib
python3：把urllib和urllib2合并常用的方法

• urllib.request.urlopen(“网址”) 作用 ：向网站发起一个请求并获取响应
• 字节流 = response.read()
• 字符串 = response.read().decode(“utf-8”)
• urllib.request.Request"网址",headers=“字典”) urlopen()不支持重构User-Agent

示例1

# urllib.request实现
# urllib.request.urlopen('网址')
# 作用：向网站发起请求并响应
import urllib.request

response = urllib.request.urlopen('https://www.baidu.com/')
print(type(response))  # 
print(response.read())
'''
b'\r\n\r\n\t\r\n\r\n\r\n\t\r\n\r\n'
'''   # 1.字节流bytes，需要解码     2.数据不对（网站做了反爬），需要添加ua

示例2

# 示例2
import urllib.request
headers = {
     'User-Agent': 'Mozilla/5.0'}
req = urllib.request.Request("https://www.baidu.com/",headers=headers)
response = urllib.request.urlopen(req)
print(response.read().decode('utf-8'))
# 这样出来的数据没有问题

2.响应对象

• read() 读取服务器响应的内容
• getcode() 返回HTTP的响应码
• geturl() 返回实际数据的URL(防止重定向问题)

示例

import urllib.request

# 向指定的url地址发起请求，并返回服务器响应的数据（文件的对象）
url = "http://www.baidu.com"
# 编码
newUrl2 = urllib.request.quote(url)
print(newUrl2)  # http%3A//www.baidu.com
# 解码
newUrl1 = urllib.request.unquote(newUrl2)
print(newUrl1)  # http://www.baidu.com

response = urllib.request.urlopen(newUrl1)
data = response.read()
print(data)
'''
b'\n\n\n    
# 返回当前环境的有关信息
print(response.info())
'''
Bdpagetype: 1
Bdqid: 0x0932eb0001c8f
Cache-Control: private
Content-Type: text/html;charset=utf-8
Date: Thu, 22 Apr 2021 02:28:48 GMT
Expires: Thu, 22 Apr 2021 02:27:56 GMT
P3p: CP=" OTI DSP COR IVA OUR IND COM "
P3p: CP=" OTI DSP COR IVA OUR IND COM "
Server: BWS/1.1
Set-Cookie: BAIDUID=1153AD40DBAF90F5435353FC10B:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=214453447; path=/; domain=.baidu.com
Set-Cookie: BIDUPSID=1153AD40DBAF90F5432D31787A9FC10B; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: PSTM=163534548; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BAIDUID=1153AD35DBAF90F65468EA7EB4533595:FG=1; max-age=345435300; expires=Fri, 22-Apr-22 02:28:48 GMT; domain=.baidu.com; path=/; version=1; comment=bd
Set-Cookie: BDSVRTM=0; path=/
Set-Cookie: BD_HOME=1; path=/
Set-Cookie: H_PS_PSSID=33345_3345_315353_3364_3465_265740_28657; path=/; domain=.baidu.com
Traceid: 1745674674688087589780678584646519
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Ua-Compatible: IE=Edge,chrome=1
Connection: close
Transfer-Encoding: chunked'''
# 返回状态码
print(response.getcode())  # 200
# if response.getcode() == 200 or response.getcode() == 304:
# 处理网页信息
#    pass

# 返回当前只在爬取的URL地址
print(response.geturl())  # http://www.baidu.com

3.urllib.parse模块

常用方法

• urlencode(字典)
• quote(字符串) (这个里面的参数是个字符串)

示例1

import urllib.request

# 如何编码 3个%是一个汉字
url = 'https://tieba.baidu.com/f?fr=wwwt&ie=utf-8&kw=%E7%BE%BD%E5%93%A5'
url1 = 'https://tieba.baidu.com/f?fr=wwwt&ie=utf-8&kw=羽哥'

# res = urllib.request.urlopen(url1)
# 如果我通过urllib向一个携带中文字样的url发起请求，这个时候需要注意把中文转换为 % + 十六进制 的这种数据类型：%E7%BE%BD%E5%93%A5
import urllib.parse

wd = {
     'wd': '羽哥'}
result = urllib.parse.urlencode(wd)
print(result)  # wd=%E7%BE%BD%E5%93%A5
new_url = 'https://tieba.baidu.com/f?fr=wwwt&ie=utf-8&' + result

示例2

# 示例2

import urllib.request
url2 = "https://tieba.baidu.com/f?kw=%E7%BE%BD%E5%93%A5"

# 解码
newUrl = urllib.request.unquote(url2)
print(newUrl)
'''
https://tieba.baidu.com/f?kw=羽哥'''
# 编码
newUrl2 = urllib.request.quote(newUrl)
print(newUrl2)
'''
https%3A//tieba.baidu.com/f%3Fkw%3D%E7%BE%BD%E5%93%A5'''
案例1：爬取王者荣耀高清壁纸

```python
# 爬取王者荣耀高清壁纸
# 网页分析：
# 主页： https://pvp.qq.com/web201605/wallpaper.shtml
# 第一页：https://pvp.qq.com/web201605/wallpaper.shtml
# 最后一页：https://pvp.qq.com/web201605/wallpaper.shtml  网址一样，说明下一页的图片是动态加载
# 每一页图片数量4*5=20张，共25页（第25页是6张图），一共20*24+6=486张高清图
# 第一张图地址：http://shp.qpic.cn/ishow/2735042018/1618915966_84828260_2160_sProdImgNo_7.jpg/0
# 第二张图地址：http://shp.qpic.cn/ishow/2735041519/1618485631_84828260_22420_sProdImgNo_7.jpg/0
# 第三张图地址：http://shp.qpic.cn/ishow/2735040920/1617970550_84828260_22886_sProdImgNo_7.jpg/0
# 第486张图地址：http://shp.qpic.cn/ishow/2735122518/1545733077_-888937974_7302_sProdImgNo_7.jpg/0
# No_2表示分辨率：1024x768，No_5表示分辨率：1440x900，No_7表示分辨率：1920x1200  图片地址结尾是.jpg/0
# 第1页js加载出来的数据：地址为https://apps.game.qq.com/cgi-bin/ams/module/ishow/V1.0/query/workList_inc.cgi?activityId=2735&sVerifyCode=ABCD&sDataType=JSON&iListNum=20&totalpage=0&page=0&iOrder=0&iSortNumClose=1&jsoncallback=jQuery17103347171427601099_1619236609881&iAMSActivityId=51991&_everyRead=true&iTypeId=2&iFlowId=267733&iActId=2735&iModuleId=2735&_=1619236924589
#  第2页js加载出来的数据：地址为https://apps.game.qq.com/cgi-bin/ams/module/ishow/V1.0/query/workList_inc.cgi?activityId=2735&sVerifyCode=ABCD&sDataType=JSON&iListNum=20&totalpage=0&page=1&iOrder=0&iSortNumClose=1&jsoncallback=jQuery17103347171427601099_1619236609882&iAMSActivityId=51991&_everyRead=true&iTypeId=2&iFlowId=267733&iActId=2735&iModuleId=2735&_=1619237018804
# 第25页js加载出来的数据：地址为https://apps.game.qq.com/cgi-bin/ams/module/ishow/V1.0/query/workList_inc.cgi?activityId=2735&sVerifyCode=ABCD&sDataType=JSON&iListNum=20&totalpage=0&page=24&iOrder=0&iSortNumClose=1&jsoncallback=jQuery17103347171427601099_1619236609886&iAMSActivityId=51991&_everyRead=true&iTypeId=2&iFlowId=267733&iActId=2735&iModuleId=2735&_=1619237111998
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
import re
import urllib.parse


class WangzheSpider:
    def __init__(self):
        self.headers = {
     
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36'}

    def get_url(self):
        urls = []
        for i in range(25):
            url = 'https://apps.game.qq.com/cgi-bin/ams/module/ishow/V1.0/query/workList_inc.cgi?activityId=2735&sVerifyCode=ABCD&sDataType=JSON&iListNum=20&totalpage=0&page={}&iOrder=0&iSortNumClose=1&jsoncallback=jQuery17103347171427601099&iAMSActivityId=51991&_everyRead=true&iTypeId=2&iFlowId=267733&iActId=2735'.format(
                i)
            urls.append(url)
        return urls

    def req_page(self, url, headers):
        reponse = requests.get(url, verify=False, headers=self.headers).content
        return reponse

    def write_page(self, html, filename):
        print('正在保存%s' % filename)
        response = self.req_page(html,headers=self.headers)
        with open(filename,'wb')as f:
            f.write(response)
        print('%s保存完毕' % filename)

    def main(self):
        urls = self.get_url()
        hero_names = []
        hero_images = []
        for i in urls:
            response = self.req_page(i, headers=self.headers).decode('utf-8')
            pat1 = r'"sProdImgNo_7":"(.*?)",'
            content1 = re.compile(pat1, re.S)
            sProdImgNo_7_list = content1.findall(response)
            image_address_list = []
            for item in sProdImgNo_7_list:
                i = urllib.parse.unquote(item)
                image_address_list.append(i)
            # print(image_address_lists, len(image_address_lists))
            hero_images += image_address_list
            pat2 = r'"sProdName":"(.*?)",'
            content2 = re.compile(pat2, re.S)
            sProdName_list = content2.findall(response)
            hero_name_list = []
            for item in sProdName_list:
                i = urllib.parse.unquote(item)
                hero_name_list.append(i)
            # print(hero_name_list, len(hero_name_list))
            hero_names += hero_name_list
        # print(hero_names,hero_images)
        finall_list = zip(hero_names, hero_images)
        for i in finall_list:
            html = i[1].replace('7.jpg/200', '5.jpg/0')
            filename = './image/%s.jpg' % i[0]
            self.write_page(html, filename)


if __name__ == '__main__':
    requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
    spider = WangzheSpider()
    spider.main()

发送 POST请求

示例：# 简单的翻译小软件

# 需求简单的翻译小软件
import urllib.request
import urllib.parse
import json
# 请输入您要翻译的内容
content = input("请输入您要翻译的内容:")
# 目标url 发请求
# url = 'https://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule'  # 需要去掉其中的'_o',否则返回 {"errorCode":50}
url = 'https://fanyi.youdao.com/translate?smartresult=dict&smartresult=rule'
headers = {
     
    'User-Agent': '省略'}
# 携带数据
data = {
     
    'i': content,
    'from': 'AUTO',
    'to': 'AUTO',
    'smartresult': 'dict',
    'client': 'fanyideskweb',
    'salt': '16100656169337',
    'sign': '94a4176fmdr3lc0rcb9e8r410ra3a158',
    'lts': '1619064616935',
    'bv': 'b77c859e9me719deegcc45c5s42bda',
    'doctype': 'json',
    'version': '2.1',
    'keyfrom': 'fanyi.web',
    'action': 'FY_BY_REALTlME',
}
data = urllib.parse.urlencode(data)
data = bytes(data, 'utf-8')
req = urllib.request.Request(url, data=data, headers=headers)
res = urllib.request.urlopen(req)
html = res.read().decode('utf-8')
# print(html)
'''{"type":"ZH_CN2EN","errorCode":0,"elapsedTime":0,"translateResult":[[{"src":"你好","tgt":"hello"}]]}'''  # 这是一个json类型的字符串。
# 解析数据
# json类型的str --> python类型的字典
r_dict = json.loads(html)
print(r_dict['translateResult'][0][0]['tgt'])
'''
请输入您要翻译的内容:你好
hello'''

5.练习1：输入指定内容在百度中搜索，并保存网页内容

import urllib.parse
import urllib.request
# url = "https://www.baidu.com/s?wd=%E7%BE%BD%E5%93%A5"
# 构造url
key = input("请输入要搜索的内容：")
wd = {
     'wd': key}
result = urllib.parse.urlencode(wd)  # 编码
url = 'https://www.baidu.com/s?' + result
# 创建请求对象
headers = {
     
    'User-Agent': '省略'}
req = urllib.request.Request(url, headers=headers)
# 获取响应对象
response = urllib.request.urlopen(req)
# 读取数据
html = response.read().decode('utf-8')
# 保存数据
with open('%s.html' % key, 'w', encoding='utf-8')as f:
    f.write(html)

6.练习2：输入指定内容在百度贴吧中搜索，并保存多个网页内容

# 百度贴吧练习
# 输入要爬取的贴吧主题
# 进行翻页爬取  起始页和终止页
# 保存数据

import urllib.parse
import urllib.request

# 1.分析网页:
'''
第一页：https://tieba.baidu.com/f?kw=%E5%92%8C%E5%B9%B3%E7%B2%BE%E8%8B%B1&ie=utf-8&pn=0
第二页：https://tieba.baidu.com/f?kw=%E5%92%8C%E5%B9%B3%E7%B2%BE%E8%8B%B1&ie=utf-8&pn=50
最后一页：https://tieba.baidu.com/f?kw=%E5%92%8C%E5%B9%B3%E7%B2%BE%E8%8B%B1&ie=utf-8&pn=506500
共10131页'''
# url = "https://tieba.baidu.com/f?"
# 2.构造url
name = input("请输入要搜索的贴吧名称：")
begin = int(input("请输入起始页："))
end = int(input("请输入结束页："))
kw = {
     'kw': name}
result = urllib.parse.urlencode(kw)
for i in range(begin, end + 1):
    pn = (i - 1) * 50
    url = 'https://tieba.baidu.com/f?' + result + '&pn=' + str(pn)  # 可省略&ie=utf-8
    # 3.创建请求对象
    headers = {
     
        'User-Agent': '省略'}
    req = urllib.request.Request(url, headers=headers,)
    # 4.获取响应对象
    res = urllib.request.urlopen(req, timeout=20)
    # 读取数据
    html = res.read().decode('utf-8')
    # 保存数据
    with open('第%d页.html' % i, 'w', encoding='utf-8')as f:
        print('正在爬取第%d页.html' % i)
        f.write(html)
'''
请输入要搜索的贴吧名称：和平精英
请输入起始页：1
请输入结束页：3
正在爬取第1页.html
正在爬取第2页.html
正在爬取第3页.html
'''

7.优化代码

# 练习：输入指定内容在百度贴吧中搜索，并保存多个网页内容
import urllib.parse
import urllib.request


class BaiduSpider:
    def __init__(self):
        self.headers = {
     
            'User-Agent': '省略'}
        self.base_url = 'https://tieba.baidu.com/f?'

    def readPage(self, url, headers):
        req = urllib.request.Request(url, headers=self.headers)
        res = urllib.request.urlopen(req, timeout=20)
        html = res.read().decode('utf-8')
        return html

    def writePage(self, filename, html):
        with open(filename, 'w', encoding='utf-8')as f:
            f.write(html)
            print('写入成功')

    def main(self):
        name = input("请输入要搜索的贴吧名称：")
        begin = int(input("请输入起始页："))
        end = int(input("请输入结束页："))
        kw = {
     'kw': name}
        result = urllib.parse.urlencode(kw)
        for i in range(begin, end + 1):
            pn = (i - 1) * 50
            url = self.base_url + result + '&pn=' + str(pn)
            html = self.readPage(url, headers=self.headers)
            filename = './file/第%d页.html' % i
            self.writePage(filename, html)


if __name__ == '__main__':
    spider = BaiduSpider()
    spider.main()
'''
请输入要搜索的贴吧名称：法拉利
请输入起始页：5
请输入结束页：8
写入成功
写入成功
写入成功
写入成功'''

二、requests模块

1.安装

• pip install requests
• 在开发工具中安装

2.requests常用方法

• requests.get（网址）

示例1

import requests
r = requests.get('http://www.baidu.com/').text
print(r)  # 返回网页数据

示例2

import requests

'''
response = requests.get(url, headers=headers)
1.url是最基本的url 不包含参数的
2.params中的键值对为参数
response = requests.get(url, params=params, headers=headers)
'''
# 示例1
# https://tieba.baidu.com/f?kw=%E6%B5%B7%E8%B4%BC%E7%8E%8B
url = 'https://tieba.baidu.com/f?'
params = {
     'kw': '海贼王', 'pn': '250'}
headers = {
     
    'User-Agent': '省略'}
response1 = requests.get(url, params=params, headers=headers, verify=False)
# print(response1.text)  # 成功返回百度贴吧关于海贼王的第6页html数据

# 示例2
# https://tieba.baidu.com/f?kw=%E6%B5%B7%E8%B4%BC%E7%8E%8B
url = 'https://tieba.baidu.com/f?kw=海贼王&pn=250'
headers = {
     
    'User-Agent': '省略'}
response2 = requests.get(url, headers=headers, verify=False)
# print(response2.text)  # 成功返回百度贴吧关于海贼王的第6页html数据

# 示例3
url = 'https://qq.yh31.com/zjbq/2920180.html'
headers = {
     
    'User-Agent': '省略'}
response3 = requests.get(url, headers=headers, verify=False)
# print(response3.text)  # 显示