requests 模块

在python中requests模块常用于爬虫本文将会讲述requests常用函数的用法。

1.requests.get()/requests.post()

        1.基本语法

#首先导入requests
#pip install requests
import requests
#这里以百度为例
url="https://www.baidu.com/"
resp=requests.get(url)#requests.post()

print(resp)

        2.在一些页面中有反爬机制我们就需要写请求头

import requests
#这里以百度为例
url="https://www.baidu.com/"
headers={
     "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
     
}
resp=requests.get(url,headers=headers)

        3.代理设置

import requests
#这里以百度为例
url="https://www.baidu.com/"
proxies={
    "http":"http://120.0.0.0:8080",
    "https":"https://120.0.0.0:8080"
}
headers={
     "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
     
}
resp=requests.get(url,headers=headers,proxies=proxiesp)

        4.获取页面源代码或将数据转换成json

import requests
#这里以百度为例
url="https://www.baidu.com/"
proxies={
    "http":"http://120.0.0.0:8080",
    "https":"https://120.0.0.0:8080"
}
headers={
     "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 Edg/132.0.0.0"
     
}
resp=requests.get(url,headers=headers,proxies=proxiesp)
print(resp.text)
#如果url返回是字典用下面的代码
print(resp.json)

你可能感兴趣的:(爬虫学习dme,爬虫,爬虫,python)