爬虫，初学者指南

第一篇：入门测速request模块的基本使用以www.douban.com为例

import requests
url = "http://www.douban.com"
heards = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
}
r = requests.get(url=url)
print(r)
print(r.headers)


{'Connection': 'close', 'Date': 'Sat, 23 Sep 2023 14:46:44 GMT', 'Server': 'dae', 'Content-Length': '0'}

request模块用于测速发送数据的连通性，通过回复可以看出418，Connection：close表示未获取到服务器的返回值，需要添加heards信息，此服务器拒绝非浏览器发送的请求。

import requests
url = "http://www.douban.com"
heards = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
}
r = requests.get(url=url, headers=heards)
print(r)
print(r.headers)


{'Date': 'Sat, 23 Sep 2023 14:49:28 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=30', 'X-Xss-Protection': '1; mode=block', 'X-Douban-Mobileapp': '0', 'Expires': 'Sun, 1 Jan 2006 01:00:00 GMT', 'Pragma': 'no-cache', 'Cache-Control': 'must-revalidate, no-cache, private', 'Set-Cookie': 'ck="deleted"; max-age=0; domain=.douban.com; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/, dbcl2="deleted"; max-age=0; domain=.douban.com; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/, ll="108099"; path=/; domain=.douban.com; expires=Sun, 22-Sep-2024 14:49:28 GMT, bid=ArFHHYIPdMc; Expires=Sun, 22-Sep-24 14:49:28 GMT; Domain=.douban.com; Path=/', 'X-DAE-App': 'sns', 'X-DAE-Instance': 'home', 'X-DAE-Mountpoint': 'True', 'X-DOUBAN-NEWBID': 'ArFHHYIPdMc', 'Server': 'dae', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=15552000;', 'Content-Encoding': 'gzip'}

上图可以看出添加了头信息headers之后成功获取了返回值，一般情况下使用request模块发送请求爬虫时，默认情况下都需要添加头信息