Python爬虫基础
文章目录
- Python学习记录
-
- Python基础爬虫：
-
- 代码：
- 运行结果：
Python学习记录
Python基础爬虫：
代码：
import urllib.request
import random
import chardet
#请求头列表
us=[
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11"
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
]
#伪造头部请求
header = {"User-Agent": random.choice(us)}
url = "https://www.sina.com.cn"
#创建一个请求对象
req = urllib.request.Request(url=url, headers=header)
res = urllib.request.urlopen(req)#将请求对象放入打开链接
data = res.read()
res = chardet.detect(data)
print(res)
html = data.decode("ascii", errors="ignore")
with open("bz.html", "wb") as f:
f.write(data)
f.close()
运行结果：
示例代码爬取内容的能力是有限的，若网站有反爬机制则无法爬取。