超级简单的python爬虫详细教程!!

爬虫

爬虫是什么

爬虫简单的来说就是用程序获取网络上数据这个过程的一种名称。

爬虫的原理

如果要获取网络上数据,我们要给爬虫一个网址(程序中通常叫URL),爬虫发送一个HTTP请求给目标网页的服务器,服务器返回数据给客户端(也就是我们的爬虫),爬虫再进行数据解析、保存等一系列操作。

流程

爬虫可以节省我们的时间,比如我要获取豆瓣电影 Top250 榜单,如果不用爬虫,我们要先在浏览器上输入豆瓣电影的 URL ,客户端(浏览器)通过解析查到豆瓣电影网页的服务器的 IP 地址,然后与它建立连接,浏览器再创造一个 HTTP 请求发送给豆瓣电影的服务器,服务器收到请求之后,把 Top250 榜单从数据库中提出,封装成一个 HTTP 响应,然后将响应结果返回给浏览器,浏览器显示响应内容,我们看到数据。我们的爬虫也是根据这个流程,只不过改成了代码形式。

HTTP请求

HTTP 请求由请求行、请求头、空行、请求体组成。

请求行由三部分组成:

    1.请求方法,常见的请求方法有 GET、POST、PUT、DELETE、HEAD  
    2.客户端要获取的资源路径  
    3.是客户端使用的 HTTP 协议版本号  

请求头是客户端向服务器发送请求的补充说明,比如说明访问者身份,这个下面会讲到。

请求体是客户端向服务器提交的数据,比如用户登录时需要提高的账号密码信息。请求头与请求体之间用空行隔开。请求体并不是所有的请求都有的,比如一般的GET都不会带有请求体。

上图就是浏览器登录豆瓣时向服务器发送的HTTP POST 请求,请求体中指定了用户名和密码。

HTTP 响应

HTTP 响应格式与请求的格式很相似,也是由响应行、响应头、空行、响应体组成。

响应行也包含三部分,分别是服务端的 HTTP 版本号、响应状态码和状态说明。

这里状态码有一张表,对应了各个状态码的意思

第二部分就是响应头,响应头与请求头对应,是服务器对该响应的一些附加说明,比如响应内容的格式是什么,响应内容的长度有多少、什么时间返回给客户端的、甚至还有一些 Cookie 信息也会放在响应头里面。

第三部分是响应体,它才是真正的响应数据,这些数据其实就是网页的 HTML 源代码。

爬虫代码怎么写

爬虫可以用很多语言比如 Python、C++等等,但是我觉得Python是最简单的,

因为Python有现成可用的库,已经封装到几乎完美,

C++虽然也有现成的库,但是它的爬虫还是比较小众,仅有的库也不足以算上简单,而且代码在各个编译器上,甚至同一个编译器上不同版本的兼容性不强,所以不是特别好用。所以今天主要介绍python爬虫。

安装requests库

cmd运行:pip install requests ,安装 requests。

然后在 IDLE 或者编译器(个人推荐 VS Code 或者 Pycharm )上输入

import requests 运行,如果没有报错,证明安装成功。

安装大部分库的方法都是:pip install xxx(库的名字)

requests的方法

requests.request()  构造一个请求,支撑一下各方法的基本方法
requests.get()  获取HTML网页的主要方法,对应于HTTP的GET

requests.head()

获取HTML网页头信息的方法,对应于HTTP的HEAD

requests.post()  向HTML网页提交POST请求的方法,对应于HTTP的POST
requests.put()   向HTML网页提交PUT请求的方法,对应于HTTP的PUT
requests.patch( )   向HTML网页提交局部修改请求,对应于HTTP的PATCT
requests.delete()  向HTML网页提交删除请求,对应于HTTP的DELETE

最常用的get方法

r = requests.get(url)

包括两个重要的对象:

构造一个向服务器请求资源的Request对象;返回一个包含服务器资源的Response对象

r.status_code  HTTP请求的返回状态,200表示连接成功,404表示失败
r.text HTTP响应内容的字符串形式,即,url对应的页面内容
r.encoding   从HTTP header中猜测的响应内容编码方式(如果header中不存在charset,则认为编码为ISO-8859-1)
r.apparent_encoding 从内容中分析的响应内容编码方式(备选编码方式)
r.content HTTP响应内容的二进制形式
requests.ConnectionError  网络连接错误异常,如DNS查询失败、拒绝连接等
requests.HTTPError HTTP错误异常
requests.URLRequired URL缺失异常
requests.TooManyRedirects 超过最大重定向次数,产生重定向异常
requests.ConnectTimeout  连接远程服务器超时异常
requests.Timeout  请求URL超时,产生超时异常

爬虫小demo

requests是最基础的爬虫库,但是我们可以做一个简单的翻译

我先把我做的一个爬虫的小项目的项目结构放上,完整源码可以私聊我下载。

下面是翻译部分的源码

import requests
def English_Chinese():
	url = "https://fanyi.baidu.com/sug"
	s = input("请输入要翻译的词(中/英):")
	dat = {
		"kw":s
		}
	resp = requests.post(url,data = dat)# 发送post请求
	ch = resp.json() # 将服务器返回的内容直接处理成json => dict
	resp.close()
	dic_lenth = len(ch['data'])
	for i in range(dic_lenth):
		print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])

代码详解:

导入requests模块,设置 url为百度翻译网页的网址。

然后通过 post 方法发送请求,再把返回的结果打成一个 dic (字典),但是这个时候我们打印出来结果发现是这样的。

这是一个字典里套列表套字典的样子,大概就是这样的

{ xx:xx , xx:[ {xx:xx} , {xx:xx} , {xx:xx} , {xx:xx} ] }

我标红的地方是我们需要的信息。

假如说我标蓝色的列表里面有 n 个字典,我们可以通过 len() 函数获取 n 的数值,

并使用 for 循环遍历,得到结果。

dic_lenth = len(ch['data']
for i in range(dic_lenth):
    print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])

最后

好了,今天的分享就到这里了,拜拜~

哎?忘了一件事,再给你们一个爬取天气的代码!

# -*- coding:utf-8 -*-
import requests
import bs4


def get_web(url):
    header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"}
    res = requests.get(url, headers=header, timeout=5)
    # print(res.encoding)
    content = res.text.encode('ISO-8859-1')
    return content


def parse_content(content):
    soup = bs4.BeautifulSoup(content, 'lxml')

    '''
    存放天气情况
    '''
    list_weather = []
    weather_list = soup.find_all('p', class_='wea')
    for i in weather_list:
        list_weather.append(i.text)

    '''
    存放日期
    '''
    list_day = []
    i = 0
    day_list = soup.find_all('h1')
    for each in day_list:
        if i <= 6:
            list_day.append(each.text.strip())
            i += 1
    # print(list_day)

    '''
    存放温度:最高温度和最低温度
    '''
    tem_list = soup.find_all('p', class_='tem')
    i = 0
    list_tem = []
    for each in tem_list:
        if i == 0:
            list_tem.append(each.i.text)
            i += 1
        elif i > 0:
            list_tem.append([each.span.text, each.i.text])
            i += 1
    # print(list_tem)

    '''
    存放风力
    '''
    list_wind = []
    wind_list = soup.find_all('p', class_='win')
    for each in wind_list:
        list_wind.append(each.i.text.strip())
    # print(list_wind)
    return list_day, list_weather, list_tem, list_wind


def get_content(url):
    content = get_web(url)
    day, weather, tem, wind = parse_content(content)
    item = 0
    for i in range(0, 7):
        if item == 0:
            print(day[i]+':\t')
            print(weather[i]+'\t')
            print("今日气温:"+tem[i]+'\t')
            print("风力:"+wind[i]+'\t')
            print('\n')
            item += 1
        elif item > 0:
            print(day[i]+':\t')
            print(weather[i] + '\t')
            print("最高气温:"+tem[i][0]+'\t')
            print("最低气温:"+tem[i][1] + '\t')
            print("风力:"+wind[i]+'\t')
            print('\n')


> 如果是初学python爬虫,觉得自学比较难的,那么我接下来分享的这全套 Python 学习资料一定不要错过,希望能给那些想学习 Python 的小伙伴们带来帮助!

#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)python学习路线

![](https://img-blog.csdnimg.cn/img_convert/20007356b0039036a10dfd04dac0f8cd.png)

#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)环境搭建

使用 Python 首先需要搭建 Python 环境,我们直接到 [Python 官网](https://link.zhihu.com/?target=https%3A//www.python.org/downloads) 下载自己对应平台和版本的安装包进行安装即可

#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)python开发工具

俗话说:工欲善其事,必先利其器,Python 的学习也是一样,新手建议首选 PyCharm,可以快速上手,减少配置时间。

![](https://img-blog.csdnimg.cn/img_convert/f9eb58520b98389d7cce601939a25def.png)

#### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/xiangxueQAQ/article/details/122216183?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168456978716800188586716%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=168456978716800188586716&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_ecpm_v1~rank_v31_ecpm-2-122216183-null-null.142%5Ev87%5Econtrol_2,239%5Ev2%5Einsert_chatgpt&utm_term=python%E6%8E%A5%E5%8D%95&spm=1018.2226.3001.4187)

[](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)学习软件
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

工欲善其事必先利其器。学习Python常用的开发软件都在这里了,给大家节省了很多时间。

![](https://img-blog.csdnimg.cn/img_convert/f874af3c700721cfa7b482156e7da47c.png)

### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)全套PDF电子书

书籍的好处就在于权威和体系健全,刚开始学习的时候你可以只看视频或者听某个人讲课,但等你学完之后,你觉得你掌握了,这时候建议还是得去看一下书籍,看权威技术书籍也是每个程序员必经之路。

![在这里插入图片描述](https://img-blog.csdnimg.cn/7844e927f29841ed8992e71b5f3a9919.jpeg#pic_center)

### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)入门学习视频

我们在看视频学习的时候,不能光动眼动脑不动手,比较科学的学习方法是在理解之后运用它们,这时候练手项目就很适合了。

![](https://img-blog.csdnimg.cn/img_convert/06e6d22e6bd1624b0689923a2e55ff81.png)

### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)实战案例

光学理论是没用的,要学会跟着一起敲,要动手实操,才能将自己的所学运用到实际当中去,这时候可以搞点实战案例来学习。

![](https://img-blog.csdnimg.cn/img_convert/982df5e1a2e44f99cb4d19f3f7aad16c.png)

[](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)五、清华编程大佬出品《漫画看学Python》
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

用通俗易懂的漫画,来教你学习Python,让你更容易记住,并且不会枯燥乏味。

![在这里插入图片描述](https://img-blog.csdnimg.cn/097b0de3966849f6a546a68612cc6fd1.png#pic_center)  
**配套600集视频:**

![在这里插入图片描述](https://img-blog.csdnimg.cn/76e119b440784cac8f92c00b12d169e1.png#pic_center)

### [](https://blog.csdn.net/2301_78165187/article/details/130987136?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130986335?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130986335%22%2C%22source%22%3A%222301_78096295%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130806966?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78094636/article/details/130782621?spm=1001.2014.3001.5502)[](https://blog.csdn.net/2301_78165187/article/details/130782138?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130782138%22%2C%22source%22%3A%222301_78165187%22%7D)[](https://blog.csdn.net/2301_78094636/article/details/130769518?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130745110?spm=1001.2014.3001.5501)[](https://blog.csdn.net/2301_78096295/article/details/130724548?spm=1001.2014.3001.5502)[](https://blog.csdn.net/hugo233/article/details/130622783?spm=1001.2014.3001.5502)六、面试资料

我们学习Python必然是为了找到高薪的工作,下面这些面试题是来自阿里、腾讯、字节等一线互联网大厂最新的面试资料,并且有阿里大佬给出了权威的解答,刷完这一套面试资料相信大家都能找到满意的工作。

![](https://img-blog.csdnimg.cn/img_convert/017aa5fae0c560e129e56957eaa86eb4.png)  
![](https://img-blog.csdnimg.cn/img_convert/ac350b25859669a72511ef8d063425eb.png)  
**上面这份完整版的Python全套学习资料已经上传至CSDN官方,朋友如果需要可以直接点击[CSDN官方认证二维码免费领取【保证100%免费】。](https://mp.weixin.qq.com/s?__biz=MzI3OTQ1NjMyMg==&mid=2247485003&idx=1&sn=0b3e59328d8cedb37fc756fde6b9f422&chksm=eb463966dc31b070fa76b5a3644d615da265619109ec42ff48976e5a65399f9cb0fc6cc3c17b#rd)**

 

你可能感兴趣的:(python,爬虫,开发语言)