Python爬虫之BeautifulSoup4库

Python爬虫之BeautifulSoup4库

1. 安装BeautifulSoup4

• 截至2020年5月24日，BeautifulSoup4最新版本为4.9.1

pip install bs4
或
pip install BeautifulSoup4
Looking in indexes: http://pypi.douban.com/simple/
Requirement already satisfied: BeautifulSoup4 in f:\python36\lib\site-packages (4.9.1)
Requirement already satisfied: soupsieve>1.2 in f:\python36\lib\site-packages (from BeautifulSoup4) (2.0.1)

2. 中文文档

中文文档目前只支持到4.40

• BeautifulSoup 4.40 文档

3. 模块导入及简单应用

# !/usr/bin/python
# ctrl+alt+L自动加空格格式化
# --king--

from bs4 import BeautifulSoup
# 注意Beautifulsoup首字母大写B

htmls = """



Once upon a time there were three little sisters; and their names were
,
Lacie and
Tillie;
and they lived at the bottom of a well.
...

"""
# Beautifulsoup(html,'解析器')
# 通过解析器可以实现代码补全
soup = BeautifulSoup(htmls,'lxml')
# print(soup)
# 格式化输出,更好的显示结果,用.prettify()
# print(soup.prettify())

# 1.对TAG进行操作
print(soup.title)
# 注意只查早符合要求的第一个标签
# 每一个标签对应一个bs4.element.Tag对象
print(soup.p,'\n',type(soup.p))
# .name输出标签名
print(soup.head.name)
# .attrs生成该标签中的所有属性,输出为字典
print(soup.p.attrs,type(soup.p.attrs))
print(soup.p['class'])
# 也可以通过.get()方式取值
print(soup.p.get('class'))

# 属性的修改
soup.p['class'] = ['new']
print(soup.p)
print(soup.p['class'])

# 2.获取NavigableString
# .string获取内容,不包含标签,输出结果对应一个bs4.element.NavigableString
print(soup.p.string,type(soup.p.string))

# 3.BeautifulSoup对象
# 继承于Tag对象,TAG只匹配第一个符合要求的结果
# BeautifulSoup对象可以支持遍历文档树和搜索文档树,可以遍历输出
# 结果为bs4.BeautifulSoup对象
print(type(soup))

# 4.Comment对象
# Comment对象是特殊的NavigableString对象
# 用来获取注释的内容,输出结果类型为bs4.element.comment
print(soup.b.string)
print(type(soup.b.string))

• 几个知识点：
• 1. 导入时注意Beautifulsoup首字母大写B
• 2. 解析器有html.parser ; lxml.html ; lxml.xml ; html5lib
• 3. .prettify()可以格式化输出html，增加缩进，提供更好的输出显示效果
• 4. Tag操作只获取匹配的第一个标签
• 5. .string获取标签内容