BeautifulSoup

BeautifulSoup

          • BeautifulSoup类的构造
          • BeautifulSoup类的基本元素
          • BeautifulSoup节点遍历
          • BeautifulSoup节点检索--<>.find_all()
          • BeautifulSoup 其他检索方法
          • BeautifulSoup 与正则表达式相结合

BeautifulSoup类的构造
>>> import requests
>>> r=requests.get(url)
>>> r.text
'This is a python demo page\r\n\r\n

The demo python introduces several python courses.

\r\n

Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\nBasic Python and Advanced Python.

\r\n'
>>> from bs4 import BeautifulSoup >>> soup = BeautifulSoup(r.text,'html.parser') >>> print(soup.prettify()) #soup.prettify()为HTML文本智能添加换行缩进,增强可读性 <html> <head> <title> This is a python demo page </title> </head> <body> <p class="title"> <b> The demo python introduces several python courses. </b> </p> <p class="course"> Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses: <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1"> Basic Python </a> and <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2"> Advanced Python </a> . </p> </body> </html> >>>
解析器 使用方法 条件
bs4的HTML解析器 BeautifulSoup(mk,‘html.parser’) 安装bs34库
lxml的HTML解释器 BeautifulSoup(mk,‘lxml’) pip install lxml
xml的HTML解释器 BeautifulSoup(mk,‘xml’) pip install lxml
html5lb解释器 BeautifulSoup(mk,‘html5lib’) pip install html5lib
BeautifulSoup类的基本元素

...

基本元素 说明
Tag 标签,最基本的信息组织单元,分别用<>和标记开头和结尾
Name 名字,

的名字是‘p’,格式:.name
Attributes 属性,字典形式组织,格式:.attrs
NavigableString 标签内非属性字符串,格式:.string
Comment 标签内字符串的注释部分,一种特殊的Comment类型
>>> tag=soup.a                         #Tag
>>> tag
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>
>>> Name=soup.a.name                   #Name
>>> Name
'a'
>>> soup.a.parent.name
'p'
>>> soup.a.parent.parent.name
'body'
>>> Attributes=tag.attrs               #Attributes
>>> Attributes
{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
>>> tag.attrs['class']
['py1']
>>> tag.attrs['href']
'http://www.icourse163.org/course/BIT-268001'
>>> type(tag.attrs)
<class 'dict'>
>>> type(tag)
<class 'bs4.element.Tag'>
>>> NavigableString=soup.a.string      #NavigableString
>>> NavigableString
'Basic Python'
>>> soup.p
<p class="title"><b>The demo python introduces several python courses.</b></p>
>>> soup.p.string
'The demo python introduces several python courses.'
>>> type(soup.p.string)
<class 'bs4.element.NavigableString'>
>>> newsoup=BeautifulSoup("

This is not a comment

"
,"html.parser") >>> newsoup.b.string 'This is a comment' >>> type(newsoup.b.string) #Comment <class 'bs4.element.Comment'> >>> newsoup.p.string 'This is not a comment' >>> type(newsoup.p.string) <class 'bs4.element.NavigableString'>
BeautifulSoup节点遍历
属性 说明
.contents 由子节点组成的列表
.children 子节点的迭代类型,与.contents类似,用于循环遍历
.decendants 子孙节点的迭代类型,用于循环遍历
.parent 父节点的标签
.parents 祖先节点的迭代类型,用于循环遍历
.next_sibling 返回按照HTML文本顺序的下一平行节点
.next_siblings 返回按照HTML文本顺序的后续平行节点
.previous_sibling 返回按照HTML文本顺序的前一平行节点
.previous_siblings 返回按照HTML文本顺序的前序平行节点
BeautifulSoup节点检索–<>.find_all()

.find_all(name,attrs,recursive,string,**kwargs)
.find_all(标签名称,属性,搜索的所有子孙节点还是仅子节点(默认True,即前者),标签内非属性字符串,??
.find_all(name)

>>> soup.find_all('a')
[<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>, <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>]
>>> soup.find_all(['a','b'])
[<b>The demo python introduces several python courses.</b>, <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>, <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>]

.find_all(True) 返回所有子孙节点

>>> for tag in soup.find_all(True):
	print(tag.name)
html
head
title
body
p
b
p
a
a

(…) 等效于 .find_all(…)
soup(..) 等效于 soup.find_all(..)

BeautifulSoup 其他检索方法
方法 说明
<>.find() 返回.find_all()的第一个结果
<>.find_parents() 在祖先节点中搜索,返回列表类型
<>.find_parent() 返回.find_parents()的第一个结果
<>.find_next_siblings() 在平行后续节点中搜索,返回列表类型
<>.find_next_sibling() 返回。find_next_siblings()的第一个结果
<>.find_previous_siblings() 在平行前序节点中搜索,返回列表类型
<>.find_previous_sibling() 返回<>.find_previous_siblings()的第一个结果
BeautifulSoup 与正则表达式相结合
>>> soup.find_all(id='link1')
[<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>]
>>> soup.find_all(id='link')
[]
>>> import re
>>> soup.find_all(id=re.compile("link"))
[<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python</a>, <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">Advanced Python</a>]
>>> 

你可能感兴趣的:(HTML,Python,学习笔记)