【Python爬虫】Beautiful Soup库入门

BeautifulSoup库的安装

 

安装

pip install beautifulsoup4

 

 

 

 测试是否安装成功

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> import requests
>>> r=requests.get("https://www.baidu.com/")
>>> r.text
'\r\n   
 
 
 
 
 
  
 
       
 
 
 
 æ\x96°é\x97» hao123 å\x9c°å\x9b¾ è§\x86é¢\x91 è´´å\x90§   æ\x9b´å¤\x9a产å\x93\x81 
 
 
 
 
 
 å\x85³äº\x8eç\x99¾åº¦ About Baidu 
 
©2017 Baidu ä½¿ç\x94¨ç\x99¾åº¦å\x89\x8då¿\x85读  æ\x84\x8fè§\x81å\x8f\x8dé¦\x88 äº¬ICPè¯\x81030173å\x8f·   
 
 
 
  \r\n'
>>> r.encoding='UTF-8'
>>> r.text
'\r\n   
 
 
 
 
 
  
 
       
 
 
 
 新闻 hao123 地图 视频 贴吧   更多产品 
 
 
 
 
 
 关于百度 About Baidu 
 
©2017 Baidu 使用百度前必读  意见反馈 京ICP证030173号   
 
 
 
  \r\n'
>>> demo=r.text
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> print(soup.prettify())

>>> 

Beautiful Soup库的基本元素

 

 

 

 

 

 

 

 

 

 

 

练习记录

>>> import requests
>>> r=requests.get("https://python123.io/ws/demo.html")
>>> r.text
'\r\n\r\n
The demo python introduces several python courses.
\r\n
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\nBasic Python and Advanced Python.
\r\n'
>>> demo=r.text
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> print(soup.prettify())

 
  
 
 
  
class="title">
   
    The demo python introduces several python courses.
   
  
  
class="course">
   Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
   class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
    Basic Python
   
   and
   class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
    Advanced Python
   
   .
  
 

>>> from bs4 import BeautifulSoup
>>> soup=Beautiful("data","html.parser")
Traceback (most recent call last):
  File "", line 1, in 
    soup=Beautiful("data","html.parser")
NameError: name 'Beautiful' is not defined
>>> soup=BeautifulSoup("data","html.parser")
>>> soup2=BeautifulSoup(open("D://MyProject//Python学习//爬虫学习//demo.html")
4444
            
SyntaxError: invalid syntax
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> soup.title

>>> tag=soup.a
>>> tag
class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> soup.a.name
'a'
>>> soup.a.paernt.name
Traceback (most recent call last):
  File "", line 1, in 
    soup.a.paernt.name
AttributeError: 'NoneType' object has no attribute 'name'
>>> soup.a.parent.name
'p'
>>> soup.a.parent.parent.name
'body'
>>> tag=soup.a
>>> tag.attrs
{'href': 'http://www.icourse163.org/course/BIT-268001', 'class': ['py1'], 'id': 'link1'}
>>> tag.attrs['class']
['py1']
>>> tag.attrs['href']
'http://www.icourse163.org/course/BIT-268001'
>>> type(tag.attrs)
<class 'dict'>
>>> type(tag)
<class 'bs4.element.Tag'>
>>> soup.a
class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">Basic Python
>>> soup.a.string
'Basic Python'
>>> soup.p
class="title">The demo python introduces several python courses.
>>> soup.p.string
'The demo python introduces several python courses.'
>>> type(soup.p.string)
<class 'bs4.element.NavigableString'>
>>> newsoup=BeautifulSoup("   
 
 
 
 
 
  
 
       
 
 
 
 æ\x96°é\x97» hao123 å\x9c°å\x9b¾ è§\x86é¢\x91 è´´å\x90§   æ\x9b´å¤\x9a产å\x93\x81 
 
 
 
 
 
 å\x85³äº\x8eç\x99¾åº¦ About Baidu 
 
©2017 Baidu ä½¿ç\x94¨ç\x99¾åº¦å\x89\x8då¿\x85读  æ\x84\x8fè§\x81å\x8f\x8dé¦\x88 äº¬ICPè¯\x81030173å\x8f·   
 
 
 
  \r\n'
>>> r.encoding='UTF-8'
>>> r.text
'\r\n   
 
 
 
 
 
  
 
       
 
 
 
 新闻 hao123 地图 视频 贴吧   更多产品 
 
 
 
 
 
 关于百度 About Baidu 
 
©2017 Baidu 使用百度前必读  意见反馈 京ICP证030173号   
 
 
 
  \r\n'
>>> demo=r.text
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> soup.head
"text/html;charset=utf-8" http-equiv="content-type"/>"IE=Edge" http-equiv="X-UA-Compatible"/>"always" name="referrer"/>"https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>
>>> soup.head.contents
["text/html;charset=utf-8" http-equiv="content-type"/>, "IE=Edge" http-equiv="X-UA-Compatible"/>, "always" name="referrer"/>, "https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>, ]
>>> soup.body.contents
[' ', 
"wrapper"> 
"head"> 
class="head_wrapper"> 
class="s_form"> 
class="s_form_wrapper"> 
"lg"> "129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> 
 
"//www.baidu.com/s" class="fm" id="form" name="f"> "bdorz_come" type="hidden" value="1"/> "ie" type="hidden" value="utf-8"/> "f" type="hidden" value="8"/> "rsv_bp" type="hidden" value="1"/> "rsv_idx" type="hidden" value="1"/> "tn" type="hidden" value="baidu"/>class="bg s_ipt_wr">"off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/>class="bg s_btn_wr">"" class="bg s_btn" id="su" type="submit" value="百度一下"/> 
 
 
 
"u1"> class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻 class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123 class="mnav" href="http://map.baidu.com" name="tj_trmap">地图 class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频 class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧   class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品 
 
 
 
"ftCon"> 
"ftConw"> 
"lh"> "http://home.baidu.com">关于百度 "http://ir.baidu.com">About Baidu 
 
"cp">©2017 Baidu "http://www.baidu.com/duty/">使用百度前必读  class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈 京ICP证030173号  "//www.baidu.com/img/gs.gif"/> 
 
 
 
, ' ']
>>> len(soup.body.contents)
3
>>> soup.body.contents[1]
"wrapper"> 
"head"> 
class="head_wrapper"> 
class="s_form"> 
class="s_form_wrapper"> 
"lg"> "129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> 
 
"//www.baidu.com/s" class="fm" id="form" name="f"> "bdorz_come" type="hidden" value="1"/> "ie" type="hidden" value="utf-8"/> "f" type="hidden" value="8"/> "rsv_bp" type="hidden" value="1"/> "rsv_idx" type="hidden" value="1"/> "tn" type="hidden" value="baidu"/>class="bg s_ipt_wr">"off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/>class="bg s_btn_wr">"" class="bg s_btn" id="su" type="submit" value="百度一下"/> 
 
 
 
"u1"> class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻 class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123 class="mnav" href="http://map.baidu.com" name="tj_trmap">地图 class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频 class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧   class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品 
 
 
 
"ftCon"> 
"ftConw"> 
"lh"> "http://home.baidu.com">关于百度 "http://ir.baidu.com">About Baidu 
 
"cp">©2017 Baidu "http://www.baidu.com/duty/">使用百度前必读  class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈 京ICP证030173号  "//www.baidu.com/img/gs.gif"/> 
 
 
 
>>> 

 

 

 

 

 

 练习

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> import requests
>>> r=requests.get("https://www.baidu.com/")
>>> r.encoding='UTF-8'
>>> demo=r.text
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> soup.title.parent
"text/html;charset=utf-8" http-equiv="content-type"/>"IE=Edge" http-equiv="X-UA-Compatible"/>"always" name="referrer"/>"https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/>
>>> soup.html.parent


 "text/html;charset=utf-8" http-equiv="content-type"/>"IE=Edge" http-equiv="X-UA-Compatible"/>"always" name="referrer"/>"https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css" rel="stylesheet" type="text/css"/> "#0000cc"> 
"wrapper"> 
"head"> 
class="head_wrapper"> 
class="s_form"> 
class="s_form_wrapper"> 
"lg"> "129" hidefocus="true" src="//www.baidu.com/img/bd_logo1.png" width="270"/> 
 
"//www.baidu.com/s" class="fm" id="form" name="f"> "bdorz_come" type="hidden" value="1"/> "ie" type="hidden" value="utf-8"/> "f" type="hidden" value="8"/> "rsv_bp" type="hidden" value="1"/> "rsv_idx" type="hidden" value="1"/> "tn" type="hidden" value="baidu"/>class="bg s_ipt_wr">"off" autofocus="autofocus" class="s_ipt" id="kw" maxlength="255" name="wd" value=""/>class="bg s_btn_wr">"" class="bg s_btn" id="su" type="submit" value="百度一下"/> 
 
 
 
"u1"> class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻 class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123 class="mnav" href="http://map.baidu.com" name="tj_trmap">地图 class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频 class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧   class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品 
 
 
 
"ftCon"> 
"ftConw"> 
"lh"> "http://home.baidu.com">关于百度 "http://ir.baidu.com">About Baidu 
 
"cp">©2017 Baidu "http://www.baidu.com/duty/">使用百度前必读  class="cp-feedback" href="http://jianyi.baidu.com/">意见反馈 京ICP证030173号  "//www.baidu.com/img/gs.gif"/> 
 
 
 
  

>>> soup.parent
>>> 

 

 实践

 

 

 练习：

>>> soup.a.next_sibling
' '
>>> soup.a.next_sibling.next_sibling
class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123
>>> soup.a.previous_sibling
' '
>>> soup.a.previous_sibling.previous_sibling
>>> soup.a.parent
"u1"> class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻 class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123 class="mnav" href="http://map.baidu.com" name="tj_trmap">地图 class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频 class="mnav" href="http://tieba.baidu.com" name="tj_trtieba">贴吧   class="bri" href="//www.baidu.com/more/" name="tj_briicon" style="display: block;">更多产品 
>>> 

 

 基于bs4库的HTML格式化和编码

 

练习：

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> import requests
>>> r=requests.get("https://www.baidu.com/")
>>> r.encoding='UTF-8'
>>> demo=r.text
>>> from bs4 import BeautifulSoup
>>> soup=BeautifulSoup(demo,"html.parser")
>>> soup.prettify()
'\n\n\n \n  \n  \n  \n  \n  \n \n \n  
\n   
\n    
\n     
\n      
\n       
\n        \n       
\n       
\n        \n        \n        \n        \n        \n        \n        \n         \n        \n        \n         \n        \n       
\n      
\n     
\n     
\n      \n       新闻\n      \n      \n       hao123\n      \n      \n       地图\n      \n      \n       视频\n      \n      \n       贴吧\n      \n      \n      \n      \n       更多产品\n      \n     
\n    
\n   
\n   
\n    
\n     
\n      \n       关于百度\n      \n      \n       About Baidu\n      \n     
\n     
\n      ©2017\xa0Baidu\n      \n       使用百度前必读\n      \n      \n       意见反馈\n      \n      京ICP证030173号\n      \n     
\n    
\n   
\n  
\n \n\n'
>>> print(soup.prettify())

新闻


hao123


地图


视频


贴吧




更多产品

关于百度


About Baidu

©2017 Baidu

使用百度前必读


意见反馈

京ICP证030173号

>>> 

 

 

 

 编码要用“UTF-8”