《Python 网络数据采集》正则表达式
- 如果你用过 Python 2.x 里的 urllib2 库，可能会发现 urllib2 与 urllib 有些不同。在 Python 3.x 里，urllib2改名为 urllib，被分成一些子模块： urllib.request 、urllib.parse 和 urllib.error 。尽管函数名称大多和原来一样，但是在用新的 urllib 库时需要注意哪些函数被移动到子模块里了。
urlopen 用来大家并读取一个从网络获取的远程对象。
from urllib.request import u rlopen
html = urlopen("http://pythonscraping.com/pages/page1.html")
print(html.read())- BeautifulSoup通过定位 HTML 标签来格式化和组织复杂的网络信息，用简单易用的 Python 对象为我们展现 XML 结构信息。
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
def getTitle(url):
try:
html = urlopen(url)
except HTTPError as e:
print(e)
return None
try:
bsObj = BeautifulSoup(html.read(), "lxml")
title = bsObj.body.h1
except AttributeError as e:
print(e)
return None
return title
title = getTitle("http://pythonscraping.com/pages/page1.html")
if title == None:
print("Title could not be found!")
else:
print(title)- 复杂的HTML解析
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
try:
html = urlopen("http://www.pythonscraping.com/pages/warandpeace.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
namelist = bsObj.findAll("span", {"class":"green"})
for name in namelist:
print(name.get_text())- findAll()和find()
findAll(tag, attributes, recursive, text, limit, keywords)
find(tag, attributes, recursive, text, keywords)
tag:：传一个标签名称或多个标签组成的列表
attributes：传一个Python字典封装一个标签的若干属性和属性值。例如：.findAll("span", {"class":{"green", "red"}})
recursive：是一个递归参数，要求传一个布尔变量，默认值是Ture，所以findAll默认会去查找标签参数的所有子标签，以及子标签的子标签。改为False，findAll就至查找文档的以及标签。
text：用标签的文本内容去匹配，而不是标签的属性。
limit：范围限制参数，显然只用于findAll，find其实等价与findAll中的limit=1的情况。limit参数设置后，它的返回的前limit项结果是按照网页上的顺序排序的。
keyword：可以让你选择指定属性的标签，是BeautifulSoup设置的一个冗余功能，可替代，且偶尔会出现问题。例如bsObj.findAll(class="green")，会产生一个语法错误，因为class是Python的保留字。
- get_text()
.get_text() 会把你正在处理的 HTML 文档中所有的标签都清除，然后返回一个只包含文字的字符串。假如你正在处理一个包含许多超链接、段落和标签的大段源代码，那么 .get_text() 会把这些超链接、段落和标签都清除掉，只剩下一串不带标签的文字。
- 导航树
- 子标签和后代标签
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
try:
html = urlopen("http://www.pythonscraping.com/pages/page3.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
for child in bsObj.find("table", {"id":"giftList"}).children:
print(child)- 处理兄弟标签
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
try:
html = urlopen("http://www.pythonscraping.com/pages/page3.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
for sibling in bsObj.find("table", {"id":"giftList"}).tr.next_siblings:
print(sibling)- 父标签处理
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
try:
html = urlopen("http://www.pythonscraping.com/pages/page3.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
print(bsObj.find("img", {"src":"../img/gifts/img1.jpg"}).parent.previous_sibling.get_text())- 正则表达式
邮箱：[A-Za-z0-9\._+]+@[A-Za-z]+\.(com|org|edu|net)
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
import re
try:
html = urlopen("http://www.pythonscraping.com/pages/page3.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
images = bsObj.findAll("img", {"src":re.compile("\.\.\/img\/gifts\/img.*\.jpg")})
for img in images:
print(img["src"])- 获取属性
- 对于一个标签可以用myTag.attrs获取所有属性
- myTag.attrs["src"]表示myTag的src的属性
from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
import re
try:
html = urlopen("http://www.pythonscraping.com/pages/page3.html")
except HTTPError as e:
print(e)
try:
bsObj = BeautifulSoup(html, "lxml")
except AttributeError as e:
print(e)
images = bsObj.findAll("img", {"src":re.compile("\.\.\/img\/gifts\/img.*\.jpg")})
for img in images:
print(img.attrs["src"])