九Python之HTML的解析（网页抓取一）
对html的解析是网页抓取的基础，分析抓取的结果找到自己想要的内容或标签以达到抓取的目的。
HTMLParser是python用来解析html的模块。它可以分析出html里面的标签、数据等等，是一种处理html的简便途径。 HTMLParser采用的是一种事件驱动的模式，当HTMLParser找到一个特定的标记时，它会去调用一个用户定义的函数，以此来通知程序处理。它主要的用户回调函数的命名都是以handler_开头的，都是HTMLParser的成员函数。当我们使用时，就从HTMLParser派生出新的类，然后重新定义这几个以handler_开头的函数即可。这几个函数包括：
- handle_startendtag 处理开始标签和结束标签
- handle_starttag 处理开始标签，比如<xx>
- handle_endtag 处理结束标签，比如</xx>
- handle_charref 处理特殊字符串，就是以&#开头的，一般是内码表示的字符
- handle_entityref 处理一些特殊字符，以&开头的，比如
- handle_data 处理数据，就是<xx>data</xx>中间的那些数据
- handle_comment 处理注释
- handle_decl 处理<!开头的，比如<!DOCTYPE html PUBLIC “-//W3C//DTD HTML 4.01 Transitional//EN”
- handle_pi 处理形如<?instruction>的东西
1. 基本解析，找到开始和结束标签
from html.parser import HTMLParser
class MyHTMLParser(HTMLParser):
def handle_starttag(self, tag, attrs):
print("Encountered the beginning of a %s tag" %(tag))
def handle_endtag(self, tag):
print ("Encountered the end of a %s tag" %(tag))
if __name__ == '__main__':
a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'
m = MyHTMLParser()
#传入要分析的html模块
m.feed(a)
运行结果
Encountered the beginning of a html tag
Encountered the beginning of a head tag
Encountered the beginning of a title tag
Encountered the end of a title tag
Encountered the beginning of a body tag
Encountered the beginning of a a tag
Encountered the end of a a tag
Encountered the end of a body tag
Encountered the end of a html tag
Encountered the beginning of a head tag
Encountered the beginning of a title tag
Encountered the end of a title tag
Encountered the beginning of a body tag
Encountered the beginning of a a tag
Encountered the end of a a tag
Encountered the end of a body tag
Encountered the end of a html tag
2. 解析html的超链接和链接显示的内容
from html.parser import HTMLParser
class MyHTMLParser(HTMLParser):
def __init__(self):
HTMLParser.__init__(self)
self.flag=None
def handle_starttag(self, tag, attrs):
# 这里重新定义了处理开始标签的函数
if tag == 'a':
# 判断标签<a>的属性
self.flag='a'
for name,value in attrs:
if name == 'href':
print("href:"+value)
def handle_data(self,data):
if self.flag == 'a':
print("data:"+data)
if __name__ == '__main__':
a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'
my = MyHTMLParser()
my.feed(a)
运行结果
href:http: //www.163.com
data:链接到163
data:链接到163
3. 完整解析html元素，拼接后原样输出
from html.parser import HTMLParser
import html.entities
class BaseHTMLProcessor(HTMLParser):
def reset(self):
# extend (called by HTMLParser.__init__)
self.pieces = []
HTMLParser.reset(self)
def handle_starttag(self, tag, attrs):
# called for each start tag
# attrs is a list of (attr, value) tuples
# e.g. for <pre class="screen">, tag="pre", attrs=[("class", "screen")]
# Ideally we would like to reconstruct original tag and attributes, but
# we may end up quoting attribute values that weren't quoted in the source
# document, or we may change the type of quotes around the attribute value
# (single to double quotes).
# Note that improperly embedded non-HTML code (like client-side Javascript)
# may be parsed incorrectly by the ancestor, causing runtime script errors.
# All non-HTML code must be enclosed in HTML comment tags (<!-- code -->)
# to ensure that it will pass through this parser unaltered (in handle_comment).
strattrs = "".join([' %s="%s"' % (key, value) for key, value in attrs])
self.pieces.append("<%(tag)s%(strattrs)s>" % locals())
def handle_endtag(self, tag):
# called for each end tag, e.g. for </pre>, tag will be "pre"
# Reconstruct the original end tag.
self.pieces.append("</%(tag)s>" % locals())
def handle_charref(self, ref):
# called for each character reference, e.g. for " ", ref will be "160"
# Reconstruct the original character reference.
self.pieces.append("&#%(ref)s;" % locals())
def handle_entityref(self, ref):
# called for each entity reference, e.g. for "©", ref will be "copy"
# Reconstruct the original entity reference.
self.pieces.append("&%(ref)s" % locals())
# standard HTML entities are closed with a semicolon; other entities are not
if entities.entitydefs.has_key(ref):
self.pieces.append(";")
def handle_data(self, text):
# called for each block of plain text, i.e. outside of any tag and
# not containing any character or entity references
# Store the original text verbatim.
self.pieces.append(text)
def handle_comment(self, text):
# called for each HTML comment, e.g. <!-- insert Javascript code here -->
# Reconstruct the original comment.
# It is especially important that the source document enclose client-side
# code (like Javascript) within comments so it can pass through this
# processor undisturbed; see comments in unknown_starttag for details.
self.pieces.append("<!--%(text)s-->" % locals())
def handle_pi(self, text):
# called for each processing instruction, e.g. <?instruction>
# Reconstruct original processing instruction.
self.pieces.append("<?%(text)s>" % locals())
def handle_decl(self, text):
# called for the DOCTYPE, if present, e.g.
# <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
# "http://www.w3.org/TR/html4/loose.dtd">
# Reconstruct original DOCTYPE
self.pieces.append("<!%(text)s>" % locals())
def output(self):
"""Return processed HTML as a single string"""
return "".join(self.pieces)
if __name__ == '__main__':
a = '<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>'
bhp =BaseHTMLProcessor()
bhp.feed(a)
print(bhp.output())
运行结果
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html><head><!--insert javaScript here!--><title>test</title><body><a href="http: //www.163.com">链接到163</a></body></html>