3-5 Html解析器

解析器封装成类

上节课我们通过Beautiful Soup将网页中的数据进行解析获取了我们想要的数据，这节课我们要把相关的代码进行封装，详细封装流程参考视频信息

class HtmlParse():
    # 需要获取网页中的所有图片
    def __init__(self):
        # 保存待爬取url地址
        self.url_set = set()
        # 保存图片的地址
        self.img_set = set()

    # 此方法用来解析网页的url地址
    def __url_parse(self, base:str,soup: BeautifulSoup):
        # 获取页面中所有的a标签
        a_list = soup.find_all('a')
        for a in a_list:
            # 重复地址默认会自动过滤
            self.url_set.add(urljoin(base,a['href']))
        return self.url_set

    # 解析网页图片地址 (双下划线代表私有函数)
    def __img_parse(self, base:str, soup: BeautifulSoup):
        img_list = soup.find_all('img')
        for img in img_list:
            # 重复地址默认会自动过滤
            self.img_set.add(urljoin(base,img['src']))
        return self.img_set
	# 对外界提供此函数，然后此函数调用私有的方法，进行相关数据的解析操作
    def html_parse(self,base:str,html: str):
        soup = BeautifulSoup(html)
        # 调用一个方法完成一个功能
        url_set = self.__url_parse(base,soup)
        img_set = self.__img_parse(base,soup)
        return url_set, img_set

urljoin实现地址拼接

在解析的过程中html源码中有些地址是相对地址，此时应该要把相对地址拼接为绝对地址。因此可以采用urljon内置函数类实现

print(urljoin("http://www.163.com","/images/abc.png"))
# >> http://www.163.com/images/abc.png
print(urljoin("http://www.163.com","http://www.163.com/images/abc.png"))
# >> http://www.163.com/images/abc.png
print(urljoin("http://www.163.com","http://www.baidu.com/images/abc.png"))
# >> http://www.baidu.com/images/abc.png

解析器模块测试

可以把测试代码编写在 name == “main” 中这样在当前模块为非主模块时则不会执行

if __name__ == "__main__":
    html_doc = """
    
    
    
The Dormouse's story

    
Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
    and they lived at the bottom of a well.
    
    
...
    """
    hp = HtmlParse()
    url_set,img_set = hp.html_parse("http://www.163.com",html_doc)
    print(url_set)
    print(img_set)