第一章 初见网络爬虫

1.1、网络连接

# scrapetest.py
from urllib.request import urlopen # 查找Python的request模块（在urllib库里面），指导人一个urlopen函数
html = urlopen("http://pythonscraping.com/pages/page1.html").read() # urlopen用来打开并读取一个从网络获取的远程对象
print(html)
-------------------------------------------------------------------------
# 打印输出
b'\n\n\n\n\n
An Interesting Title
\n
\nLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n
\n\n\n'

1.2、运行BeautifulSoup

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://pythonscraping.com/pages/page1.html").read()
bs0bj = BeautifulSoup(html, 'lxml')
print(bs0bj)
print(bs0bj.h1)
-------------------------------------------------------------------------
# 打印输出





An Interesting Title

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.




An Interesting Title

1.3、异常处理

from urllib.request import urlopen
from urllib.error import HTTPError
from bs4 import BeautifulSoup
def getTitle(url):
    try:
        html = urlopen(url).read()
    except HTTPError as e:
        return None
    try:
        bsobj = BeautifulSoup(html, 'lxml')
        title = bsobj.body.h1
    except AttributeError as e:
        return None
    return title
title = getTitle("http://www.pythonscraping.com/pages/page1.html")
if title == None:
    print("Title could not found")
else:
    print(title)
---------------------------------------------------------------------------
# 打印输出
An Interesting Title