Python爬虫笔记二——爬取爱因斯坦名言

这次的笔记主要和大家分享BeautifulSoup的一些用法。

---

数据定位

查找

BS一个很大的作用就是可以对HTML中的tag进行定位。其中最常用的函数就是find()和findAll()，这两个函数其实功能相仿，差距在于一个只寻找最近的tag，另一个会查找所有的标签。其主要参数如下：

tag : 所要查找的tag，格式为字符串或列表（一系列tag）
attributes : 所要查找tag的attributes，格式为字典，例如
.find("span", { "class" : "green", "class" : "red" })
这两个基本是最常用的参数
text : 指定tag的内容，注意是全部内容而非部分内容，但是可以使用正则表达式进行模糊匹配
keyword : 类似于attributes，不过前者是“或”判断，后者为“和”判断

移动

BS也可以在不同节点间移动
.children ： 下一级的子节点
.descendants : 所有子节点
.parent : 父节点
.next_siblings() .next_sibling() : 之后所有/一个兄弟（同一级）节点，不包括这个节点
.previous_siblings() .previous_sibling() : 之前所有/一个兄弟节点，不包括这个节点
.find_next_sibling() .find_previous_sibling() : 同前

查找爱因斯坦的名言

我们这次要爬取的网站是Quotes to Scrape,这个网站是Scrapy这个包给出的测试网站，网站的内容是一些名人名言，我们准备把其中所有爱因斯坦说的话爬取下来。网站代码如下：

    


    

        “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
        by Albert Einstein
        (about)
        
        

            Tags:
             
            
            change
            
            deep-thoughts
            
            thinking
            
            world
            
        
    

爬虫代码如下：

word_list = obj.findAll("small", { “class" : "author", "itemprop" : "author" }, text="Albert Einstein")
if len(word_list)>1:
    for i in word_list:
        print(i.parent.find_previous_sibling().get_text())

这段代码十分简单，obj是已经处理好的bs对象。我们用findAll基本定位之后，再用之前讲的方法找到目标文本。当然之前要对网页的结构进行分析，找到合适的定位方法。这就是bs的简单应用，下面是我的成果：

“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”
“Try not to become a man of success. Rather become a man of value.”
“If you can't explain it to a six year old, you don't understand it yourself.”
“If you want your children to be intelligent, read them fairy tales. If you want them to be more intelligent, read them more fairy tales.”
“Logic will get you from A to Z; imagination will get you everywhere.”
“Any fool can know. The point is to understand.”
“Life is like riding a bicycle. To keep your balance, you must keep moving.”
“If I were not a physicist, I would probably be a musician. I often think in music. I live my daydreams in music. I see my life in terms of music.”
“Anyone who has never made a mistake has never tried anything new.”

怎么样，爱翁的话是不是很有哲理？祝大家玩的开心！