使用xpath解析爬虫数据

1. 获取网页资源

    url = "https://www.douban.com/group/explore"
    headers = {
        "User-Agent": "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)"
    }
    response = requests.get(url, headers=headers)

2. xpath解析页面(from lxml import etree)

2.1 先将html文本对象转换为etree_html对象

etree_html = etree.HTML(html)

2.2 匹配所有节点

# 匹配所有节点 //*
result = etree_html.xpath('//*')

2.3 匹配所有子节点

# 匹配所有子节点a 并文本获取：text()
result = etree_html.xpath('//a/text()')
print(result)

2.4 查找元素子节点

# 查找元素子节点 /
result = etree_html.xpath('//div/p/text()')
print(result)

2.5 获取当前节点的父节点

父节点 ..  类似于cd ../,查找当前节点的父节点
result = etree_html.xpath('//span[@class="pubtime"]/../span/a/text()')

2.6 属性

单属性匹配 [@class="xxx"]
#文本匹配 text() 获取所有文本//text()
result = etree_html.xpath('//div[@class="article"]//text()')

属性多值匹配 contains(@class 'xx'),说明:当该class有多个值时,使用contains包含
result = etree_html.xpath('//div[contains(@class, "grid-16-8")]//div[@class="likes"]/text()[1]')

多属性匹配
# 多属性匹配 or, and, mod, //book | //cd, + - * div = != < > <= >=
# 按序选择 [1] [last()] [position() < 3] [last() -2] 该选择器是放在xpath内部的,如上多值匹配
result = etree_html.xpath('//span[@class="pubtime" and contains(text(), "昨天")]/text()')
print(result)

属性获取
result = etree_html.xpath('//div[@class="article"]/div/div/@class')[0]
print(result)
result = etree_html.xpath('//div[@class="bd"]/h3/a/@href')
print(result)

2.7 节点轴(了解)

//li/ancestor::*  所有祖先节点
//li/ancestor::div div这个祖先节点
//li/attribute::* attribute轴，获取li节点所有属性值
//li/child::a[@href="link1.html"]  child轴，获取直接子节点
//li/descendant::span 获取所有span类型的子孙节点
//li/following::* 选取文档中当前节点的结束标记之后的所有节点
//li/following-sibling::*     选取当前节点之后的所有同级节点,第一个不算
result = etree_html.xpath('//div[@class="channel-item"]/following-sibling::*')
print(result)
print(len(result))
result = etree_html.xpath('//div[@class="channel-item"][1]/following-sibling::*')
print(result)
print(len(result))