day03---python学习笔记----------xpath解析

day03—python学习笔记----------xpath解析

---

• 实例化一个etree对象

tree = etree.HTML(html)

xpath表达式:

• /:表示从根节点开始定位，表示层次
• //:表示多个层级可从任意的方开始定位
• 属性定位://div[@class = "song] tag[@attrName="attrValue]
• 索引定位://div[@class = "song /p3 索引从1开始
• 取文本:
  • /text()获取标签中直系文本内容
  • //text获取所有文本内容

---

案例4K圖片爬取

链接

---

import requests
import os
from lxml import etree
if not os.path.exists('./picLibs'):#创建文件夹
    os.mkdir('./picLibs')
url = "http://pic.netbian.com/4kdongman/index_2.html"
head = {            #模拟浏览器头部信息
        "User-Agent": "Mozilla / 5.0(Windows NT 10.0;Win64;x64;rv: 85.0) Gecko / 20100101Firefox / 85.0" #伪装成浏览器访问
    }
response_text = requests.get(url = url, headers=head).text
tree = etree.HTML(response_text)    #src和alt属性值
list_xpath = tree.xpath('//div [@class="slist"]//li')
for li in list_xpath:
    img_src = 'http://pic.netbian.com' + li.xpath('./a/img/@src')[0]
    img_name = li.xpath('./a/img/@alt')[0] + '.jpg'
    img_name=img_name.encode('iso-8859-1').decode('gbk')  # 通用处理中文乱码方法
    img_data = requests.get(url = img_src,headers=head).content#二进制进行保存
    img_path = 'picLibs/'+img_name
    with open(img_path,'wb') as fp:
        fp.write(img_data)
        print(img_name,'下载成功!!!')