scrapy中专门用于二进制和bytes类型的数据下载的管道（下载图片）

要在setting中加上

#图片存储文件夹的名称+路径
IMAGES_STORE = './imgLibs'

 

img.py

# -*- coding: utf-8 -*-
import scrapy
from imgPro.items import ImgproItem

class ImgSpider(scrapy.Spider):
    name = 'img'
    # allowed_domains = ['www.xxx.com']
    start_urls = ['http://sc.chinaz.com/tupian/meinvtupian.html']

    def parse(self, response):
        div_list = response.xpath('//*[@id="container"]/div')
        for div in div_list:
            img_src = div.xpath('./div/a/img/@src2').extract_first()
            item = ImgproItem()
            item['img_src'] = img_src

            yield item

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# class ImgproPipeline(object):
#     def process_item(self, item, spider):
#         return item
import scrapy
from scrapy.pipelines.images import ImagesPipeline
class ImgproPipeline(ImagesPipeline):
    #是用来对媒体资源进行请求的（数据下载），参数item就是接收到的爬虫类提交的item对象
    def get_media_requests(self, item, info):
        yield scrapy.Request(item['img_src'])
    #指明数据存储的路径（文件名）
    def file_path(self, request, response=None, info=None):
        return request.url.split('/')[-1]
    #将item传递个下一个即将被执行的管道类
    def item_completed(self, results, item, info):
        return item

- 图片懒加载
    - 应用到标签的伪属性，数据捕获的时候一定是基于伪属性进行！！！
- ImagePileline：专门用作于二进制数据下载和持久化存储的管道类