使用scrapy下载视频

前言

上一篇文章写了scrapy下载图片这个文章的内容是使用scrapy下载视频其实他们两个大致上并没有什么差距

items文件的配置

依旧是配置两个参数据一个是名字一个是url

import scrapy


class VidoeItem(scrapy.Item):
    # define the fields for your item here like:
    name = scrapy.Field()
    url = scrapy.Field()

piplines 文件配置

需要引入FilesPipeline模块

import scrapy
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
from scrapy.pipelines.files import FilesPipeline

class VidoePipeline(FilesPipeline):
    pass
class VideoDownloadPipeline(FilesPipeline):

    def get_media_requests(self, item, info):
        # 依次对视频地址发送请求，meta用于传递视频的文件名
        yield scrapy.Request(url=item['url'], meta={'name': item['name']})

    def file_path(self, request, response=None, info=None, *, item=None):
        filename = request.meta['name']  # 获取视频文件名
        return filename  # 返回下载的视频文件名

    def item_completed(self, results, item, info):
        return item

settings中注册

将通道激活并设置下载位置

ITEM_PIPELINES = {
   'vidoe.pipelines.VideoDownloadPipeline': 300,
}
FILES_STORE = r'./视频'

run文件

from scrapy.cmdline import execute
execute('scrapy crawl vispider'.split())

愿君前程似锦，未来可期去,感谢您的阅读，如果对您有用希望您留下宝贵的点赞和收藏