Scrapy爬虫实战项目【001】 - 抓取猫眼电影TOP100

爬取猫眼电影TOP100

参考来源：静觅丨崔庆才的个人博客 https://cuiqingcai.com/5534.html

目的：使用Scrapy爬取猫眼电影TOP100并保存至MONGODB数据库

目标网址：http://maoyan.com/board/4?offset=0

分析/知识点：

1. 爬取难度：
   a. 入门级，网页结构简单，静态HTML，少量JS，不涉及AJAX；
   b. 处理分页需要用到正则；

2. MONGODB的update语句使用：
   a. update语句：具备查重/插入新数据功能，以title为查重标准

def process_item(self, item, spider):
    self.db['movies'].update({'title': item['title']}, {'$set': item}, upsert=True) #注意upsert=True，更新并插入
    return item

实际步骤：

1. 创建Scrapy项目/maoyan(spider)

Terminal: > scrapy startproject maoyan_movie
Terminal: > scrapy genspider maoyan maoyan.com/board/4?offset=   

2. 配置settings.py文件

# MONGODB配置
MONGO_URI = 'localhost'
MONGO_DB = 'maoyan_movie'
...

# 启用MongoPipeline
ITEM_PIPELINES = {
   'maoyan_movie.pipelines.MongoPipeline': 300,
}

3. 编写items.py文件

from scrapy import Item, Field

class MovieItem(Item):
    title = Field()     #电影标题
    actors = Field()    #演员
    releasetime = Field()   #上映时间
    cover_img = Field()     #缩略图
    detail_page = Field()   #电影详情页url
    score = Field()         #电影评分

4. 编写pipelines.py文件
   根据Scrapy官方文档修改：https://doc.scrapy.org/en/latest/topics/item-pipeline.html?highlight=mongo

import pymongo

class MongoPipeline(object):
    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def close_spider(self, spider):
        self.client.close()

    # !! 更新MONGODB，使用UPDATE方法，查重功能(以title字段进行判断)
    def process_item(self, item, spider):
        self.db['movies'].update({'title': item['title']}, {'$set': item}, upsert=True)
        return item

5. 编写spiders > maoyan.py文件
   注意：
   a) 使用scrapy的css selector进行节点解析；
   b) 获取电影缩略图url时，注意需要根据网页源代码写css选择器，和审查元素中看到的不同；

item['cover_img'] = movie.css('a.image-link img.board-img::attr(data-src)').extract_first()

c) 获取下一页节点时，直接使用xpath或css难以直接获得，需要使用正则匹配；

next = response.xpath('.').re_first(r'href="(.*?)">下一页')

d) 完整代码如下：

from scrapy import Spider, Request
from maoyan_movie.items import MovieItem

class MaoyanSpider(Spider):
    name = 'maoyan'
    allowed_domains = ['maoyan.com/board/4?offset=']
    start_urls = ['http://maoyan.com/board/4?offset=']

    # 每部电影详情页的基本前缀url
    base_url = 'http://maoyan.com'

    # 下一页前缀url
    next_base_url = 'http://maoyan.com/board/4'

    def parse(self, response):
        if response:
            # 获取每页所有电影的节点
            movies = response.css('dl.board-wrapper dd')  # 获取所有电影相关节点，切记！！不能加上extract()
            item = MovieItem()
            for movie in movies:
                item['title'] = movie.css('p.name a::text').extract_first()
                item['actors'] = movie.css('p.star::text').extract_first().strip()
                item['releasetime'] = movie.css('p.releasetime::text').extract_first().strip()
                item['score'] = movie.css('i.integer::text').extract_first() + movie.css(
                    'i.fraction::text').extract_first()
                item['detail_page'] = self.base_url + movie.css('p.name a::attr(href)').extract_first()
                item['cover_img'] = movie.css(
                    'a.image-link img.board-img::attr(data-src)').extract_first()  # 注意：需要根据网页源码写css选择器，和审查元素中的不同，估计是受JS影响
                yield item

            # 处理下一页
            next = response.xpath('.').re_first(r'href="(.*?)">下一页')
            if next:
                next_url = self.next_base_url + next
                yield Request(url=next_url, callback=self.parse, dont_filter=True)

6. 运行结果
   
   temp.png

小结

1. 入门级项目，熟悉了Scrapy的基本使用流程；
2. Scrapy的css/xpath选择器、正则写法需要进一步熟悉；
3. pymongo的update语句，需要进一步熟练掌握。