起点爬虫-scrapy实践

通过这几个月以来对爬虫的基础库的研究和使用之后，个人觉得已经可以进一步拓展技能深度，学习当今流行的开源爬虫框架。当然，前期的调研工作需要做好，即了解下目前市场上的主流爬虫框架。

经过初步搜索，市面上流行的主要就Scrapy和Pyspider这两个框架，考虑到框架自身的知识深度以及将来分布式爬虫的开发与研究，我决定从scrapy入手，毕竟其具有高度的可定制性和可拓展性。

众所周知，scrapy主要由5个核心组件构成分别是 Engine，Scheduler，Downloader，Spider和Item Pipeline，此外还有两大中间件：Downloader middleware和Spider middlewares。

此处就不涉及scrapy的具体机制了，强烈推荐大家一篇解析scrapy代码的好文章，作者从架构概览，运行入口，核心组件初始化和核心抓取流程这四个方向入手，向我们详细解说了scrapy的运行机制。网址如下：

http://kaito-kidd.com/2016/11/01/scrapy-code-analyze-architecture/

好了，直奔主题，本次我们分享的是起点小说网的爬虫。
item定义如下：

import scrapy

class QidianItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    author = scrapy.Field()
    book_id = scrapy.Field()
    book_type = scrapy.Field()
    book_sub_type = scrapy.Field()
    book_name = scrapy.Field()
    book_url = scrapy.Field()
    total_words = scrapy.Field()
    click_count = scrapy.Field()
    recommand_count = scrapy.Field()
    book_status = scrapy.Field()
    rank_score = scrapy.Field()
    rank_ppl_involved = scrapy.Field()
    last_upload_date = scrapy.Field()

Spider:

# -*- coding: utf-8 -*-
import scrapy
from scrapy import Selector
from qidian.items import QidianItem
from urllib.parse import urlencode
import json

class QidianSpider(scrapy.Spider):
    name = 'Qidian'
    allowed_domain=['www.qidian.com',
                    'book.qidian.com']
    

    def start_requests(self):
        urls = 'https://www.qidian.com/all?'
#        print('request start')
#        url = 'https://www.qidian.com/all?chanId=21&orderId=&page=1&style=2&pageSize=50&siteid=1&pubflag=0&hiddenField=0'
        request_body= self.settings.getdict('DEFAULT_PARAM')
        for chanid in self.settings.getdict('CHANIDLIST').values():
            request_body['chanId']= chanid
            for page in range(1,20):
                request_body['page']=page
                url = 'https://www.qidian.com/all?'+urlencode(request_body)
                yield scrapy.Request(url,dont_filter=True)

    def parse(self,response):
        bot = Selector(response)
        csrfToken = self.get_cookies('_csrfToken',response)
        contents = bot.xpath('//tbody/tr')
        for content in contents:
            item = QidianItem()
            item['book_type'] = content.xpath('td[1]/a[1]/text()').extract_first()
            item['book_sub_type'] = content.xpath('td[1]/a[2]/text()').extract_first()
            item['book_name']= content.xpath('td[2]/a[1]/text()').extract_first()
            item['book_url'] = 'https:' + content.xpath('td[2]/a[1]/@href').extract_first()
            item['total_words'] = content.xpath('td[4]/span/text()').extract_first()
            item['author'] = content.xpath('td[5]/a/text()').extract_first()
            item['last_upload_date'] = content.xpath('td[6]/text()').extract_first()
            yield scrapy.Request(item['book_url'],meta={'item':item,'csrfToken':csrfToken},
                            callback=self.parse_detail)


    def parse_detail(self,response):
        bot = Selector(response)
        item = response.meta['item']
        item['book_status'] = bot.xpath("//div/p/span[@class='blue'][1]/text()").extract_first()
        click_content=bot.xpath('normalize-space(//div[2]/p[3])').extract_first()
        item['click_count'] = click_content.split('|')[1].split('·')[0]
        item['recommand_count'] = click_content.split('|')[2].split('·')[0]
        item['book_id'] = item['book_url'].split('/')[-1]
        detail_params = self.settings.getdict('DEFAULT_COMEMENT_PARM')
        detail_params['_csrfToken'] = response.meta['csrfToken']
        detail_params['bookId']= item['book_id']
        next_url = self.settings.get('COMMENTS_URL')+urlencode(detail_params)
        yield scrapy.Request(next_url,meta={'item':item},callback=self.parse_rank)


    def get_cookies(self,cookie_key,response):
        cookie_list = response.headers.getlist('Set-Cookie')[0].decode('utf-8').split(';')
        for cookie in cookie_list:
            if cookie_key in cookie:
                return cookie.strip().split('=')[-1]
        self.logger.info('ERORR:Cookie not found!! KEY is {}'.format(cookie_key))
        return ''


    def parse_rank(self,response):
        jsondata = json.loads(response.text)
        item = response.meta['item']
        item['rank_score'] = jsondata['data']['rate']
        item['rank_ppl_involved'] = jsondata['data']['userCount']
        yield item
    

还有其他代码，涉及到数据库的写入和简单的数据清洗，由于篇幅限制，此处不再贴出，请大家移步我的github，里面有具体的代码，网址如下：

https://github.com/xiaxia47/qidian

文章虽然已告一段落，但是这个爬虫还有需要改进的地方，比如说middleware中貌似random-user-agent的功能并没有被调用，还有ajax异步接口这边的middleware也没有起作用，另外爬虫的效率有待提升。接下来一段时间我的目标就是继续优化这个起点爬虫，使之更高效，更简洁，同时往分布式发展。

最后以我最爱的一句短语结束,极致优雅，极致简洁！