4、scrapy内置服务

本篇介绍scrapy的Logging模块、Stats Collections模块和Sending-email模块。

Logging

简单使用方法

import logging


logging.warning("This is a Warning")

执行结果为：

WARNING:root:This is a Warning

通用的记录日志的方法，可加入日志的级别

import logging


logging.log(logging.WARNING, "This is a Warning")

通用logger记录日志

import logging


logger = logging.getLogger(__name__)
logger.warning("This is a Warning")

运行结果:

WARNING:main:This is a warning

在scrapy中使用logging

import scrapy


class MySpider(scrapy.Spider):

    name = 'myspider'
    start_urls = ['http://scrapyinghub.com']

    def parse(self, response):
        self.logger.info('Parse function called on %s' % response.url)

settings中设置log

• LOG_FILE
• LOG_ENABLE
• LOG_ENCODING
• LOG_LEVEL
• LOG_FORMAT
• lOG_DATEFORMAT
• lOG_STDOUT

Stats Collections

使用方法（待学习）

class ExtensionThatAccessStats(object):

    def __init__(self, stats):
        self.stats = stats

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.stats)

scrapy内置的stats Collections

• class scrapy.statscollectors.MemoryStatsCollector(linux)
• class scrapy.statscollectors.DummyStatsCollector

Sending-email

通用使用方法

from scrapy.mail import MailSender


mailer = MailSender()
# mailer = MailSender.from_settings(settins)

mailer.send(to=["[email protected]"], subject="Some subject", body="Some body", cc=["another.example.com"])