Luke Liu
Luke Liu

Scrapy 爬虫框架01—— logging（日志）

1.为什么要使用Logging?

  在进行测试的时候，许多初学Pythoner喜欢使用Print函数，这样本身没什么不好，

但是在使用scrapy的时候，由于Scrapy 是多线程，所以你有时候没有办法确定在terminal上显示的data到底是哪里的data.

这时，我们可以使用logging 模块。

 

首先看在scrapy中使用logging模块。

首先是看scrapy 文件：

# -*- coding: utf-8 -*-
import scrapy
import  logging

logger  = logging.getLogger(__name__)

class ItcastSpider(scrapy.Spider):
    name = "itcast"   #  爬虫名
    allowed_domains = ["itcast.cn"]    #允许爬取的范围
    start_urls = ['http://www.itcast.cn/channel/teacher.shtml']  #最开始请求url地址

    def parse(self, response):

        data = response.xpath("//div[@class='tea_con']//li")
        # 分组组成一个item字典
        for li in data:
            item1 = {}
            item1['name'] = li.xpath(".//h3/text()").extract_first()
            item1['title']= li.xpath('.//h4/text()').extract_first()
            logger.warning('this is spider!!')
            yield item1

 logger.warning(item1)   #  把logger中warning的部分output

这里是根据您的settings.py 里面的 LOG_LEVEL确定的

LOG_LEVEL='WARNING'

Scrapy 爬虫框架01—— logging（日志）_第1张图片

ok，我们也可以在pipelines里面写，这样就能看清楚数据被哪个py文件处理。

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import  logging
logger = logging.getLogger(__name__)

class Myspiderpro1Pipeline(object):
    def process_item(self, item, spider):
        if spider.name=='itcast':
            logger.warning('this is pipelines!!')
            pass

        return item


class Myspiderpro1Pipeline2(object):
    def process_item(self, item, spider):
        item['rank']="A"
        return item

执行爬虫：

scrapy crawl itcast

2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!

这么长的logger..很清楚是吧。。

当然我们可以把日志保存。

在settings.py中进行设置：（LOG_FILE=filename）

LOG_LEVEL='WARNING'
LOG_FILE='./log1.log'

再次执行，就不会有长输出而是多了log 文件：

Scrapy 爬虫框架01—— logging（日志）_第2张图片

通过看log 文件就可以知道爬虫的进度与效果。

你可能感兴趣的:(Scarpy)

  • 2019-05-14--重学Python30--添加环境变量有啥用？如何添加？ heiqimingren
    在安装scarpy框架的过程中，我被告知，我需要把scarpy所在的路径添加到环境变量当中，于是找到了教程。地址：https://www.cnblogs.com/xiaoli2018/p/4566639.html环境变量有啥用呢？1，简单的说计算机在执行命令的时候是在环境变量找对应的命令的位置的。如果不正确设置环境变量就不能正确使用相应的命令。如果觉得麻烦可以安装IDE，这样一般就不用自己手动设置
  • 【头歌】——数据分析与实践-python-网络爬虫-Scrapy爬虫基础-网页数据解析-requests 爬虫-JSON基础 くらんゆうき 【头歌】——数据分析与实践答案数据分析python爬虫
    【头歌】——数据分析与实践-python-网络爬虫-Scrapy爬虫基础-网页数据解析-requests爬虫-JSON基础Pandas初体验第1关爬取网页的表格信息第2关爬取表格中指定单元格的信息第3关将单元格的信息保存到列表并排序第4关爬取div标签的信息第5关爬取单页多个div标签的信息第6关爬取多个网页的多个div标签的信息Scrapy爬虫基础第1关Scarpy安装与项目创建第2关Scrap
  • 使用Scarpy框架爬取链家网站经纪人列表 NAVI.electroNic scrapypythonbeautifulsouppycharm
    简介在网络数据抓取和处理中，Scrapy是一个强大的工具，它能够简化爬虫的构建过程。本文将介绍如何使用Scrapy抓取链家网站的经纪人信息，并通过Python对这些信息进行处理。一、环境设置安装Python确保您的计算机上安装了Python。您可以在Python官方网站上下载并安装最新版本的Python。安装pip在安装Python时，通常会一并安装pip，它是Python包管理工具。但如果您的P
  • scrapy_redis概念作用和流程 攒了一袋星辰 Python爬虫scrapyredis数据库
    scrapy_redis概念作用和流程学习目标了解分布式的概念及特点了解scarpy_redis的概念了解scrapy_redis的作用了解scrapy_redis的工作流程在前面scrapy框架中我们已经能够使用框架实现爬虫爬取网站数据,如果当前网站的数据比较庞大,我们就需要使用分布式来更快的爬取数据1.分布式是什么简单的说分布式就是不同的节点（服务器，ip不同）共同完成一个任务2.scrapy
  • 程序员用一个爬虫小案例，带你了解scarpy爬虫框架 Python末末
    内容Scrapyshell只要了解了Scrapyshell，就会发现它是开发和调试你的爬虫的一个非常宝贵的工具。配置shell可以通过scrapy设置配置为使用中的任何一个ipython，bpython或标准python外壳，安装无论哪个。这是通过设置SCRAPY_PYTHON_SHELL环境变量来完成的;或通过在scrapy.cfg中定义它：启动shell使用shell命令，可以启动Scrapy
  • 基于Scarpy爬取Shopee网站商品数据 spartanfuk 爬虫pythonScarpyShopee爬虫
    一、实现的主要功能：1、基于Scarpy框架爬取Shopee网站马来西亚站点商品数据，包括商品url，最低价、最高价、销量、商品图片url、店铺名称、店铺开通时间；2、针对反爬，获取免费ip代理网站的ip自动更换ip；关注公众号，发送shopee获取项目代码下载链接二、项目结构：三、用到的接口有：接口1、获取当前页50个商品数据：https://my.xiapibuy.com/api/v2/sea
  • 2022-07-08 冷月墨兰
    OlympicGames：奥运会YohanBlake：约翰·布雷克scarpy包如何使用的问题。200mFinal：200m决赛
  • Scarpy2.5从入门到高级系列教程（一）：快速了解Scrapy框架 大器晚成你别不信 Scrapy2.5从初级到高级爬虫python
    Scrapy快速一览Scrapy是一个用于抓取网站和提取结构化数据的应用程序框架，可用于各种有用的应用程序，如数据挖掘、信息处理或历史存档。尽管Scrapy最初是为网页元素抓取而设计的，但是它也能通过使用Apis来提取数据，比如亚马逊的服务接口，或者就一个通用的目的，网页元素抓取。一个蜘蛛的例子为了向您展示Scrapy带来了什么，我们将向您展示一个ScrapySpider示例，使用最简单的方式运行
  • python爬虫教程--Scrapy爬虫之旅 马骁尧 Python爬虫python爬虫scrapy
    目录一.Scarpy项目的目录结构二.Scrapy常用令1.全局命令1.1fetch命令1.2runspider命令1.3settings命令1.4shell命令1.5startproject命令1.6version命令1.7view命令2.项目命令2.1Bench命令2.2Genspider命令2.3Check命令2.4Crawl命令2.5Edit命令2.6Parse命令三.Item实战编写四.
  • 从零入门Scarpy【1】：框架、数据流 和 实战案例 小白菜_scc 爬虫python
    目录一Scrapy框架介绍二Scrapy的运作流程三Pycharm中创建Scrapy项目四制作简单Scrapy爬虫五运行爬虫Scrapy英文文档https://docs.scrapy.org/en/latestScrapy1.7中文文档https://www.osgeo.cn/scrapy/一Scrapy框架介绍ScrapyEngine：引擎。负责Scheduler、Downloader、Spid
  • 8章：scrapy框架 刘某某. 爬虫学习scrapy
    文章目录scrapy框架如何学习框架？什么是scarpy？scrapy的使用步骤1.先转到想创建工程的目录下：cd...2.创建一个工程3.创建之后要转到工程目录下4.在spiders子目录中创建一个爬虫文件5.执行工程setting文件中的参数scrapy数据解析scrapy持久化存储基于终端指令：基于管道持久化存储操作scrapy框架如何学习框架？专门学习框架封装的各种功能的详细用法。什么是s
  • 电信集团政企项目爬虫部分 yyqq188 架构scrapyddocker
    电信集团政企项目爬虫部分1项目用到的技术点有scrapyscrapydscrapyd-clientdockerdocker-compose2需求是要爬取全国各个省级以及省会的招投标信息。之前做过浙江省级的招投标爬取，利用的scarpy的本地爬取，这次的爬取网站多，而且要用到定时去爬取，并做到项目的架构的可复用以及可扩充。所以我在scrapy的基础上，利用docker以及scrapyd的技术特点来实
  • 2019-06-21 python day-09 Aidann
    1.scrapy爬虫框架的使用:一Scrapy爬虫框架发送请求--->获取响应数据--->解析数据--->保存数据**Scarpy框架介绍**1、引擎(EGINE)引擎负责控制系统所有组件之间的数据流，并在某些动作发生时触发事件。有关详细信息，请参见上面的数据流部分。2、调度器(SCHEDULER)用来接受引擎发过来的请求,压入队列中,并在引擎再次请求的时候返回.可以想像成一个URL的优先级队列,
  • 用python写网络爬虫(第二版)技术PDF书籍分享 14026da90cfa
    本书讲解了如何使用Python来编写网络爬虫程序，内容包括网络爬虫简介，从页面中抓取数据的3种方法，提取缓存中的数据，使用多个线程和进程进行并发抓取，抓取动态页面中的内容，与表单进行交互，处理页面中的验证码问题，以及使用Scarpy和Portia进行数据抓取，并在最后介绍了使用本书讲解的数据抓取技术对几个真实的网站进行抓取的实例，旨在帮助读者活学活用书中介绍的技术。本书适合有一定Python编程经
  • Python爬虫框架Scrapy安装使用步骤 weixin_33859231 python爬虫操作系统
    一、爬虫框架Scarpy简介Scrapy是一个快速的高层次的屏幕抓取和网页爬虫框架，爬取网站，从网站页面得到结构化的数据，它有着广泛的用途，从数据挖掘到监测和自动测试，Scrapy完全用Python实现，完全开源，代码托管在Github上，可运行在Linux，Windows，Mac和BSD平台上，基于Twisted的异步网络库来处理网络通讯，用户只需要定制开发几个模块就可以轻松的实现一个爬虫，用来
  • Python爬虫入门：详解Scrapy爬虫框架的基本使用（附零基础学习资料） Python副业 python爬虫scrapy爬虫教程编程免费教程
    前言在Scrapy中要抓取和解析一些逻辑内容和提取网站的链接，其实都是需要在Spider中完成的。在上一篇文章中我们介绍了Scarpy框架的简单使用，后面一些文章我们要陆续介绍框架里面的Spider、配置、管道、中间件等。（文末送福利哈）scrapy框架分为spider爬虫和CrawlSpider（规则爬虫），本篇文章主要介绍Spider爬虫的使用。spider在实现Scrapy爬虫项目时，最核心
  • Scrapy API 启动爬虫 程序猿-张益达 Python爬虫scrapy爬虫python
    scarpy不仅提供了scrapycrawlspider命令来启动爬虫，还提供了一种利用API编写脚本来启动爬虫的方法。scrapy基于twisted异步网络库构建的，因此需要在twisted容器内运行它。可以通过两个API运行爬虫：scrapy.crawler.CrawlerProcess和scrapy.crawler.CrawlerRunnerscrapy.crawler.CrawlerPro
  • scrapy爬虫框架 艾派森 python
    目录Scrapy的介绍Scarpy开发第一个爬虫Scrapy项目的启动介绍Scrapy启动-命令启动Scrapy启动-脚本启动Scrapy数据的提取Scrapy保存数据到文件ItemPipeline的使用Scrapy使用ImagePipeline保存图片Scrapy自定义ImagePipelineScrapy中settings配置的使用Scrapy中Request的使用Scrapy的介绍Scrap
  • Python爬虫之Scrapy框架系列（10）——Scrapy选择器selector 孤寒者 Python全栈系列教程Scrapy框架从入门到实战python爬虫scrapyselector选择器
    目录：1.Scrapy选择器1.1构造selector选择器1.2使用selector选择器1.Scrapy选择器Scrapy提供基于lxml库的解析机制，它们被称为选择器。因为，它们“选择”由XPath，re正则和CSS表达式指定的HTML文档的某部分。Scarpy选择器的API非常小，且非常简单。有点像bs4。1.1构造selector选择器Scrapy选择器是通过scrapy.Selecto
  • 头歌实训答案:Scrapy爬虫基础 我不是卡夫卡 头歌实训爬虫python数据挖掘
    Scrapy爬虫基础第1关：Scarpy安装与项目创建pipinstallscrapycd/rootscrapystartprojectHelloWorldcdHelloWorldscrapygenspiderworldwww.baidu.com第2关：Scrapy核心原理#-*-coding:utf-8-*-importscrapyclassWorldSpider(scrapy.Spider):
  • Scarpy爬取当当网书籍 天才少年137 python爬虫python爬虫
    目录1：Scarpy(1)Scrapy是什么：(2）安装scrapy：2.scrapy项目的创建以及运行1.创建scrapy项目：2.项目组成：3.创建爬虫文件4.爬虫文件的基本组成：5.运行爬虫文件：3.scrapy工作原理3.yield4.爬取当当网案例1：项目结构2：dang.py文件2.items文件3.pipelines文件5运行截图1：Scarpy(1)Scrapy是什么：Scrapy
  • python scrapy介绍+豆瓣案列 brkalsy python爬虫笔记python
    pythonscrapy介绍scrapy命令cd路径下，scrapystartproject命名命名的doubanspider.py：item文件：middlewaves：pipline：setting：scarpy执行命令1——终端输入scrapycrawl名字运行结果解决：复制一个headers放在setting的请求头处加了请求头后爬取豆瓣的标题用xpath,之前用的是fromlxmlimp
  • scarpy初识 听城
    Scrapy是一个为了爬取网站数据，提取结构性数据而编写的应用框架。可以应用在包括数据挖掘，信息处理或存储历史数据等一系列的程序中。Scrapy的一个最主要的优势:请求(request)是被异步调度和处理的。这意味着，Scrapy并不需要等待一个请求(request)完成及处理，在此同时，也发送其他请求或者做些其他事情。这也意味着，当有些请求失败或者处理过程中出现错误时，其他的请求也能继续处理。在
  • Scrapy 抓取链家租房（深圳）信息&高德地图Map Lab 可视化 兜兜的动感超人
    一、项目介绍项目目标1.获取链家网上的深圳市租房数据2.将获取的数据可视化文章略长，为节约部分读者时间，提前展示可视化效果工具python3.6、pycharm2018.1、高德地图MapLab技术数据抓取：Scarpy数据展示：高德地图API（MapLab）整体思路分析链家租房模块url（地区、翻页变化），找出请求url的规则分析租房条目的类别（大致分为两类，青年公寓和普通租房）分析房间详情页h
  • AttributeError: 'module' object has no attribute 'Spider' 解决了 zouxfbj
    参照上上篇安装scrapy的文章，把scarpy的依赖项以及scrapy安装完成之后，用scrapystartproject创建了一个爬虫项目，然后写了一个最简单的爬虫模块保存在项目的spider目录下，代码也很简单，如下#coding:utf-8importscrapyclassCnblogsSpider(scrapy.Spider):name="cnblogs"allowd_domains=[
  • 爬虫 Scrapy框架"链家爬取" Meter_Bulacn 爬虫
    完整的Scarpy框架我们爬取的是首页里面的详情页内容![在这里插入图片描述](https://img-blog.csdnimg.cn/20181112195618749.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxNTAw
  • scarpy 整合 djangoitem，摆脱保存数据时SQL报错的困扰 呆萌的代Ma python
    这里有一个官方的链接，虽然说的并不完整，但是思路讲解的还是很不错的：https://pypi.python.org/pypi/scrapy-djangoitem/1.1.1这里我做了一个小案例，会一步步从头进行整合，再次熟悉整个流程。为了达到更好的效果，有些变量名包含近似的含义，方便理解思路，而有的变量名完全没有意义，便于展示不同类与方法之间的指向关系。这篇博客从以下几个部分阐述：基本环境安装思路
  • 学习python爬虫scrapy框架，学习心路历程（一） 一盒白沙烟 scrapy
    scarpy框架是什么？scarpy框架是什么？scarpy框架与request、bs4的的区别常见的python爬虫框架scarpy框架是什么？Scrapy是一套基于基于Twisted的异步处理框架，纯python实现的爬虫框架，用户只需要定制开发几个模块就可以轻松的实现一个爬虫，用来抓取网页内容以及各种图片，非常之方便～scarpy框架与request、bs4的的区别reuqest和bs4（B
  • scarpy使用遇到的坑，总结 chaowanghn pythonpython
    除了抓取第一页外，抓取第2,3等下一页参考：Python+Scrapy抓取豆瓣电影top250http://www.jianshu.com/p/62e0a588ee0d#翻页next_page=response.xpath('//span[@class="next"]/a/@href')ifnext_page:url=response.urljoin(next_page[0].extract())
  • requests+selenium 爬虫项目和 scrapy 爬虫项目的区别 Felix-微信(AXiaShuBai) 网络爬虫
    爬虫项目request+selenium爬虫项目周期项目介绍爬了XXXXX，XXX，XXX，等网站，获取网站上的XXX，XXX，XXX，数据，每个月定时抓取XXX数据，使用该数据实现了XXX，XXX，XX，开发环境linux+pycharm+requests+mongodb+redis+crontab+scrapy_redis+scarpy+mysql+gevent+celery+threadin
  • JAVA基础 灵静志远 位运算加载Date字符串池覆盖
    一、类的初始化顺序 1 （静态变量，静态代码块）-->（变量，初始化块）--> 构造器 同一括号里的，根据它们在程序中的顺序来决定。上面所述是同一类中。如果是继承的情况，那就在父类到子类交替初始化。 二、String 1 String a = "abc"; JAVA虚拟机首先在字符串池中查找是否已经存在了值为"abc"的对象，根
  • keepalived实现redis主从高可用 bylijinnan redis
    方案说明 两台机器（称为A和B），以统一的VIP对外提供服务 1.正常情况下，A和B都启动，B会把A的数据同步过来（B is slave of A） 2.当A挂了后，VIP漂移到B；B的keepalived 通知redis 执行：slaveof no one，由B提供服务 3.当A起来后，VIP不切换，仍在B上面；而A的keepalived 通知redis 执行slaveof B，开始
  • java文件操作大全 0624chenhong java
    最近在博客园看到一篇比较全面的文件操作文章，转过来留着。 http://www.cnblogs.com/zhuocheng/archive/2011/12/12/2285290.html 转自http://blog.sina.com.cn/s/blog_4a9f789a0100ik3p.html 一.获得控制台用户输入的信息    &nbs
  • android学习任务 不懂事的小屁孩 工作
    任务 完成情况 搞清楚带箭头的pupupwindows和不带的使用 已完成 熟练使用pupupwindows和alertdialog，并搞清楚两者的区别 已完成 熟练使用android的线程handler,并敲示例代码 进行中 了解游戏2048的流程，并完成其代码工作 进行中-差几个actionbar 研究一下android的动画效果，写一个实例 已完成 复习fragem
  • zoom.js 换个号韩国红果果 oom
    它的基于bootstrap 的 https://raw.github.com/twbs/bootstrap/master/js/transition.js  transition.js模块引用顺序 <link rel="stylesheet" href="style/zoom.css"> <script src=&q
  • 详解Oracle云操作系统Solaris 11.2 蓝儿唯美 Solaris
    当Oracle发布Solaris 11时，它将自己的操作系统称为第一个面向云的操作系统。Oracle在发布Solaris 11.2时继续它以云为中心的基调。但是，这些说法没有告诉我们为什么Solaris是配得上云的。幸好，我们不需要等太久。Solaris11.2有4个重要的技术可以在一个有效的云实现中发挥重要作用：OpenStack、内核域、统一存档（UA）和弹性虚拟交换（EVS）。 
  • spring学习——springmvc（一） a-john springMVC
    Spring MVC基于模型-视图-控制器（Model-View-Controller，MVC）实现，能够帮助我们构建像Spring框架那样灵活和松耦合的Web应用程序。   1，跟踪Spring MVC的请求 请求的第一站是Spring的DispatcherServlet。与大多数基于Java的Web框架一样，Spring MVC所有的请求都会通过一个前端控制器Servlet。前
  • hdu4342 History repeat itself-------多校联合五 aijuans 数论
    水题就不多说什么了。 #include<iostream>#include<cstdlib>#include<stdio.h>#define ll __int64using namespace std;int main(){ int t; ll n; scanf("%d",&t); while(t--)
  • EJB和javabean的区别 asia007 beanejb
    EJB不是一般的JavaBean,EJB是企业级JavaBean,EJB一共分为3种,实体Bean,消息Bean,会话Bean,书写EJB是需要遵循一定的规范的,具体规范你可以参考相关的资料.另外,要运行EJB,你需要相应的EJB容器,比如Weblogic,Jboss等,而JavaBean不需要,只需要安装Tomcat就可以了   1.EJB用于服务端应用开发, 而JavaBeans
  • Struts的action和Result总结 百合不是茶 strutsAction配置Result配置
        一:Action的配置详解:      下面是一个Struts中一个空的Struts.xml的配置文件     <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE struts PUBLIC &quo
  • 如何带好自已的团队 bijian1013 项目管理团队管理团队
    在网上看到博客" 怎么才能让团队成员好好干活"的评论，觉得写的比较好。 原文如下： 我做团队管理有几年了吧，我和你分享一下我认为带好团队的几点： 1.诚信         对团队内成员，无论是技术研究、交流、问题探讨，要尽可能的保持一种诚信的态度，用心去做好，你的团队会感觉得到。 2.努力提
  • Java代码混淆工具 sunjing ProGuard
    Open Source Obfuscators ProGuard http://java-source.net/open-source/obfuscators/proguardProGuard is a free Java class file shrinker and obfuscator. It can detect and remove unused classes, fields, m
  • 【Redis三】基于Redis sentinel的自动failover主从复制 bit1129 redis
    在第二篇中使用2.8.17搭建了主从复制，但是它存在Master单点问题，为了解决这个问题，Redis从2.6开始引入sentinel，用于监控和管理Redis的主从复制环境，进行自动failover，即Master挂了后，sentinel自动从从服务器选出一个Master使主从复制集群仍然可以工作，如果Master醒来再次加入集群，只能以从服务器的形式工作。   什么是Sentine
  • 使用代理实现Hibernate Dao层自动事务 白糖_ DAOspringAOP框架Hibernate
    都说spring利用AOP实现自动事务处理机制非常好，但在只有hibernate这个框架情况下，我们开启session、管理事务就往往很麻烦。 public void save(Object obj){ Session session = this.getSession(); Transaction tran = session.beginTransaction(); try
  • maven3实战读书笔记 braveCS maven3
    Maven简介 是什么？ Is a software project management and comprehension tool.项目管理工具 是基于POM概念(工程对象模型) [设计重复、编码重复、文档重复、构建重复，maven最大化消除了构建的重复] [与XP：简单、交流与反馈；测试驱动开发、十分钟构建、持续集成、富有信息的工作区]     功能：
  • 编程之美-子数组的最大乘积 bylijinnan 编程之美
    public class MaxProduct { /** * 编程之美 子数组的最大乘积 * 题目: 给定一个长度为N的整数数组，只允许使用乘法，不能用除法，计算任意N-1个数的组合中乘积中最大的一组，并写出算法的时间复杂度。 * 以下程序对应书上两种方法，求得“乘积中最大的一组”的乘积——都是有溢出的可能的。 * 但按题目的意思，是要求得这个子数组，而不
  • 读书笔记-2 chengxuyuancsdn 读书笔记
    1、反射 2、oracle年-月-日 时-分-秒 3、oracle创建有参、无参函数 4、oracle行转列 5、Struts2拦截器 6、Filter过滤器(web.xml) 1、反射 (1)检查类的结构 在java.lang.reflect包里有3个类Field,Method,Constructor分别用于描述类的域、方法和构造器。 2、oracle年月日时分秒 s
  • [求学与房地产]慎重选择IT培训学校 comsci it
          关于培训学校的教学和教师的问题,我们就不讨论了,我主要关心的是这个问题       培训学校的教学楼和宿舍的环境和稳定性问题       我们大家都知道，房子是一个比较昂贵的东西，特别是那种能够当教室的房子... &nb
  • RMAN配置中通道(CHANNEL)相关参数 PARALLELISM 、FILESPERSET的关系 daizj oraclermanfilespersetPARALLELISM
    RMAN配置中通道(CHANNEL)相关参数 PARALLELISM 、FILESPERSET的关系 转 PARALLELISM --- 我们还可以通过parallelism参数来指定同时"自动"创建多少个通道： RMAN > configure device type disk parallelism 3 ; 表示启动三个通道，可以加快备份恢复的速度。
  • 简单排序:冒泡排序 dieslrae 冒泡排序
    public void bubbleSort(int[] array){ for(int i=1;i<array.length;i++){ for(int k=0;k<array.length-i;k++){ if(array[k] > array[k+1]){
  • 初二上学期难记单词三 dcj3sjt126com sciet
    concert 音乐会 tonight 今晚 famous 有名的；著名的 song 歌曲 thousand 千 accident 事故；灾难 careless 粗心的，大意的 break 折断；断裂；破碎 heart 心（脏） happen  偶尔发生，碰巧 tourist 旅游者；观光者 science （自然）科学 marry 结婚 subject 题目；
  • I.安装Memcahce 1. 安装依赖包libevent Memcache需要安装libevent,所以安装前可能需要执行 Shell代码 收藏代码 dcj3sjt126com redis
    wget http://download.redis.io/redis-stable.tar.gz tar xvzf redis-stable.tar.gz cd redis-stable make   前面3步应该没有问题，主要的问题是执行make的时候，出现了异常。 异常一： make[2]: cc: Command not found 异常原因：没有安装g
  • 并发容器 shuizhaosi888 并发容器
       通过并发容器来改善同步容器的性能，同步容器将所有对容器状态的访问都串行化，来实现线程安全，这种方式严重降低并发性，当多个线程访问时，吞吐量严重降低。    并发容器ConcurrentHashMap       替代同步基于散列的Map，通过Lock控制。   &nb
  • Spring Security（12）——Remember-Me功能 234390216 Spring SecurityRemember Me记住我
    Remember-Me功能   目录   1.1     概述 1.2     基于简单加密token的方法 1.3     基于持久化token的方法 1.4     Remember-Me相关接口和实现
  • 位运算 焦志广 位运算
    一、位运算符Ｃ语言提供了六种位运算符： & 按位与 | 按位或 ^ 按位异或 ~ 取反 << 左移 >> 右移 1. 按位与运算 按位与运算符"&"是双目运算符。其功能是参与运算的两数各对应的二进位相与。只有对应的两个二进位均为1时，结果位才为1 ，否则为0。参与运算的数以补码方式出现。 例如：9&am
  • nodejs 数据库连接 mongodb mysql liguangsong mongodbmysqlnode数据库连接
    1.mysql 连接    package.json中dependencies加入     "mysql":"~2.7.0"    执行 npm install      在config 下创建文件 database.js    
  • java动态编译 olive6615 javaHotSpotjvm动态编译
        在HotSpot虚拟机中，有两个技术是至关重要的，即动态编译(Dynamic compilation)和Profiling。     HotSpot是如何动态编译Javad的bytecode呢？Java bytecode是以解释方式被load到虚拟机的。HotSpot里有一个运行监视器，即Profile Monitor,专门监视
  • Storm0.9.5的集群部署配置优化 roadrunners 优化storm.yaml
    nimbus结点配置（storm.yaml）信息： # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional inf
  • 101个MySQL 的调节和优化的提示 tomcat_oracle mysql
    　1. 拥有足够的物理内存来把整个InnoDB文件加载到内存中——在内存中访问文件时的速度要比在硬盘中访问时快的多。 　　2. 不惜一切代价避免使用Swap交换分区 – 交换时是从硬盘读取的，它的速度很慢。 　　3. 使用电池供电的RAM（注：RAM即随机存储器）。 　　4. 使用高级的RAID（注：Redundant Arrays of Inexpensive Disks，即磁盘阵列
  • zoj 3829 Known Notation(贪心) 阿尔萨斯 ZOJ
    题目链接：zoj 3829 Known Notation 题目大意：给定一个不完整的后缀表达式，要求有2种不同操作，用尽量少的操作使得表达式完整。 解题思路：贪心，数字的个数要要保证比∗的个数多1，不够的话优先补在开头是最优的。然后遍历一遍字符串，碰到数字+1，碰到∗-1,保证数字的个数大于等1，如果不够减的话，可以和最后面的一个数字交换位置（用栈维护十分方便），因为添加和交换代价都是1
按字母分类： ABCDEFGHIJKLMNOPQRSTUVWXYZ其他