Python爬虫系列之----Scrapy(五)网页提取的三种方式(正则,Beautiful Soup,Lxml)

一、提取方式

从网页中提取数据有很多方法,概况起来大概有这么三种方式,首先是正则,然后是流行的Beautiful Soup模块,最后是强大的Lxml模块。

1、正则表达式:最原始的方法,通过编写一些正则表达式,然后从HTML/XML中提取数据。

2、Beautiful Soup模块:Beautiful Soup 是一个可以从 HTML 或 XML 文件中提取数据的 Python 库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup 会帮你节省数小时甚至数天的工作时间。

3、Lxml模块:lxml是基于libxm12这一XML解析库的Python封装,该模块使用C语言编写,解析速读比Beautiful Soup模块快，不过安装更为复杂。

Scrapy使用了一种基于 XPath 和 CSS 表达式机制: Scrapy Selectors。 关于selector和其他提取机制的信息请参考 Selector文档

这里给出XPath表达式的例子及对应的含义:

• /html/head/title: 选择HTML文档中  标签内的 '] In [6]: response.xpath('//title/text()') Out[6]: [] In [7]: response.xpath('//title/text()').extract() Out[7]: ['Python爬虫系列之----Scrapy(一)爬虫原理 - fendo\r\n - 博客频道 - CSDN.NET']
  
  

  
  

  
  

  二、提取数据
  

  
  

  让我们从返回的response中提取数据，可以结合Firebug来观察HTML源码并确定合适的XPath表达式

  
  

  文章的标题:
  

  
  

  response.xpath('//h1/span/a/text()').extract()

  
  
  

  
  

  
  

  输出的结果为:

  
  

  
  

  
  

  
  

  网页中的链接:

  
  

  response.xpath("//span/a[@href='https://scrapy.org/']/text()").extract()

  
  
  

  
  

  
  

  输出结果为:

  
  

  
  

  
  

  网页的描述:

  
  

  response.xpath("//meta[@name='description']").extract()

  
  

  
  

  
  

  输出结果为:

  
  

  
  

  
  

  
  

  三、使用Item

  
  

  Item 对象是自定义的python字典。 您可以使用标准的字典语法来获取到其每个字段的值。(字段即是我们之前用Field赋值的属性):
  

  
  

  一般来说，Spider将会将爬取到的数据以 Item 对象返回。所以为了将爬取的数据返回，我们最终的代码将是:
  

  
  

  # -*- coding: utf-8 -*-
  import scrapy
  from myfendo.items import MyfendoItem
  
  class MyfendosSpider(scrapy.Spider):
      name = "myfendos"
      allowed_domains = ["csdn.net"]
      start_urls = [
          "http://blog.csdn.net/u011781521/article/details/70182815",
      ]
  
      def parse(self, response):
          item=MyfendoItem()
          item['title']=response.xpath('//h1/span/a/text()')
          item['link']=response.xpath("//span/a[@href='https://scrapy.org/']/text()")
          item['desc']=response.xpath("//meta[@name='description']")
          yield item
     

  
  
  

  现在运行爬虫将会产生 MyfendoItem对象:
  

  
  

  G:\Scrapy_work\myfendo>scrapy crawl myfendos
  2017-04-15 21:50:54 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: myfendo)
  2017-04-15 21:50:54 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
  2017-04-15 21:50:54 [scrapy.middleware] INFO: Enabled extensions:
  ['scrapy.extensions.corestats.CoreStats',
   'scrapy.extensions.telnet.TelnetConsole',
   'scrapy.extensions.logstats.LogStats']
  2017-04-15 21:50:54 [scrapy.middleware] INFO: Enabled downloader middlewares:
  ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
   'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
   'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
   'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
   'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
   'scrapy.downloadermiddlewares.retry.RetryMiddleware',
   'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
   'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
   'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
   'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
   'scrapy.downloadermiddlewares.stats.DownloaderStats']
  2017-04-15 21:50:54 [scrapy.middleware] INFO: Enabled spider middlewares:
  ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
   'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
   'scrapy.spidermiddlewares.referer.RefererMiddleware',
   'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
   'scrapy.spidermiddlewares.depth.DepthMiddleware']
  2017-04-15 21:50:54 [scrapy.middleware] INFO: Enabled item pipelines:
  []
  2017-04-15 21:50:54 [scrapy.core.engine] INFO: Spider opened
  2017-04-15 21:50:54 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2017-04-15 21:50:54 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
  2017-04-15 21:50:55 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)
  2017-04-15 21:50:55 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)
  2017-04-15 21:50:55 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/u011781521/article/details/70182815>
  {'desc': [],
   'link': [],
   'title': []}
  2017-04-15 21:50:55 [scrapy.core.engine] INFO: Closing spider (finished)
  2017-04-15 21:50:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 467,
   'downloader/request_count': 2,
   'downloader/request_method_count/GET': 2,
   'downloader/response_bytes': 17315,
   'downloader/response_count': 2,
   'downloader/response_status_count/200': 2,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2017, 4, 15, 13, 50, 55, 392037),
   'item_scraped_count': 1,
   'log_count/DEBUG': 4,
   'log_count/INFO': 7,
   'response_received_count': 2,
   'scheduler/dequeued': 1,
   'scheduler/dequeued/memory': 1,
   'scheduler/enqueued': 1,
   'scheduler/enqueued/memory': 1,
   'start_time': datetime.datetime(2017, 4, 15, 13, 50, 54, 786607)}
  2017-04-15 21:50:55 [scrapy.core.engine] INFO: Spider closed (finished)
  
  G:\Scrapy_work\myfendo>

  
  
  

  四、保存爬取到的数据
  

  
  

  最简单存储爬取的数据的方式是使用 Feed exports:
  

  
  

  scrapy crawl myfendos -o items.json

  
  

  输出的结果:

  
  

  G:\Scrapy_work\myfendo>scrapy crawl myfendos -o items.json
  2017-04-15 22:19:02 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: myfendo)
  2017-04-15 22:19:02 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'FEED_FORMAT': 'json', 'FEED_URI': 'items.json', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
  2017-04-15 22:19:02 [scrapy.middleware] INFO: Enabled extensions:
  ['scrapy.extensions.corestats.CoreStats',
   'scrapy.extensions.telnet.TelnetConsole',
   'scrapy.extensions.feedexport.FeedExporter',
   'scrapy.extensions.logstats.LogStats']
  2017-04-15 22:19:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
  ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
   'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
   'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
   'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
   'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
   'scrapy.downloadermiddlewares.retry.RetryMiddleware',
   'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
   'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
   'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
   'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
   'scrapy.downloadermiddlewares.stats.DownloaderStats']
  2017-04-15 22:19:03 [scrapy.middleware] INFO: Enabled spider middlewares:
  ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
   'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
   'scrapy.spidermiddlewares.referer.RefererMiddleware',
   'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
   'scrapy.spidermiddlewares.depth.DepthMiddleware']
  2017-04-15 22:19:03 [scrapy.middleware] INFO: Enabled item pipelines:
  []
  2017-04-15 22:19:03 [scrapy.core.engine] INFO: Spider opened
  2017-04-15 22:19:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2017-04-15 22:19:03 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
  2017-04-15 22:19:03 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)
  2017-04-15 22:19:04 [scrapy.core.engine] DEBUG: Crawled (200)  (referer: None)
  2017-04-15 22:19:04 [scrapy.core.scraper] DEBUG: Scraped from <200 http://blog.csdn.net/u011781521/article/details/70182815>
  {'desc': [''],
   'link': ['https://scrapy.org/'],
   'title': ['\r\n        Python爬虫系列之----Scrapy(一)爬虫原理            \r\n        ']}
  2017-04-15 22:19:04 [scrapy.core.engine] INFO: Closing spider (finished)
  2017-04-15 22:19:04 [scrapy.extensions.feedexport] INFO: Stored json feed (1 items) in: items.json
  2017-04-15 22:19:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 467,
   'downloader/request_count': 2,
   'downloader/request_method_count/GET': 2,
   'downloader/response_bytes': 17313,
   'downloader/response_count': 2,
   'downloader/response_status_count/200': 2,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2017, 4, 15, 14, 19, 4, 130548),
   'item_scraped_count': 1,
   'log_count/DEBUG': 4,
   'log_count/INFO': 8,
   'response_received_count': 2,
   'scheduler/dequeued': 1,
   'scheduler/dequeued/memory': 1,
   'scheduler/enqueued': 1,
   'scheduler/enqueued/memory': 1,
   'start_time': datetime.datetime(2017, 4, 15, 14, 19, 3, 323475)}
  2017-04-15 22:19:04 [scrapy.core.engine] INFO: Spider closed (finished)
  
  G:\Scrapy_work\myfendo>

  
  

  该命令将采用 JSON 格式对爬取的数据进行序列化，生成 items.json 文件。
  

  
  

  也可以采用下面的方法保存JSON数据:

  
  

  1.修改pipelines.py文件

  
  

  from scrapy import signals
  import json
  import codecs
  
  
  class JsonWithEncodingCnblogsPipeline(object):
      def __init__(self):
          self.file = codecs.open('myitems.json', 'w', encoding='utf-8')
      def process_item(self, item, spider):
          line = json.dumps(dict(item), ensure_ascii=False) + "\n"
          self.file.write(line)
          return item
      def spider_closed(self, spider):
          self.file.close()

  
  注意类名为JsonWithEncodingCnblogsPipeline哦！settings.py中会用到
  

  
  

  2.修改settings.py，添加以下两个配置项
  

  
  

  ITEM_PIPELINES = {
      'myfendo.pipelines.JsonWithEncodingCnblogsPipeline': 300,
  }
  LOG_LEVEL = 'INFO'

  
  

  然后运行爬虫:

  
  

  scrapy crawl myfendos 

  
  

  输出结果为:

  
  

  G:\Scrapy_work\myfendo>scrapy crawl myfendos
  2017-04-15 22:24:42 [scrapy.utils.log] INFO: Scrapy 1.3.3 started (bot: myfendo)
  2017-04-15 22:24:42 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'myfendo', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'myfendo.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['myfendo.spiders']}
  2017-04-15 22:24:42 [scrapy.middleware] INFO: Enabled extensions:
  ['scrapy.extensions.corestats.CoreStats',
   'scrapy.extensions.telnet.TelnetConsole',
   'scrapy.extensions.logstats.LogStats']
  2017-04-15 22:24:42 [scrapy.middleware] INFO: Enabled downloader middlewares:
  ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
   'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
   'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
   'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
   'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
   'scrapy.downloadermiddlewares.retry.RetryMiddleware',
   'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
   'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
   'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
   'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
   'scrapy.downloadermiddlewares.stats.DownloaderStats']
  2017-04-15 22:24:42 [scrapy.middleware] INFO: Enabled spider middlewares:
  ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
   'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
   'scrapy.spidermiddlewares.referer.RefererMiddleware',
   'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
   'scrapy.spidermiddlewares.depth.DepthMiddleware']
  2017-04-15 22:24:42 [scrapy.middleware] INFO: Enabled item pipelines:
  ['myfendo.pipelines.JsonWithEncodingCnblogsPipeline']
  2017-04-15 22:24:42 [scrapy.core.engine] INFO: Spider opened
  2017-04-15 22:24:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2017-04-15 22:24:43 [scrapy.core.engine] INFO: Closing spider (finished)
  2017-04-15 22:24:43 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 467,
   'downloader/request_count': 2,
   'downloader/request_method_count/GET': 2,
   'downloader/response_bytes': 17315,
   'downloader/response_count': 2,
   'downloader/response_status_count/200': 2,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2017, 4, 15, 14, 24, 43, 434983),
   'item_scraped_count': 1,
   'log_count/INFO': 7,
   'response_received_count': 2,
   'scheduler/dequeued': 1,
   'scheduler/dequeued/memory': 1,
   'scheduler/enqueued': 1,
   'scheduler/enqueued/memory': 1,
   'start_time': datetime.datetime(2017, 4, 15, 14, 24, 42, 660934)}
  2017-04-15 22:24:43 [scrapy.core.engine] INFO: Spider closed (finished)
  
  G:\Scrapy_work\myfendo>

  
  
  会生成一个myitems.json的文件

  
  

  
  
  

  完整的示例: http://download.csdn.net/detail/u011781521/9815346