爬虫入门(4)-Scrapy框架简单入门

scrapy作为一个强大的爬虫框架，就不多作介绍。
今天刚入门scrapy，所以做个简单的使用

Scrapy中文文档

http://scrapy-chs.readthedocs.io/zh_CN/latest/index.html

Ubuntu安装scrapy

在Ubuntu下安装scrapy，需要的命令为：

pip install zope.interface
pip install twisted
pip install pyopenssl
pip install scrapy

开始

1. 使用scrapy创建文件夹命令：

scrapy startproject doubanTest

命令执行完系统将在当前目录下生成doubanTest文件

2. 看看最后的代码文件目录（开始时创建不存在douban_spider.py，main.py文件）

3. 网页提取的元素需要在items.py定义

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class DoubantestItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field() # 抓取电影标题
    content = scrapy.Field() # 抓取电影简介

比如我在代码中定义了抓取电影的标题和电影的简介

关于item的使用，官方文档这么描述:

Item用法类似字典dict

4. 在settings.py定义一些防反爬元素

# -*- coding: utf-8 -*-

# Scrapy settings for doubanTest project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'doubanTest'

SPIDER_MODULES = ['doubanTest.spiders']
NEWSPIDER_MODULE = 'doubanTest.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36'

FEED_URI = 'doubanmovie.csv' # 将信息存储到doubanmovie.csv文件中
FEED_FORMAT = 'CSV' # 存储的文件类型

5. 在spiders下创建douban_spider.py，编写主要代码

#-*_coding:utf8-*-

import scrapy
from scrapy.spider import Spider
from scrapy.http import Request
from scrapy.selector import Selector
from doubanTest.items import DoubantestItem

class doubanSpider(Spider):
    name="doubanSpider" # 爬虫程序名称
    start_urls = ['https://movie.douban.com/top250'] # URL访问列表

    def parse(self, response):
        # print response.body
        item = DoubantestItem() # 定义一个Item对象
        selector = Selector(response)
        Movies = selector.xpath('//div[@class="info"]')
        for each in Movies:
            title = each.xpath('div[@class="hd"]/a/span/text()').extract() # 获取电影标题
            full_title = "".join(title)
            content = each.xpath('div[@class="bd"]/p/text()').extract() # 获取电影简介

            # item类似dict, 它的key名称必须与items.py里的名称一致
            item['title'] = full_title
            item['content'] = ';'.join(content).strip()
            yield item

6. 在doubanTest目录下创建一个main.py

编写这两句代码，就可以使用pycharm运行了

#-*_coding:utf8-*-
from scrapy import cmdline
cmdline.execute("scrapy crawl doubanSpider".split())

运行main.py得到的结果

ok,就当作scrapy入门例子吧