Scrapy爬取多页数据

一、创建项目

打开pycharm下面的Terminal窗口

scrapy startproject 项目名

例如：scrapy startproject crawler51job

二、定义要爬取的数据  

编写items文件

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class Crawler51JobItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    position = scrapy.Field()  # 职位
    company = scrapy.Field()  # 公司名
    place = scrapy.Field()  # 工作地点
    salary = scrapy.Field()  # 薪资

三、创建并编写爬虫文件

1-创建爬虫文件：

在Terminal窗口中

scrapy genspider -t basic 爬虫文件名 域名

 

 2-编写爬虫文件

【注意】allowed_domains是指要爬取的网址的域名。start_urls是指爬取的起始网页

在parse函数中编写代码

要先导入items文件中的Crawler51JobItem

from crawler51job.items import Crawler51JobItem

# -*- coding: utf-8 -*-
import scrapy
from crawler51job.items import Crawler51JobItem
from scrapy.http import Request


class Spider51jobSpider(scrapy.Spider):
    name = 'spider51job'
    allowed_domains = ['51job.com']
    start_urls = [
        'https://search.51job.com/list/010000,000000,0000,32,9,99,Java%25E5%25BC%2580%25E5%258F%2591,2,1.html']

    def parse(self, response):
        item = Crawler51JobItem()
        # 利用xpath提取网页信息
        item['position'] = response.xpath('//div[@class="el"]/p[@class="t1 "]/span/a/@title').extract()
        item['company'] = response.xpath('//div[@class="el"]/span[@class="t2"]/a/@title').extract()
        item['place'] = response.xpath('//div[@class="el"]/span[@class="t3"]/text()').extract()
        item['salary'] = response.xpath('//div[@class="el"]/span[@class="t4"]/text()').extract()
        yield item

        for i in range(2, 4):  # 爬取第2，3页的数据
            url = "https://search.51job.com/list/010000,000000,0000,32,9,99,Java%25E5%25BC%2580%25E5%258F%2591,2," + str(i) + ".html"
            yield Request(url, callback=self.parse)

  编写pipelines文件（主要用于对这些item进行处理）

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
from pandas import DataFrame
import pandas as pd


class Crawler51JobPipeline(object):
    def __init__(self):
        self.jobinfoAll = DataFrame()

    def process_item(self, item, spider):
        jobInfo = DataFrame([item['position'], item['company'], item['place'], item['salary']]).T
        jobInfo.columns = ['职位名', '公司名', '工作地点', '薪资']
        self.jobinfoAll = pd.concat([self.jobinfoAll, jobInfo])
        self.jobinfoAll.to_csv('jobinfoAll.csv', encoding='utf-8')  # 设置编码格式，防止乱码
        return item

编写settings文件（settings文件为爬虫项目的设置文件，主要是爬虫项目的一些设置信息。）

例如，启用了pipelines，需要把settings中相关代码的注释取消

四、爬虫运行及调试

在Terminal

scrapy crawl 爬虫文件名

如：scrapy crawl spider51job

如果运行出错：ModuleNotFoundError: No module named 'win32api'

则需要安装pywin32,去官网下载合适的版本

https://sourceforge.net/projects/pywin32/files/pywin32/Build%20221/