【Scrapy实战】爬取链家租房信息

Scrapy爬虫创建

1.创建scrapy项目

scrapy startproject LianJia

2.创建scrapy爬虫

cd LianJia

scrapy genspider lianjia lianjia.com

链家网站分析

1. 获取爬取的 start_urls

决定爬取北京海淀区的全部租房信息
设置 start_urls = ['https://bj.lianjia.com/zufang/haidian/']

2.设置爬取的域名 allowed_domains

设置 allowed_domains = ['lianjia.com']

3.分析链家页面的地址

海淀租房第1页网址：https://bj.lianjia.com/zufang/haidian/
海淀租房第2页网址：https://bj.lianjia.com/zufang/haidian/pg2/#contentList
海淀租房第10也网址：https://bj.lianjia.com/zufang/haidian/pg10/#contentList

测试第1页网址为：https://bj.lianjia.com/zufang/haidian/pg1
测试第2页网址为：https://bj.lianjia.com/zufang/haidian/pg2

4.构建页面地址

basic_url = 'https://bj.lianjia.com/zufang/haidian/pg'
有页面地址为 url = basic_url + str(pg)

'''页面请求如下'''
for pg in range(1, 98):
    url = basic_url + str(pg)
    yield scrapy.Request(url, callback=self.parse)

5.获取详情页地址

检查页面，分析详情页地址

检查页面，分析详情页地址

分析，得到每一个页面的详情页的 detail_urls 的 xpath

detail_urls = response.xpath('//div[@id="content"]/div[1]/div[1]//div/div/p[1]/a/@href').extract()

查看详情页，检查需要爬取的数据

• 标题信息

检查详情页标题

得到详情页标题的 xpath

item['title'] = response.xpath('//div/p[@class="content__title"]/text()').extract_first()

• 支付方式，以及价格

检查详情页的价格及支付方式

得到详情页支付方式和价格的 xpath

item['pay'] = response.xpath('//*[@id="aside"]/p[1]/text()').extract_first()
item['price'] = response.xpath('//*[@id="aside"]/p[1]/span/text()').extract_first()

• 房子的租赁方式

检查详情页的租赁方式

得到详情页租赁方式的 xpath

item['house'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[1]/text()').extract_first()

• 户型

检查详情页的户型

得到详情页的户型的 xpath

item['type'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[2]/text()').extract_first()

• 房子的面积

检查详情页的面积

得到详情页的面积的 xpath

item['area'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[3]/text()').extract_first()

配置Scrapy爬虫

• settings.py

BOT_NAME = 'LianJia'

SPIDER_MODULES = ['LianJia.spiders']
NEWSPIDER_MODULE = 'LianJia.spiders'

# User-Agent
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'

# Robot协议，不遵守
ROBOTSTXT_OBEY = False

# 配置管道，用于下载
ITEM_PIPELINES = {
   'LianJia.pipelines.LianjiaPipeline': 300,
}

• items.py

import scrapy

class LianjiaItem(scrapy.Item):
    title = scrapy.Field()
    pay = scrapy.Field()
    price = scrapy.Field()
    house = scrapy.Field()
    type = scrapy.Field()
    area = scrapy.Field()
    pass

• pipelines.py

from scrapy.exporters import CsvItemExporter

class LianjiaPipeline(object):
    def open_spider(self, spider):
        self.file = open("lianjia.csv", 'ab')
        self.exporter = CsvItemExporter(self.file)
        self.exporter.start_exporting()

    def process_item(self, item, spider):
        self.exporter.export_item(item)
        return item

    def close_spider(self, spider):
        self.exporter.finish_exporting()
        self.file.close()

• middlewares.py（这里不用配置 middlewares.py）

• lianjia.py（爬虫代码）

# -*- coding: utf-8 -*-
import scrapy
from LianJia.items import LianjiaItem


class LianjiaSpider(scrapy.Spider):
    name = 'lianjia'
    allowed_domains = ['lianjia.com']
    start_urls = ['https://bj.lianjia.com/zufang/haidian/pg1']

    basic_url = 'https://bj.lianjia.com/zufang/haidian/pg'
    basic_detail_url = 'https://bj.lianjia.com'

    def parse(self, response):
        detail_urls = response.xpath('//div[@id="content"]/div[1]/div[1]//div/div/p[1]/a/@href').extract()
        for detail_url in detail_urls:
            detail_url = self.basic_detail_url + detail_url
            yield scrapy.Request(detail_url, callback=self.parse_detail)

        for pg in range(2, 98):
            url = self.basic_url + str(pg)
            yield scrapy.Request(url, callback=self.parse)

    def parse_detail(self, response):
        item = LianjiaItem()
        item['title'] = response.xpath('//div/p[@class="content__title"]/text()').extract_first()
        item['pay'] = response.xpath('//*[@id="aside"]/p[1]/text()').extract_first()
        item['price'] = response.xpath('//*[@id="aside"]/p[1]/span/text()').extract_first()
        item['house'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[1]/text()').extract_first()
        item['type'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[2]/text()').extract_first()
        item['area'] = response.xpath('//*[@id="aside"]/ul[1]/p/span[3]/text()').extract_first()
        yield item

运行爬虫

scrapy crawl lianjia

得到 lianjia.csv文件

lianjia.csv