如何爬取链家小区成交数据——Ruia异步爬虫框架笔记

Ruia是一个基于asyncio和aiohttp的异步爬虫框架，它的目标是让你更加方便且迅速地编写出属于自己的爬虫。编写的方式与Scrapy十分相似，同时支持Xpath和CssSelector等多种页面解析方式，易于上手。作者的Github与知乎首页可以进去交流学习。

• github：

  https://github.com/howie6879/ruia

• 知乎：

  https://www.zhihu.com/people/howie6879

01

Python环境安装

1. 下载安装Anaconda镜像，国内推荐清华大学的镜像地址，选择下载Python 3.7版本，镜像地址： 

   https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/ 

   

2. 安装mongo数据库，这里推荐一个教程，可以将mongo注册成windows服务。教程地址：

   https://www.runoob.com/mongodb/mongodb-window-install.html

   

3. 安装ruia，pip install -U ruia

4. 安装motor，pip install -U motor

02

定义爬虫请求

1. 选择爬取链家上海浦东地区的小区，页面上可以看到小区近期的二手房成交数量，出租房屋数量，成交均价，在售数量，建成时间，地段特点等信息

   

2. 定义爬虫请求，需要继承ruia中的Spider类

   

3. 定义数据解析，ruia支持使用css_select与xpath_select来选择页面元素

   

4. 数据解析入库

   

   

5. 完整代码如下：

   from ruia import AttrField, TextField, Item, Spider, Response
   from ruia_ua import middleware
   import asyncio
   from motor.motor_asyncio import AsyncIOMotorClient
   
   
   
   
   class MotorBase:
       _db = {}
       _collection = {}
   
   
       def __init__(self, loop=None):
           self.motor_uri = ""
           self.loop = loop or asyncio.get_event_loop()
   
   
       def client(self, db):
           # motor
           self.motor_uri = f"mongodb://localhost:27017/{db}"
           return AsyncIOMotorClient(self.motor_uri, io_loop=self.loop)
   
   
       def get_db(self, db="test"):
           """
           Get a db instance
           :param db: database name
           :return: the motor db instance
           """
           if db not in self._db:
               self._db[db] = self.client(db)[db]
   
   
           return self._db[db]
   
   
   
   
   class InfoItem(Item):
       target_item = TextField(css_select='li[class="clear xiaoquListItem"]')
       title = TextField(css_select='div.title')
       href = AttrField(css_select='div.title>a', attr='href')
       esf_sale = TextField(css_select='a[title$="网签"]')
       hse_rent = TextField(css_select='a[title$="租房"]')
       district = TextField(css_select='a[class="district"]')
       bizcircle = TextField(css_select='a[class="bizcircle"]')
       buildtime = TextField(css_select='div[class="positionInfo"]')
       totalPrice = TextField(css_select='div[class="totalPrice"]')
       sellCount = TextField(css_select='a[class="totalSellCount"]')
       taglist = TextField(css_select='div.tagList')
   
   
   
   
   class Community(Spider):
       # 设置启动URL
       start_urls = ['https://sh.lianjia.com/xiaoqu/pudong/']
       # 爬虫模拟请求的配置参数
       request_config = {
           'RETRIES': 5,
           'DELAY': 1,
           'TIMEOUT': 20
       }
       concurrency = 10
   
   
       async def parse(self, response: Response):
           self.mongo_db = MotorBase().get_db("ruia_test")
           urls = [self.start_urls[0]+'pg{}/'.format(str(i)) for i in range(1, 101)]
           async for response in self.multiple_request(urls, is_gather=True):
               yield self.parse_page(response)
   
   
       async def parse_page(self, response):
           async for item in InfoItem.get_items(html=response.html):
               yield item
   
   
       async def process_item(self, item: InfoItem):
           buildtime = item.buildtime[len(item.district)+len(item.bizcircle)+1:].strip()
   
   
           try:
               await self.mongo_db.pudong.update_one(
                   {"url": item.href},
                   {"$set": {"url": item.href,
                             "title": item.title,
                             "esf_sale": item.esf_sale,
                             "hse_rent": item.hse_rent,
                             "district": item.district,
                             "bizcircle": item.bizcircle,
                             "buildtime": buildtime,
                             "totalPrice": item.totalPrice,
                             "sellCount": item.sellCount,
                             "taglist": item.taglist}},
                   upsert=True,
               )
           except Exception as e:
               self.logger.exception(e)
   
   
   
   
   if __name__ == "__main__":
       Community.start(middleware=middleware)
   

03

使用Mongo Compass查看数据

1. 使用建成年份过滤

   

2. 使用商圈位置过滤

   
   

在看点这里