基于scrapy框架的爬虫项目（近乎完美的可视化以及分词呈现）
python爬虫项目（完整项目流程以及源码分享）
注：本网站采集的数据来源于51job.com，仅做学习参考
项目流程大体概述：
友情提示：之后的源代码文件顺序按照如下次序排列。
1.首先，分析网站，明确采集数据，创建scrapy框架环境，编写爬虫主程序；
（本文为：jobspider.py文件）
2.开辟临时存储空间，将主程序爬取的数据进行临时存储（类似于字典封装）；
（本文为：items.py文件）
3.通过管道文件获取存储空间中的采集信息，并输出打印到控制台（类似于获取value值的操作）；
（本文为：pipelines.py管道文件）
4.更改初始的settings.py文件设置；
（本文为：settings.py文件）
5.编写启动爬虫程序（目的是为了更友好的采集多个不同职位的数据），之后还包括向jobpositiondao.py文件中传入采集记录的方法；
（本文为：startspider.py文件）
6.创建dao包，在其中创建pymysql.json配置文件，存储连接数据库的基本信息；
（本文为：pymysql.json文件）
7.在dao包中创建连接mysql数据库的通用主程序文件；
（本文为：basedao.py文件）
8.在dao包中创建写入（方法）以及查询（方法）数据库中数据的文件；
（本文为：jobpositiondao.py文件）
9.在dao包中创建将用户采集记录写入到数据库的方法文件；
（本文为：taskdao.py文件）
10.数据的优化处理以及向jobpositiondao.py文件中传入采集数据的方法（数据来源于临时存储空间中主程序所采集的数据）；
（本文为：mysqlpipelines.py管道文件）
11.编写启动sql查询以及进行可视化显示的文件；
（本文为：showstatics.py文件）
如下为最终可视化效果以及项目源码：
jobspider.py文件，代码如下：
# -*- coding: utf-8 -*-
import scrapy
from ..items import SpiderprojectItem
class JobspiderSpider(scrapy.Spider):
name = 'jobspider'
# allowed_domains = ['www.baidu.com']
start_urls = []
def __init__(self, start_urls=None, taskid=0, *args, **kwargs):
super(JobspiderSpider, self).__init__(*args, **kwargs)
# print(start_urls)
# print(taskid)
self.start_urls.append(start_urls)
self.taskid = int(taskid)
pass
def parse(self, response):
jobItems=response.xpath("//div[@class = 'el']") #返回xpath的选择器列表结果
#遍历选择器列表
jobLen = len(jobItems)
jobCount = 0
# 分页
nextURL = response.xpath("//li[@class='bk']/a/@href").extract()
nextText = response.xpath("//li[@class='bk']/a/text()").extract()
realURL = ""
if nextURL and nextText[-1].strip()=="下一页":
# if nextText[-1].strip()=="下一页":
realURL = response.urljoin(nextURL[-1])
# print('url',url)
pass
pass
for jobItem in jobItems:
jobCount += 1
sItem = SpiderprojectItem()
sItem['taskId']=self.taskid
#extract()解析职位 strip()去掉多余空格
jobposition = jobItem.xpath("p/span/a/text()")
if jobposition:
sItem['jobposition']=jobposition.extract()[0].strip() #相当于字典
#取链接
positionDetail = jobItem.xpath("p[@class='t1 ']/span/a/@href")#返回的是选择器
#解析公司名称
jobCompany = jobItem.xpath("span[@class='t2']/a/text()")
if jobCompany:
sItem['jobCompany'] = jobCompany.extract()[0].strip()
#解析公司地点
jobAddress = jobItem.xpath("span[@class='t3']/text()")
if jobAddress:
sItem['jobAddress'] = jobAddress.extract()[0].strip()
#解析月薪
jobMMoney = jobItem.xpath("span[@class='t4']/text()")
if jobMMoney:
sItem['jobMMoney'] = jobMMoney.extract()[0].strip()
#解析发布日期
FBTime = jobItem.xpath("span[@class='t5']/text()")
if FBTime:
sItem['FBTime'] = FBTime.extract()[0].strip()
if jobposition and jobCompany and jobAddress and jobMMoney and FBTime and positionDetail:
detailURL = positionDetail.extract()[0]
# print(detailURL)
sItem['nextURL'] = realURL
# 访问二级页面
yield scrapy.Request(url=detailURL, callback=self.parse_detail,meta={'item':sItem,'jobLen':jobLen, 'jobCount':jobCount},dont_filter=True)
pass
pass
#将解析的数据写到数据库（用管道）
#实现分页爬取
#定义爬取详情(点进去)页的方法
def parse_detail(self, response):
sItem = response.meta['item']
jobLen = response.meta['jobLen']
jobCount = response.meta['jobCount']
detailData = response.xpath("//div[@class='bmsg job_msg inbox']")
print('detailData：', detailData)
if detailData:
contents=detailData.xpath('//p/text()') #返回当前选择器
ct = ""
if contents:
for temp in contents.extract():
if temp.strip()=="" or temp.strip()=="/":
continue
ct += temp +"\n"
sItem['jobDetail'] = ct
yield sItem #保顺序
pass
# 判断当前页是否爬取完成了，完成就继续爬取下一页
if jobLen == jobCount:
if sItem['nextURL']:
yield scrapy.Request(sItem['nextURL'], self.parse, dont_filter=False)
pass
pass
pass
items.py文件，代码如下：
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
#数据封装的实体类
class SpiderprojectItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
jobposition = scrapy.Field()
jobCompany = scrapy.Field()
jobAddress = scrapy.Field()
jobMMoney = scrapy.Field()
FBTime = scrapy.Field()
nextURL = scrapy.Field()
jobDetail = scrapy.Field() #存放详情页
taskId = scrapy.Field()
pass
pipelines.py管道文件，代码如下：
class SpiderprojectPipeline(object):
def process_item(self, item, spider):
print("通过管道输出数据")
print(item['jobposition'])
print(item['jobCompany'])
print(item['jobAddress'])
print(item['jobMMoney'])
print(item['FBTime'])
print(item['jobDetail'])
return item
settings.py文件，将如下内容前的注释（#）去掉即可：
ROBOTSTXT_OBEY = False #注意，去掉注释符号#后，将原来的值改为False
DOWNLOAD_DELAY = 5 #注意，去掉注释符号#后，将原来的3改为5
SPIDER_MIDDLEWARES = {
'spidermovieproject.middlewares.SpidermovieprojectSpiderMiddleware': 543,
}
DOWNLOADER_MIDDLEWARES = {
'spidermovieproject.middlewares.SpidermovieprojectDownloaderMiddleware': 543,
}
ITEM_PIPELINES = {
'spidermovieproject.pipelines.SpidermovieprojectPipeline': 300,
}
startspider.py文件，代码如下：
#此脚本是爬虫启动脚本(不用去cmd输入命令)
from scrapy.cmdline import execute
from Include.day022.spiderproject.spiderproject.dao.taskdao import TaskDao
#启动爬虫
td = TaskDao()
result,taskId = td.create(('Python职位数据采集','https://search.51job.com/list/000000,000000,0000,00,9,99,python,2,2.html?lang=c&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare= '))
if result>0:
execute(['scrapy','crawl','jobspider',
'-a','start_urls=https://search.51job.com/list/000000,000000,0000,00,9,99,python,2,2.html?lang=c&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare= ',
'-a','taskid='+str(taskId)])
# result,taskId = td.create(('Java职位数据采集','https://search.51job.com/list/000000,000000,0000,00,9,99,java,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='))
#
# if result>0:
# execute(['scrapy','crawl','jobspider',
# '-a','start_urls=https://search.51job.com/list/000000,000000,0000,00,9,99,java,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=',
# '-a','taskid='+str(taskId)])
# result,taskId = td.create(('C++职位数据采集','https://search.51job.com/list/000000,000000,0000,00,9,99,c%252B%252B,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='))
# if result>0:
# execute(['scrapy','crawl','jobspider',
# '-a','start_urls=https://search.51job.com/list/000000,000000,0000,00,9,99,c%252B%252B,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=',
# '-a','taskid='+str(taskId)])
# result,taskId = td.create(('js职位数据采集','https://search.51job.com/list/000000,000000,0000,00,9,99,js,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='))
#
# if result>0:
# execute(['scrapy','crawl','jobspider',
# '-a','start_urls=https://search.51job.com/list/000000,000000,0000,00,9,99,js,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=',
# '-a','taskid='+str(taskId)])
# result,taskId = td.create(('php职位数据采集','https://search.51job.com/list/000000,000000,0000,00,9,99,php,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='))
#
# if result>0:
# execute(['scrapy','crawl','jobspider',
# '-a','start_urls=https://search.51job.com/list/000000,000000,0000,00,9,99,php,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=',
# '-a','taskid='+str(taskId)])
pymysql.json文件，代码样式如下：
{"host":"127.0.0.1","user":"你的数据库的用户名","password":"你的数据库的密码", "database" :"db_movie_data","port":3306}
basedao.py连接数据库文件，代码如下：
import pymysql
import json
import logging
import sys
import os
class BaseDao():
def __init__(self,configPath = 'pymysql.json'):
self.__connection =None
self.__cursor = None
self.__config = json.load(open(os.path.dirname(__file__) + os.sep + configPath,'r')) #通过json配置获得数据的连接配置信息
print(self.__config)
pass
def getConnection(self):
#当有连接对象时，直接返回连接对象
if self.__connection:
return self.__connection
#否则通过建立新的连接对象
try:
self.__connection = pymysql.connect(**self.__config)
return self.__connection
except pymysql.MySQLError as e:
print("Exception"+str(e))
pass
pass
#用于执行Sql语句的通用方法
def execute(self,sql,params):
try:
self.__cursor = self.getConnection().cursor()
#execute:返回的是修改数据的条数
if params:
result = self.__cursor.execute(sql,params)
else:
result = self.__cursor.execute(sql)
return result
except (pymysql.MySQLError,pymysql.DatabaseError,Exception) as e:#捕获多个异常
print("出现数据库访问异常："+str(e))
self.rollback()
pass
pass
pass
def fetch(self):
if self.__cursor: #提高代码健壮性
return self.__cursor.fetchall()
pass
def commit(self):
if self.__connection:
self.__connection.commit() #回滚问题
pass
def rollback(self):
if self.__connection:
self.__connection.rollback()
pass
def getLastRowId(self):
if self.__cursor:
return self.__cursor.lastrowid
def close(self):
if self.__cursor:
self.__cursor.close()
if self.__connection:
self.__connection.close()
pass
if __name__=="__main__":
ms = BaseDao()
jobpositiondao.py数据库读写文件，代码如下：
from .basedao import BaseDao # . 代表当前目录
#定义一个职位数据操作的数据库访问类
class JobPositionDao(BaseDao):
def __init__(self):
super().__init__()
#向数据库插入职位信息
def create(self,params):
sql = "insert into job_position (jobposition,jobCompany,jobAddress,job_MMoney,FBTime,job_taskid,job_lowsalary,job_highsalary,job_meansalary,job_city) values(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"
result = self.execute(sql,params)
lastRowId = self.getLastRowId()
self.commit()
return result,lastRowId
pass
def createDetail(self,params):
sql = "insert into job_position_detail (detail_desciption,detail_positionid) values (%s,%s)"
result = self.execute(sql,params)
self.commit()
return result
pass
pass
def findPositionClassify(self):
sql = "select avg(job_meansalary),job_taskid,task_title from job_position,job_collect_task where job_position.job_taskid = job_collect_task.task_id group by job_taskid,task_title;"
result = self.execute(sql,params=None)
self.commit()
return self.fetch()
def findCityPositionClassify(self):
sql = "select avg(t1.job_meansalary) as m,t1.job_taskid,t2.task_title,t1.job_city from job_position t1 left join job_collect_task t2 on t1.job_taskid = t2.task_id group by job_taskid,job_city,t2.task_title order by t1.job_taskid asc,m desc;"
result = self.execute(sql,params=None)
self.commit()
return self.fetch()
pass
def findPositionDetail(self): #查询各职位具体内容
sql = "select detail_desciption from job_position_detail"
result = self.execute(sql,params=None)
self.commit()
return self.fetch()
pass
taskdao.py数据库写入采集记录文件，代码如下：
from .basedao import BaseDao
class TaskDao(BaseDao):
def create(self,params):
sql = "insert into job_collect_task (task_title,task_url) values (%s,%s)"
result = self.execute(sql,params)
lastRowId = self.getLastRowId()
self.commit()
self.close()
return result,lastRowId
pass
mysqlpipelines.py数据库管道文件，代码如下：
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
from .dao.jobpositiondao import JobPositionDao
class SpidermysqlPipeline(object):
def process_item(self, item, spider):
jobPositionDao = JobPositionDao()
try:
jobAddress = item['jobAddress']
jobMMoney = item['jobMMoney']
lowSalary = 0
highSalary = 0
meanSalary = 0
#根据 地址信息 拆分出 城市信息 例：北京市-海淀区
jobCity = jobAddress.split('-')[0]
#处理薪资数据 1.判断单位
if jobMMoney.endswith('万/月'):
jobMMoney = jobMMoney.replace('万/月','')
if jobMMoney.find('-'):
MoneyArray = jobMMoney.split('-')
lowSalary = float(MoneyArray[0])*10000
highSalary = float(MoneyArray[1])*10000
pass
else:
lowSalary = highSalary =float(jobMMoney)*10000
meanSalary = (lowSalary + highSalary)/2
pass
elif jobMMoney.endswith('千/月'):
jobMMoney = jobMMoney.replace('千/月', '')
if jobMMoney.find('-'):
MoneyArray = jobMMoney.split('-')
lowSalary = float(MoneyArray[0]) * 1000
highSalary = float(MoneyArray[1]) * 1000
pass
else:
lowSalary = highSalary = float(jobMMoney) * 1000
meanSalary = (lowSalary + highSalary) / 2
elif jobMMoney.endswith('万/年'):
jobMMoney = jobMMoney.replace('万/年', '')
if jobMMoney.find('-'):
MoneyArray = jobMMoney.split('-')
lowSalary = float(MoneyArray[0]) * 10000/12
highSalary = float(MoneyArray[1]) * 10000/12
pass
else:
lowSalary = highSalary = float(jobMMoney) * 10000/12
meanSalary = (lowSalary + highSalary) / 2
pass
elif jobMMoney.endswith('元/天'):
jobMMoney = jobMMoney.replace('元/天', '')
if jobMMoney.find('-'):
MoneyArray = jobMMoney.split('-')
lowSalary = float(MoneyArray[0]) * 22
highSalary = float(MoneyArray[1]) * 22
pass
else:
lowSalary = highSalary = float(jobMMoney) * 22
meanSalary = (lowSalary + highSalary) / 2
pass
else:
return
result,lastRowId=jobPositionDao.create((item['jobposition'],item['jobCompany'],item['jobAddress'],item['jobMMoney'],item['FBTime'],item['taskId'],lowSalary,highSalary,meanSalary,jobCity))
if result:
jobPositionDao.createDetail((item['jobDetail'],lastRowId))
pass
except Exception as e:
print(e)
finally:
jobPositionDao.close()
return item
showstatics.py是sql查询以及可视化文件，具体代码如下：
from Include.day022.spiderproject.spiderproject.dao.jobpositiondao import JobPositionDao
import numpy as np
import matplotlib.pyplot as plt
jp = JobPositionDao()
# print(jp.findPositionClassify())
# print(jp.findCityPositionClassify())
# 两行代码支持显示中文
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False
#分块代码：
fig = plt.figure()
subplot = fig.add_subplot(2,2,1)
#------绘制各职位平均月薪条状图（第一个子图）---------------
Result_fnPosCla = np.array(jp.findPositionClassify())
Result_fnPosCla = Result_fnPosCla.T
# print(Result_fnPosCla)
xlabel = Result_fnPosCla[2]
x = Result_fnPosCla[1].astype(np.float) #刻度
y = Result_fnPosCla[0].astype(np.float) #工资
bars = subplot.bar(x,y,width=0.3)
subplot.set_xlabel("热门语言职位分类")
subplot.set_ylabel("月薪资")
subplot.set_xticks(x)
subplot.set_xticklabels(xlabel)
subplot.grid(linestyle="--")
for x,y in zip(x,y):
subplot.text(x,y+0.05,'{0}元/月'.format(np.floor(y)),ha = 'center',va = 'bottom')
#--------------绘制折线图（第二个子图）---------------------------
subplot2 = fig.add_subplot(2,2,2)
Result_fnCiPosClas = np.array(jp.findCityPositionClassify())
# print(Result_fnCiPosClas)
#准备数据
Py = [temp for temp in Result_fnCiPosClas if temp[1] == '1']
Py=np.array(Py)
# print(Py)
Ja= [temp for temp in Result_fnCiPosClas if temp[1] == '2']
Ja=np.array(Ja)
# print(Ja)
Cpp= [temp for temp in Result_fnCiPosClas if temp[1] == '3']
Cpp=np.array(Cpp)
# print(Cpp)
Js= [temp for temp in Result_fnCiPosClas if temp[1] == '4']
Js=np.array(Js)
# print(Js)
datalist = []
for py in Py:
for ja in Ja:
for cpp in Cpp:
for js in Js:
if py[3] == ja[3] == cpp[3] == js[3]:
if py[3]==ja[3] == cpp[3] == js[3]!='异地招聘':
datalist.append([py,ja,cpp,js])
data = np.array(datalist)
data = data.T
print(data)
##绘制多折线图
xlabel = [address for address in data[3,0]]
print(xlabel)
pyData = [Salary.astype(np.float) for Salary in data[0,0]]
jaData = [Salary.astype(np.float) for Salary in data[0,1]]
cppData =[Salary.astype(np.float) for Salary in data[0,2]]
jsData = [Salary.astype(np.float) for Salary in data[0,3]]
subplot2.plot(xlabel,pyData,'bp-',alpha=0.5,label = 'python')
subplot2.plot(xlabel,jaData,'mp-',alpha=0.5,label = 'java')
subplot2.plot(xlabel,cppData,'yp-',alpha=0.5,label = 'c++')
subplot2.plot(xlabel,jsData,'gp-',alpha=0.5,label = 'javascript')
#绘制辅助标签
subplot2.set_xlabel("全国热门IT省份")
subplot2.set_ylabel("平均月薪资/元")
subplot2.legend(loc='best')
subplot2.grid(linestyle="--")
#----------------------第三个子图（饼图）-----------------------
subplot3 = fig.add_subplot(2,2,3)
# Result_fnPosCla = np.array(jp.findPositionClassify())
# Result_fnPosCla = Result_fnPosCla.T
# print(Result_fnPosCla)
##准备数据
jobs = [job for job in Result_fnPosCla[2]]
colorlist = ["green","red","orange","yellow"]
colors = [color for color in colorlist]
#比例处理
Salaries = [salary.astype(np.float) for salary in Result_fnPosCla[0]]
sumSalary = np.sum(Salaries) #各语言的月薪资总和
SalaBlis = [temp.astype(np.float)/sumSalary*100 for temp in Result_fnPosCla[0]] #全国范围内大多数城市各热门编程语言的月薪资比例
# print(SalaBlis)
#绘饼图如下
labels = ["{0}\n{1} %\n{2}元/月".format(job,np.floor(SalaBli*1000)/1000,str(salary).split(".")[0] )for job,SalaBli,salary in zip(jobs,SalaBlis,Result_fnPosCla[0])]
explode = np.where(np.array(jobs)=="Python职位数据采集" ,1,0)
subplot3.pie(SalaBlis,colors=colors,labels=labels,explode=explode)
subplot3.axis('equal')
#-------------------词云以及jieba的使用（第四个子图）---------------------------
#jieba
import jieba
from wordcloud import WordCloud, STOPWORDS
subplot4 = fig.add_subplot(2,2,4)
print("内容")
text = str(jp.findPositionDetail())
with open("job_detail_text.txt","w") as fp: #写成.txt文件
fp.write(text)
signatures = []
with open('job_detail_text.txt', mode='r', encoding='gb18030') as fp:
signatures = fp.read()
# 设置屏蔽词，去除特殊符号以及低价值的分词
stopwords = STOPWORDS.copy()
stopwords.add('xa0')
stopwords.add('xa03')
stopwords.add('n')
stopwords.add('n1')
stopwords.add('n2')
stopwords.add('n3')
stopwords.add('n4')
stopwords.add('n5')
stopwords.add('无忧')
stopwords.add('推荐')
stopwords.add('xa009')
stopwords.add('发布')
stopwords.add('有限公司')
stopwords.add('民营公司')
# 导入背景图
bg_image = plt.imread('xiaoxi.jpg')
# 设置词云参数，参数分别表示：画布宽高、背景颜色、背景图形状、字体、屏蔽词、最大词的字体大小
wc = WordCloud(width=1000,height=800,background_color='white', mask=bg_image, font_path='STKAITI.TTF', stopwords=stopwords,max_font_size=220, random_state=50)
# 将分词后数据传入云图
wc.generate_from_text(words)
subplot4.imshow(wc) # 绘制图像
subplot4.axis('off') # 不显示坐标轴
# 保存结果到本地
wc.to_file('51job词云图.jpg')
plt.show()
jp.close()
如下图所示，为可视化结果：
mysql数据表结构如下（本文章用到了三张数据表）：
job_collect_task数据表：主要存储采集数据记录
job_position数据表：主要存储一级界面的基本信息
job_position_detail数据表：主要存储二级界面的详细信息
以上就是本项目的具体内容，如有纰漏，请多指教 ^ _ ^