我的第一个数据分析项目——51job“数据分析”岗位分析（爬虫篇）

自学python断断续续快一年了，从最初看邹琪鲜老师的视频，到参考网上的代码修修改改，再到现在能够根据实际需求写点东西，虽然期间有过较长时间的停滞期，但很高兴自己仍旧走在这条荆棘小路上。

最初我把重心都放在编写爬虫代码上，后来发现如果要走数据分析这条路的话，爬虫仅仅是个辅助工具，真正重要的数据分析思维，所以有了我的第一个数据分析项目。

我在网上找到了一个类似的分析案例，学习并参考，当然还有取长补短。参考案例链接：https://blog.csdn.net/lbship/article/details/79452459

筛选条件

我这里用的关键字是“数据分析师”，因为搜索“数据分析”的显示的岗位太杂，相关性比较差。
搜索地点是“全国”，结果按时间排序显示。

数据分析搜索结果，岗位相关性差

51job爬虫代码

# -*- coding: utf-8 -*-
"""
Created on Fri Sep  7 15:08:19 2018

@author: Shirley
"""
import requests
from lxml import etree
import time
import random
import csv


urllist = []
def Geturl(page):#获取列表链接
    url = "https://search.51job.com/list/000000,000000,0000,00,9,99,数据分析师,2,%d.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%%2C0&radius=-1&ord_field=1&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare="%page
    #关键词“数据分析师”，地点：全国，按时间排序
    resp = requests.get(url)
    try:
        urlcontent = etree.HTML(resp.text,parser=etree.HTMLParser(encoding="gbk"))
        myurls = urlcontent.xpath("//p[@class='t1 ']/span/a/@href")
        for myurl in myurls:
            urllist.append(myurl)
    except:
        print("第%d页没有获取！"%page)
        

data = []
def Getcontent(urllist):#根据链接抓取详细内容
    for myurl in urllist:
        try:
            resp = requests.get(myurl)
            maincontent = etree.HTML(resp.text,parser=etree.HTMLParser(encoding="gbk"))
            job = maincontent.xpath("string(//div[@class='cn']/h1)")
            salary = maincontent.xpath("string(//div[@class='cn']/strong)")
            company = maincontent.xpath("string(//a[@class='catn'])")
            companytype = maincontent.xpath("string(//p[@class='at'][1])")
            companysize = maincontent.xpath("string(//p[@class='at'][2])")
            companyfield = maincontent.xpath("string(//p[@class='at'][3])")
            requirements = maincontent.xpath("string(//p[@class='msg ltype'])")
            welfare = maincontent.xpath("string(//div[@class='t1'])")
            jobdescription = maincontent.xpath("string(//div[@class='bmsg job_msg inbox'])")
            workplace = maincontent.xpath("string(//div[@class='bmsg inbox']/p[@class='fp'])")
            data.append([myurl,job.replace("\t","").encode("gbk","ignore").decode("gbk"),salary.replace("\t","").encode("gbk","ignore").decode("gbk"),company.replace("\t","").encode("gbk","ignore").decode("gbk"),companytype.replace("\t","").encode("gbk","ignore").decode("gbk"),companysize.replace("\t","").encode("gbk","ignore").decode("gbk"),companyfield.replace("\t","").encode("gbk","ignore").decode("gbk"),requirements.replace("\t","").encode("gbk","ignore").decode("gbk"),welfare.replace("\t","").encode("gbk","ignore").decode("gbk"),jobdescription.replace("\t","").encode("gbk","ignore").decode("gbk"),workplace.replace("\t","").encode("gbk","ignore").decode("gbk")])
        except:
            print("%s没有获取"%myurl)
if __name__ == "__main__":
    for page in range(1,243):#共242页，根据实际情况修改
        Geturl(page)
        time.sleep(random.uniform(0.7,1.5))
    #print(urllist,len(urllist))
    Getcontent(urllist)
    time.sleep(random.uniform(0.7,2.5))
    with open ("51job.csv","w",newline = "") as f:
        writer = csv.writer(f)
        writer.writerow(["url","job","salary","company","companytype","companysize","companyfield","requirements","welfare","jobdescription","workplace"])
        for k in data:
            writer.writerow(k)

爬虫结果

一共爬取了11000条左右的数据，但是又删除了很多无效数据，请听下回分解。