自学python断断续续快一年了，从最初看邹琪鲜老师的视频，到参考网上的代码修修改改，再到现在能够根据实际需求写点东西，虽然期间有过较长时间的停滞期，但很高兴自己仍旧走在这条荆棘小路上。
最初我把重心都放在编写爬虫代码上，后来发现如果要走数据分析这条路的话，爬虫仅仅是个辅助工具，真正重要的数据分析思维，所以有了我的第一个数据分析项目。
我在网上找到了一个类似的分析案例，学习并参考，当然还有取长补短。参考案例链接：https://blog.csdn.net/lbship/article/details/79452459
筛选条件
我这里用的关键字是“数据分析师”，因为搜索“数据分析”的显示的岗位太杂，相关性比较差。
搜索地点是“全国”，结果按时间排序显示。
数据分析搜索结果，岗位相关性差
51job爬虫代码
# -*- coding: utf-8 -*-
"""
Created on Fri Sep 7 15:08:19 2018
@author: Shirley
"""
import requests
from lxml import etree
import time
import random
import csv
urllist = []
def Geturl(page):#获取列表链接
url = "https://search.51job.com/list/000000,000000,0000,00,9,99,数据分析师,2,%d.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%%2C0&radius=-1&ord_field=1&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare="%page
#关键词“数据分析师”，地点：全国，按时间排序
resp = requests.get(url)
try:
urlcontent = etree.HTML(resp.text,parser=etree.HTMLParser(encoding="gbk"))
myurls = urlcontent.xpath("//p[@class='t1 ']/span/a/@href")
for myurl in myurls:
urllist.append(myurl)
except:
print("第%d页没有获取！"%page)
data = []
def Getcontent(urllist):#根据链接抓取详细内容
for myurl in urllist:
try:
resp = requests.get(myurl)
maincontent = etree.HTML(resp.text,parser=etree.HTMLParser(encoding="gbk"))
job = maincontent.xpath("string(//div[@class='cn']/h1)")
salary = maincontent.xpath("string(//div[@class='cn']/strong)")
company = maincontent.xpath("string(//a[@class='catn'])")
companytype = maincontent.xpath("string(//p[@class='at'][1])")
companysize = maincontent.xpath("string(//p[@class='at'][2])")
companyfield = maincontent.xpath("string(//p[@class='at'][3])")
requirements = maincontent.xpath("string(//p[@class='msg ltype'])")
welfare = maincontent.xpath("string(//div[@class='t1'])")
jobdescription = maincontent.xpath("string(//div[@class='bmsg job_msg inbox'])")
workplace = maincontent.xpath("string(//div[@class='bmsg inbox']/p[@class='fp'])")
data.append([myurl,job.replace("\t","").encode("gbk","ignore").decode("gbk"),salary.replace("\t","").encode("gbk","ignore").decode("gbk"),company.replace("\t","").encode("gbk","ignore").decode("gbk"),companytype.replace("\t","").encode("gbk","ignore").decode("gbk"),companysize.replace("\t","").encode("gbk","ignore").decode("gbk"),companyfield.replace("\t","").encode("gbk","ignore").decode("gbk"),requirements.replace("\t","").encode("gbk","ignore").decode("gbk"),welfare.replace("\t","").encode("gbk","ignore").decode("gbk"),jobdescription.replace("\t","").encode("gbk","ignore").decode("gbk"),workplace.replace("\t","").encode("gbk","ignore").decode("gbk")])
except:
print("%s没有获取"%myurl)
if __name__ == "__main__":
for page in range(1,243):#共242页，根据实际情况修改
Geturl(page)
time.sleep(random.uniform(0.7,1.5))
#print(urllist,len(urllist))
Getcontent(urllist)
time.sleep(random.uniform(0.7,2.5))
with open ("51job.csv","w",newline = "") as f:
writer = csv.writer(f)
writer.writerow(["url","job","salary","company","companytype","companysize","companyfield","requirements","welfare","jobdescription","workplace"])
for k in data:
writer.writerow(k)
爬虫结果
一共爬取了11000条左右的数据，但是又删除了很多无效数据，请听下回分解。