【51Job数据爬取日志】前程无忧URL请求字段分析

像我们爬取过的其他类型的网页一样，对于前程无忧的爬取亦然。

第一步首先是登录前程无忧界面，提取Cookies值并且对URL的字段进行分析。

https://search.51job.com/list/000000,000000,0000,00,9,99,Java%2520,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

我们都知道在URL中“？”后的内容是网站请求字段的键值。

通过F12打开开发者工具，进入到Network标签下，选中跟url类似的html网页。在headers中的Query String Parameters中即可看到url中包含的所有请求字段。如下图所示：

 

具体内容如下：其中的众多字段其实是对网站自带查询字段的综合展示与反应，字段键值对越完善的查询，其筛选的结果越精细。

分析请求字段

因为好奇，我决定拿出些时间分析一下各字段的表示。

① 对石家庄的Java的查询页面对应的URL：

https://search.51job.com/list/160200,000000,0000,00,9,99,Java,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

② 对北京的Java的查询页面对应的URL：

https://search.51job.com/list/010000,000000,0000,00,9,99,python,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

通过①②的比对，易知url中城市的代表位置与岗位的代表位置。

另外，如果有同学不晓得如何解决前程无忧url的编码问题，看见参考代码。https://blog.csdn.net/m0_38052500/article/details/88688804

③ 对北京、石家庄的python以及行业是计算机软件的查询URL：

https://search.51job.com/list/010000%252C160200,000000,0000,01,9,99,python,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

④  对北京、石家庄、上海的python以及行业是计算机软件的查询URL：

https://search.51job.com/list/010000%252C160200%252C020000,000000,0000,01,9,99,python,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

⑤ Java开发工程师  北京  计算机软件 软件工程师+算法工程师

https://search.51job.com/list/010000,000000,0107%252C0148,01,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

通过以上小例子，可得下图图示：

——————————————————————————————————————————————

⑥ {Java开发工程师  北京 计算机硬件 软件工程师  } 发布日期：所有

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=5&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

⑦  {Java开发工程师  北京 计算机硬件 软件工程师  } + 发布日期：近三天

https://search.51job.com/list/010000,000000,0107,37,1,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=5&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

⑧ {Java开发工程师  北京 计算机硬件 软件工程师  } + 发布日期：近一周

https://search.51job.com/list/010000,000000,0107,37,2,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=5&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

⑨ {Java开发工程师  北京 计算机硬件 软件工程师  } + 发布日期：近一月

https://search.51job.com/list/010000,000000,0107,37,3,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=5&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

⑩ {Java开发工程师  北京 计算机硬件 软件工程师  } + 发布日期：24小时以内

https://search.51job.com/list/010000,000000,0107,37,0,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=5&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

通过 ⑥ ⑦ ⑧ ⑨知 红色部分位置 代表的发布日期代码。完整的代码映射从红色标识即知。

十一   {Java开发工程师  北京 计算机硬件 软件工程师  } + 月薪范围：2千以下

https://search.51job.com/list/010000,000000,0107,37,9,01,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=21&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

十二 {Java开发工程师  北京 计算机硬件 软件工程师  } + 月薪范围：2-3千

https://search.51job.com/list/010000,000000,0107,37,9,02,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=21&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

依照以上两个例子，我们预测4-5万 是11:，接下来验证以下：

https://search.51job.com/list/010000,000000,0107,37,9,11,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=21&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

看来是对了。不过在审查过程中，4-5万，其实列表所列的职位的薪资知识0.8-1.2K，由此，如果爬虫自动爬取，最后数据真实性存疑，需要手动分析。

十三、{Java开发工程师  北京 计算机硬件 软件工程师  } + 公司性质：外资（欧美）https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=01°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=8&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

十四、{Java开发工程师  北京 计算机硬件 软件工程师  } + 公司性质：外资（非欧美）

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=02°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=8&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

十五、{Java开发工程师  北京 计算机硬件 软件工程师  } + 公司性质：外资（非欧美）

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=04°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=8&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

所以：cotype表示公司性质。其他类型的队形代码不再赘述。不过分析中发现fromType的值会发生变化。影响如何，大家操练中自行探索。

十六、{Java开发工程师  北京 计算机硬件 软件工程师  } + 工作年限：无经验

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=01&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=6&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

十七、{Java开发工程师  北京 计算机硬件 软件工程师  } + 工作年限：1-3年

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=02&cotype=99°reefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=6&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

所以：workyear 表示工作年限，同时需要改动fromType=9。

十八、{Java开发工程师  北京 计算机硬件 软件工程师  } + 学历要求：初中及以下

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=01&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=7&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

十九、{Java开发工程师  北京 计算机硬件 软件工程师  } + 学历要求：本科

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=04&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=7&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

所以：degreefrom表示学历要求。而fromType对应的值也产生了变化。

二十、{Java开发工程师  北京 计算机硬件 软件工程师  } + 公司规模：50-150人

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=02&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

二十一、{Java开发工程师  北京 计算机硬件 软件工程师  } + 公司规模：500-1000人

https://search.51job.com/list/010000,000000,0107,37,9,99,Java%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=04&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

所以：companysize表示公司规模，同时fromType恢复默认值 空。

至此，所有的分析结束。余下的字段如果有兴趣，大家自行分析吧，如果有人分析出来，烦请留一下链接，谢谢！

——————————————————————————————————————————————————————