【python】使用bs4以及BeautifulSoup爬虫爬取信息

首先我们的参考网站为: https://hr.tencent.com/position.php?&start=10#a

然后我们进入该网站查看信息：

通过审查元素我们可以发现我们需要的内容都在class='even'或者class='odd'的内容中，于是我们可以将这些内容全都爬取下来，然后再拼接到一起

核心代码如下:

获取到信息后将数据存入item数据字典中，然后再将item存入items序列中

完整代码如下:

# -*- encoding:utf-8 -*-

# 使用BeautifuSoup4解析器，将招聘网页上的
# 职位名称、职位类别、招聘人数、工作地点、发布时间，以及每个职位详情的点击链接存储出来。
from bs4 import BeautifulSoup

import urllib2
import urllib
import json

def tencent(output,page):
    url = 'http://hr.tencent.com/'
    print(url + 'position.php?start=%d#a' % page)
    headers = {"User-Agent": "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1 Trident/5.0;"}
    request = urllib2.Request(url + 'position.php?start=%d#a' % page,headers=headers)
    response = urllib2.urlopen(request)
    resHtml = response.read()

    html = BeautifulSoup(resHtml,'lxml')

    # 创建css选择器
    result = html.select('tr[class="even"]')
    result2 = html.select('tr[class="odd"]')
    result += result2
    items = []
    for site in result:
        item = {}
        pos_name = site.select('td a')[0].get_text()
        detailLink = site.select('td a')[0].attrs['href']
        catalog = site.select('td')[1].get_text()
        recruitNumber = site.select('td')[2].get_text()
        workLocation = site.select('td')[3].get_text()
        publishTime = site.select('td')[4].get_text()

        # 将其存入item中
        item['pos_name'] = pos_name
        item['detailLink'] = detailLink
        item['catalog'] = catalog
        item['recruitNumber'] = recruitNumber
        item['workLocation'] = workLocation
        item['publishTime'] = publishTime
        items.append(item)
    return items



if __name__ == "__main__":
    output = open('tencents.json', 'w')
    items = []
    for num in range(10):
        items.append(tencent(output,(num + 1)*10))

    # 禁用ascii编码，使用utf-8编码,将元组转换为json格式数据
    line = json.dumps(items, ensure_ascii=False)
    output.write(line.encode('utf-8'))
    output.close()

结果如下：