python 爬虫爬取新东方考研真题

女朋友要考研，想看看各个院校的真题，网上找了下，发现新东方真题库里面的考研真题还挺齐全的，网址：http://new.bj.xdf.cn/zhentiku/daxue/kaoyan/kyzyk/list_381_1.html

于是随手写了个爬虫爬取所有 新闻学 的考研题，脚本如下：

from urllib import urlopen
from bs4 import BeautifulSoup
import re

for page in range(17):
    if page == 0:
        continue
    firstUrl = "http://new.bj.xdf.cn/zhentiku/daxue/kaoyan/kyzyk/list_381_" + str(page) + ".html"
    print "[Begin] scrap page", firstUrl
    html = urlopen(firstUrl)
    data = html.read()
    bsobj = BeautifulSoup(data)

    li = bsobj.findAll("a", {"title": re.compile(u"(.*?)新闻(.*?)")})

    for l in li:
        url = "http://new.bj.xdf.cn" + l.attrs["href"]
        filename = l.attrs["title"] + ".html"
        subdata = BeautifulSoup(urlopen(url).read())
        with open(filename, 'w') as f:
            f.write('\n')
            f.write('%s' % subdata.select(".article-wrap"))
            f.close()
    print "[End]"