第四次作业

作业：

爬取大数据专题所有文章列表，并输出到文本中保存。
每篇文章需要爬取的数据：作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数、打赏数。

第一个作业爬取大数据专题所有文章列表

思路：
1. 爬取大数据专题页面的所有链接。可以直接从大数据专题界面爬取，不需要进入每篇文章获取链接
2. 获取大数据专题页面每篇文章的链接后，爬取对应链接中感兴趣的内容：标题、内容
代码：
设置作业一和作业二共同使用的参数:

import os
import time
import urllib2
import urlparse
from bs4 import BeautifulSoup
url_root = 'http://www.jianshu.com'
url_seed = 'http://www.jianshu.com/c/9b4685b6357c?page=%d'

下载页面的内容，使用download函数

def download(url, retry=2):
    """
    下载页面的函数，会下载完整的页面信息
    :param url: 要下载的url
    :param retry: 重试次数
    :return: 原生html
    """
    print "downloading", url
    # 设置header信息，模拟浏览器请求
    header = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36'
    }
    try: #爬取可能会失败，采用try-except方式来捕获处理
        request = urllib2.Request(url, headers=header)  #设置请求数据
        html = urllib2.urlopen(request).read()  #抓取url
    except urllib2.URLError as e:  #异常处理
        print "download error:", e.reason
        html = None
        if retry > 0:  #未超过重试次数，可以继续爬取
            if hasattr(e,'code') and 500 <= e.code <= 600:  #错误码范围，是请求出错才继续重试爬取
                print e.code
                return download(url, retry-1)
    time.sleep(0.2)  #等待0.2s，避免对服务器造成压力，也避免被服务器屏蔽爬取
    return html

爬取大数据专题的所有文章链接，使用crawled_links函数

def crawled_links(url_seed,url_root):
    """
    抓取文章链接
    :param url_seed: 下载的种子页面地址
    :param url_root: 爬取网站的根目录
    :return: 需要爬取的页面
    """
    crawled_url = set()  #设置无序集合 ,需要爬取的页面
    i = 1
    flag = True  #标记是否需要继续爬取
    while flag:
        url = url_seed % i #真正爬取的页面
        i += 1  #下一次需要爬取的页面
        html = download(url)  #下载页面
        if html == None:  #下载页面为空，表示已爬取到最后
            break
        soup = BeautifulSoup(html, "html.parser")  #格式化爬取的页面数据
        links = soup.find_all('a', {'class' : 'title'}) #获取标题元素
        if links.__len__() == 0:  #爬取的页面中已无有效数据，终止爬取
            flag = False
        for link in links:  #获取有效的文章地址
            link = link.get('href')
            if link not in crawled_url:
                realUrl = urlparse.urljoin(url_root, link)
                crawled_url.add(realUrl)  # 记录未重复的需要爬取的页面
            else:
                print 'end % page' %(i)
                flag = False
    paper_num = crawled_url.__len__()
    print 'total paper num:', paper_num
    return crawled_url

爬取每篇文章的内容，并将内容存入txt，使用crawled_page函数

def crawled_page(crawled_url):
    """
    爬取文章内容
    :param crawled_url: 需要爬取的页面地址集合
    """
    for link in crawled_url: #按地址逐篇文章爬取
        html = download(link)
        soup = BeautifulSoup(html, "html.parser")
        title = soup.find('h1', {'class': 'title'}).text #获取文章标题
        content = soup.find('div', {'class': 'show-content'}).text #获取文章内容     
        title = title.replace('\x08', '_') #将标题中特殊符号转换为'_'  
        title = title.replace(r'<', '_')
        title = title.replace(r'>', '_')
        title = title.replace(r':', '_')
        title = title.replace(r'|', '_')
        title = title.replace(r'?', '_')
        title = title.replace(r'*', '_')       
        title = title.replace(r'/', '_')
        title = title.replace(r'"', '_')
        if os.path.exists('spider_res/') == False:   #检查保存文件的地址
            os.mkdir('spider_res')
        file_name = 'spider_res/' + title + '.txt'   #设置要保存的文件名
        if os.path.exists(file_name):
            continue
        file = open(file_name,'wb')  #写文件
        content = unicode(content).encode('utf-8',errors = 'ignore')
        file.write(content)
        file.close()

执行函数

crawled_links = crawled_links(url_seed,url_root)
crawled_page(crawled_links)

第二个作业爬取大数据专题所有文章详情

思路：
1. 获取大数据专题第一页的每篇文章信息：作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数。
2. 通过循环函数获取大数据专题每页中的文章信息：作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数。
代码：

爬取大数据专题第一页中的文章列表，使用crawl_list函数

def crawl_list(url):
    """
    爬取对应链接的内容，
    return：爬取的文章列表
    """
    html = download(url)
    if html == None:
        return
    soup = BeautifulSoup(html, "html.parser")
    return soup.find(id = 'list-container').find('ul', {'class' : 'note-list'})

爬取每篇文章的作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数。使用函数crawl_paper_tag

def crawl_paper_tag(list, url_root):
    """
    获取文章列表详情
    """
    paperlist = []
    lists = list.find_all('li')
    for paperTag in lists:
        author = paperTag.find('div', {'class' : 'content'}).find('div', {'class' : 'author'}).text
        title = paperTag.find('div', {'class' : 'content'}).find('a', {'class' : 'title'}).text
        paperUrl = paperTag.find('div', {'class' : 'content'}).find('a', {'class' : 'title'}).get('href')
        abstract = paperTag.find('div', {'class' : 'content'}).find('p', {'class' : 'abstract'}).text
        pic = paperTag.find('a', {'class' : 'wrap-img'})
        metaRead = paperTag.find('div', {'class' : 'content'}).find('div', {'class' : 'meta'}).find('i',{'class':'iconfont ic-list-read'}).text
        metaComment = paperTag.find('div', {'class' : 'content'}).find('div', {'class' : 'meta'}).find('i',{'class':'iconfont ic-list-comments'}).text
        metaLike = paperTag.find('div', {'class' : 'content'}).find('div', {'class' : 'meta'}).find('i',{'class':'iconfont ic-list-like'}).text
        metaReward = ''
        paperAttr = {
        'author' : author,
        'title' : title,
        'url' : urlparse.urljoin(url_root, paperUrl),
        'abstract' : abstract,
        'pic' : pic,
        'read' : metaRead,
        'comment' : metaComment,
        'like' : metaLike,
        'reward' : metaReward
        }
        if os.path.exists('info/') == False:
            os.mkdir('info')
        title = deal_title(title)
        file_name = 'info/' + title + '.txt'          
        file = open(file_name, 'wb')
        content_tag = unicode(paperAttr).encode('utf-8',errors = 'ignore')
        file.write(content_tag)
        file.close()
        paperlist.append(paperAttr)
    return paperlist

使用'_'替换标题中特殊符号，使用函数deal_title

def deal_title(title):
    """
    文件命名中不能够特殊符号，转换标题中特殊符号
    """
    title = title.replace('\x08', '_')
    title = title.replace(r'<', '_')
    title = title.replace(r'>', '_')
    title = title.replace(r':', '_')
    title = title.replace(r'|', '_')
    title = title.replace(r'?', '_')
    title = title.replace(r'*', '_')       
    title = title.replace(r'/', '_')
    title = title.replace(r'"', '_')
    return title

去爬所有文章的作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数，使用函数crawl_all_paper_tag

def crawl_all_paper_tag(url_seed, url_root):
    """
    爬取所有文章的详细内容
    """
    i = 1
    flag = True
    all_tag_list = []
    while flag:
        url = url_seed % i
        i += 1
        crawl_content = crawl_list(url)
        all_tag_list = crawl_paper_tag(crawl_content, url_root)
        if all_tag_list.__len__() == 0:
            flag = False
            print 'the end'

执行函数

crawl_all_paper_tag(url_seed, url_root)

结果

    downloading http://www.jianshu.com/c/9b4685b6357c?page=1
    downloading http://www.jianshu.com/c/9b4685b6357c?page=2
    downloading http://www.jianshu.com/c/9b4685b6357c?page=3
    downloading http://www.jianshu.com/c/9b4685b6357c?page=4
    downloading http://www.jianshu.com/c/9b4685b6357c?page=5
    downloading http://www.jianshu.com/c/9b4685b6357c?page=6
    downloading http://www.jianshu.com/c/9b4685b6357c?page=7
    downloading http://www.jianshu.com/c/9b4685b6357c?page=8
    downloading http://www.jianshu.com/c/9b4685b6357c?page=9
    downloading http://www.jianshu.com/c/9b4685b6357c?page=10
    downloading http://www.jianshu.com/c/9b4685b6357c?page=11
    downloading http://www.jianshu.com/c/9b4685b6357c?page=12
    downloading http://www.jianshu.com/c/9b4685b6357c?page=13
    downloading http://www.jianshu.com/c/9b4685b6357c?page=14
    downloading http://www.jianshu.com/c/9b4685b6357c?page=15
    downloading http://www.jianshu.com/c/9b4685b6357c?page=16
    downloading http://www.jianshu.com/c/9b4685b6357c?page=17
    downloading http://www.jianshu.com/c/9b4685b6357c?page=18
    downloading http://www.jianshu.com/c/9b4685b6357c?page=19
    downloading http://www.jianshu.com/c/9b4685b6357c?page=20
    downloading http://www.jianshu.com/c/9b4685b6357c?page=21
    downloading http://www.jianshu.com/c/9b4685b6357c?page=22
    downloading http://www.jianshu.com/c/9b4685b6357c?page=23
    downloading http://www.jianshu.com/c/9b4685b6357c?page=24
    downloading http://www.jianshu.com/c/9b4685b6357c?page=25
    downloading http://www.jianshu.com/c/9b4685b6357c?page=26
    downloading http://www.jianshu.com/c/9b4685b6357c?page=27
    downloading http://www.jianshu.com/c/9b4685b6357c?page=28
    downloading http://www.jianshu.com/c/9b4685b6357c?page=29
    downloading http://www.jianshu.com/c/9b4685b6357c?page=30
    downloading http://www.jianshu.com/c/9b4685b6357c?page=31
    total paper num: 293
    downloading http://www.jianshu.com/p/45df7e3ecc78
    downloading http://www.jianshu.com/p/99ae5b28a51f
    downloading http://www.jianshu.com/p/d6243f087bd9
    downloading http://www.jianshu.com/p/ea40c6da9fec
    downloading http://www.jianshu.com/p/59e0da43136e
    downloading http://www.jianshu.com/p/e71e5d7223bb
    downloading http://www.jianshu.com/p/dc07545c6607
    downloading http://www.jianshu.com/p/99fd951a0b8b
    downloading http://www.jianshu.com/p/02f33063c258
    downloading http://www.jianshu.com/p/ad10d79255f8
    downloading http://www.jianshu.com/p/062b8dfca144
    downloading http://www.jianshu.com/p/cb4f8ab1b380
    downloading http://www.jianshu.com/p/2c557a1bfa04
    downloading http://www.jianshu.com/p/8f7102c74a4f
    downloading http://www.jianshu.com/p/77876ef45ab4
    downloading http://www.jianshu.com/p/e5475131d03f
    downloading http://www.jianshu.com/p/e0bd6bfad10b
    downloading http://www.jianshu.com/p/a425acdaf77e
    downloading http://www.jianshu.com/p/729edfc613aa
    downloading http://www.jianshu.com/p/e50c863bb465
    downloading http://www.jianshu.com/p/7107b67c47bc
    downloading http://www.jianshu.com/p/6585d58f582a
    downloading http://www.jianshu.com/p/1292d7a3805e
    downloading http://www.jianshu.com/p/7cb84cfa56fa
    downloading http://www.jianshu.com/p/41c14ef3e59a
    downloading http://www.jianshu.com/p/1a2a07611fd8
    downloading http://www.jianshu.com/p/217a4578f9ab
    downloading http://www.jianshu.com/p/d234a015fa90
    downloading http://www.jianshu.com/p/e08d1a03045f
    downloading http://www.jianshu.com/p/6f4a7a1ef85c
    downloading http://www.jianshu.com/p/faf2f4107b9b
    downloading http://www.jianshu.com/p/9dee9886b140
    downloading http://www.jianshu.com/p/e2ee86a8a32b
    downloading http://www.jianshu.com/p/9258b0495021
    downloading http://www.jianshu.com/p/7e2fccb4fad9
    downloading http://www.jianshu.com/p/f21f01a92521
    downloading http://www.jianshu.com/p/d882831868fb
    downloading http://www.jianshu.com/p/872a67eed7af
    downloading http://www.jianshu.com/p/2e64c2045be5
    downloading http://www.jianshu.com/p/565500cfb5a4
    downloading http://www.jianshu.com/p/1729787990e7
    downloading http://www.jianshu.com/p/8ca518b3b2d5
    downloading http://www.jianshu.com/p/9c7fbcac3461
    downloading http://www.jianshu.com/p/13d76e7741c0
    downloading http://www.jianshu.com/p/81d17436f29e
    downloading http://www.jianshu.com/p/148b7cc83bcd
    downloading http://www.jianshu.com/p/70b7505884e9
    downloading http://www.jianshu.com/p/ba4100af215a
    downloading http://www.jianshu.com/p/819a202adecd
    downloading http://www.jianshu.com/p/a4beefd8cfc2
    downloading http://www.jianshu.com/p/eb01f9002091
    downloading http://www.jianshu.com/p/ba43beaa186a
    downloading http://www.jianshu.com/p/14967ec6e954
    downloading http://www.jianshu.com/p/d44cc7e9a0a9
    downloading http://www.jianshu.com/p/d0de8ee83ea1
    downloading http://www.jianshu.com/p/b4670cb9e998
    downloading http://www.jianshu.com/p/9f9fb337be0c
    downloading http://www.jianshu.com/p/542f41879879
    downloading http://www.jianshu.com/p/e9f6b15318be
    downloading http://www.jianshu.com/p/f1ef93a6c033
    downloading http://www.jianshu.com/p/92a66ccc8998
    downloading http://www.jianshu.com/p/f0063d735a5c
    downloading http://www.jianshu.com/p/856c8d648e20
    downloading http://www.jianshu.com/p/b9407b2c22a4
    downloading http://www.jianshu.com/p/a36e997b8e59
    downloading http://www.jianshu.com/p/c28207b3c71d
    downloading http://www.jianshu.com/p/8448ac374dc1
    downloading http://www.jianshu.com/p/4a3fbcb06981
    downloading http://www.jianshu.com/p/b1a9daef3423
    downloading http://www.jianshu.com/p/5eb037498c48
    downloading http://www.jianshu.com/p/f756bf0beb26
    downloading http://www.jianshu.com/p/673b768c6084
    downloading http://www.jianshu.com/p/6233788a8abb
    downloading http://www.jianshu.com/p/087ce1951647
    downloading http://www.jianshu.com/p/7240db1ba0af
    downloading http://www.jianshu.com/p/289e51eb6446
    downloading http://www.jianshu.com/p/0565cd673282
    downloading http://www.jianshu.com/p/873613065502
    downloading http://www.jianshu.com/p/605644d688ff
    downloading http://www.jianshu.com/p/1ea730c97aae
    downloading http://www.jianshu.com/p/bab0c09416ee
    downloading http://www.jianshu.com/p/c6591991d1ca
    downloading http://www.jianshu.com/p/fd9536a0acfb
    downloading http://www.jianshu.com/p/ed8dc3802927
    downloading http://www.jianshu.com/p/f89c4032a0b2
    downloading http://www.jianshu.com/p/1fa23219270d
    downloading http://www.jianshu.com/p/412f8eab2599
    downloading http://www.jianshu.com/p/05c15b9f16f1
    downloading http://www.jianshu.com/p/4931d66276c3
    downloading http://www.jianshu.com/p/b5165468a32b
    downloading http://www.jianshu.com/p/2c02a7b0b382
    downloading http://www.jianshu.com/p/dffdaf11bd4c
    downloading http://www.jianshu.com/p/71c02ef761ac
    downloading http://www.jianshu.com/p/6920d5e48b31
    downloading http://www.jianshu.com/p/71b968bd8abb
    downloading http://www.jianshu.com/p/5a6c4b8e7700
    downloading http://www.jianshu.com/p/c1163e39a42e
    downloading http://www.jianshu.com/p/bd9a27c4e2a8
    downloading http://www.jianshu.com/p/88d0addf64fa
    downloading http://www.jianshu.com/p/6a7afc98c868
    downloading http://www.jianshu.com/p/733475b6900d
    downloading http://www.jianshu.com/p/f75128ec3ea3
    downloading http://www.jianshu.com/p/9ee12067f35e
    downloading http://www.jianshu.com/p/c41624a83b71
    downloading http://www.jianshu.com/p/67ae9d87cf3c
    downloading http://www.jianshu.com/p/b5c292e093a2
    downloading http://www.jianshu.com/p/0a6977eb686d
    downloading http://www.jianshu.com/p/456ab3a6ef71
    downloading http://www.jianshu.com/p/d578d5e2755f
    downloading http://www.jianshu.com/p/c9e1dffad756
    downloading http://www.jianshu.com/p/81819f27a7d8
    downloading http://www.jianshu.com/p/799c51fbe5f1
    downloading http://www.jianshu.com/p/5e4a86f8025c
    downloading http://www.jianshu.com/p/7acf291b2a5e
    downloading http://www.jianshu.com/p/6ef6b9a56b50
    downloading http://www.jianshu.com/p/210aacd31ef7
    downloading http://www.jianshu.com/p/9a9280de68f8
    downloading http://www.jianshu.com/p/d5bc50d8e0a2
    downloading http://www.jianshu.com/p/39eb230e6f15
    downloading http://www.jianshu.com/p/c0c0a3ed35d4
    downloading http://www.jianshu.com/p/74db357c7252
    downloading http://www.jianshu.com/p/6a91f948b62d
    downloading http://www.jianshu.com/p/bc75ab89fac0
    downloading http://www.jianshu.com/p/8088d1bede8d
    downloading http://www.jianshu.com/p/8ca88a90ea17
    downloading http://www.jianshu.com/p/a8037a38e219
    downloading http://www.jianshu.com/p/979b4c5c1857
    downloading http://www.jianshu.com/p/3dfedf60de62
    downloading http://www.jianshu.com/p/ada67bd7c56f
    downloading http://www.jianshu.com/p/486afcd4c36c
    downloading http://www.jianshu.com/p/8a0479f55b21
    downloading http://www.jianshu.com/p/e492d3acfe38
    downloading http://www.jianshu.com/p/b4e2e5e31154
    downloading http://www.jianshu.com/p/75fc36aec98e
    downloading http://www.jianshu.com/p/545581b0c7dd
    downloading http://www.jianshu.com/p/a015b756a803
    downloading http://www.jianshu.com/p/29062bca16aa
    downloading http://www.jianshu.com/p/3a95a09cda40
    downloading http://www.jianshu.com/p/8fbe3a7b4764
    downloading http://www.jianshu.com/p/0329f87c9ae4
    downloading http://www.jianshu.com/p/e1b28de0a1e4
    downloading http://www.jianshu.com/p/b5c31a2eeb8b
    downloading http://www.jianshu.com/p/7e556f17021a
    downloading http://www.jianshu.com/p/23144099e9f8
    downloading http://www.jianshu.com/p/a91c54f96ded
    downloading http://www.jianshu.com/p/74ef104a9f45
    downloading http://www.jianshu.com/p/afa17bc391b7
    downloading http://www.jianshu.com/p/90914aef3636
    downloading http://www.jianshu.com/p/0c0e3ace0da1
    downloading http://www.jianshu.com/p/b7eef4033a09
    downloading http://www.jianshu.com/p/7b2e81589a4f
    downloading http://www.jianshu.com/p/2f7d10b2e508
    downloading http://www.jianshu.com/p/ed499f4ecdd1
    downloading http://www.jianshu.com/p/11c103c03d4a
    downloading http://www.jianshu.com/p/333dacb0e1b2
    downloading http://www.jianshu.com/p/7c54cd046d4b
    downloading http://www.jianshu.com/p/cfaf85b24281
    downloading http://www.jianshu.com/p/356a579062aa
    downloading http://www.jianshu.com/p/460a8eed5cfa
    downloading http://www.jianshu.com/p/46e82e4fe324
    downloading http://www.jianshu.com/p/ba00a9852a02
    downloading http://www.jianshu.com/p/b6359185fc26
    downloading http://www.jianshu.com/p/a1a2dabb4bc2
    downloading http://www.jianshu.com/p/4077cbc4dd37
    downloading http://www.jianshu.com/p/90efe88727fe
    downloading http://www.jianshu.com/p/17f99100525a
    downloading http://www.jianshu.com/p/01385e2dd129
    downloading http://www.jianshu.com/p/ec3c57d6a4c7
    downloading http://www.jianshu.com/p/9632ba906ca2
    downloading http://www.jianshu.com/p/85da47fddad7
    downloading http://www.jianshu.com/p/3b47b36cc8e8
    downloading http://www.jianshu.com/p/29e304a61d32
    downloading http://www.jianshu.com/p/649167e0e2f4
    downloading http://www.jianshu.com/p/13840057782d
    downloading http://www.jianshu.com/p/11b3dbb05c39
    downloading http://www.jianshu.com/p/3a5975d6ac55
    downloading http://www.jianshu.com/p/41b1ee54d766
    downloading http://www.jianshu.com/p/0ee1f0bfc8cb
    downloading http://www.jianshu.com/p/2364064e0bc9
    downloading http://www.jianshu.com/p/09b19b8f8886
    downloading http://www.jianshu.com/p/50a2ba489685
    downloading http://www.jianshu.com/p/f0436668cb72
    downloading http://www.jianshu.com/p/c0f3d36d0c7a
    downloading http://www.jianshu.com/p/be0192aa6486
    downloading http://www.jianshu.com/p/ee43c55123f8
    downloading http://www.jianshu.com/p/af4765b703f0
    downloading http://www.jianshu.com/p/ff772050bd96
    downloading http://www.jianshu.com/p/e121b1a420ad
    downloading http://www.jianshu.com/p/ed93f7f344d0
    downloading http://www.jianshu.com/p/8f6ee3b1efeb
    downloading http://www.jianshu.com/p/3f06c9f69142
    downloading http://www.jianshu.com/p/ff2d4eadebde
    downloading http://www.jianshu.com/p/ce0e0773c6ec
    downloading http://www.jianshu.com/p/be384fd73bdb
    downloading http://www.jianshu.com/p/acc47733334f
    downloading http://www.jianshu.com/p/bf5984fb299a
    downloading http://www.jianshu.com/p/1a935c2dc911
    downloading http://www.jianshu.com/p/8982ad63eb85
    downloading http://www.jianshu.com/p/d1acbed69f45
    downloading http://www.jianshu.com/p/98cc73755a22
    downloading http://www.jianshu.com/p/bb736600b483
    downloading http://www.jianshu.com/p/3c71839bc660
    downloading http://www.jianshu.com/p/23a905cf936b
    downloading http://www.jianshu.com/p/169403f7e40c
    downloading http://www.jianshu.com/p/a9c7970bc949
    downloading http://www.jianshu.com/p/ed9ec88e71e4
    downloading http://www.jianshu.com/p/5057ab6f9ad5
    downloading http://www.jianshu.com/p/1b42a12dac14
    downloading http://www.jianshu.com/p/5dc5dfe26148
    downloading http://www.jianshu.com/p/c88a4453dd6d
    downloading http://www.jianshu.com/p/cd971afcb207
    downloading http://www.jianshu.com/p/2ccd37ae73e2
    downloading http://www.jianshu.com/p/926013888e3e
    downloading http://www.jianshu.com/p/888a580b2384
    downloading http://www.jianshu.com/p/e72c8ef71e49
    downloading http://www.jianshu.com/p/bb4a81624af1
    downloading http://www.jianshu.com/p/4b944b22fe83
    downloading http://www.jianshu.com/p/fa7dd359d7a8
    downloading http://www.jianshu.com/p/bfd9b3954038
    downloading http://www.jianshu.com/p/f6c26ef0f4cc
    downloading http://www.jianshu.com/p/56967004f8c4
    downloading http://www.jianshu.com/p/394856545ab0
    downloading http://www.jianshu.com/p/aed64f7e647b
    downloading http://www.jianshu.com/p/a32f27199846
    downloading http://www.jianshu.com/p/4b4e0c343d3e
    downloading http://www.jianshu.com/p/8f6b5a1bb3fa
    downloading http://www.jianshu.com/p/f7354d1c5abf
    downloading http://www.jianshu.com/p/1fe31cbddc78
    downloading http://www.jianshu.com/p/f7dc92913f33
    downloading http://www.jianshu.com/p/296ae7538d1f
    downloading http://www.jianshu.com/p/d43125a4ff44
    downloading http://www.jianshu.com/p/0b0b7c33be57
    downloading http://www.jianshu.com/p/b4ac4473a55d
    downloading http://www.jianshu.com/p/4b57424173a0
    downloading http://www.jianshu.com/p/e0ae002925bd
    downloading http://www.jianshu.com/p/5250518f5cc5
    downloading http://www.jianshu.com/p/de3455ed089c
    downloading http://www.jianshu.com/p/7b946e6d6861
    downloading http://www.jianshu.com/p/62e127dbb73c
    downloading http://www.jianshu.com/p/430b5bea974d
    downloading http://www.jianshu.com/p/e5d13e351320
    downloading http://www.jianshu.com/p/5d8a3205e28e
    downloading http://www.jianshu.com/p/1099c3a74336
    downloading http://www.jianshu.com/p/761a73b7eea2
    downloading http://www.jianshu.com/p/83cc892eb24a
    downloading http://www.jianshu.com/p/b223e54fe5ee
    downloading http://www.jianshu.com/p/366c2594f24b
    downloading http://www.jianshu.com/p/cc3b5d76c587
    downloading http://www.jianshu.com/p/6dbadc78d231
    downloading http://www.jianshu.com/p/d32d7ab5063a
    downloading http://www.jianshu.com/p/020f0281f1df
    downloading http://www.jianshu.com/p/f26085aadd47
    downloading http://www.jianshu.com/p/df7b35249975
    downloading http://www.jianshu.com/p/68423bfc4c4e
    downloading http://www.jianshu.com/p/601d3a488a58
    downloading http://www.jianshu.com/p/1d6fc1a9406b
    downloading http://www.jianshu.com/p/76238014a03f
    downloading http://www.jianshu.com/p/9e7cfcc85a57
    downloading http://www.jianshu.com/p/4a8749704ebf
    downloading http://www.jianshu.com/p/d2dc5aa9bf8f
    downloading http://www.jianshu.com/p/4dda2425314a
    downloading http://www.jianshu.com/p/8baa664ea613
    downloading http://www.jianshu.com/p/cbfab5db7f6f
    downloading http://www.jianshu.com/p/bd78a49c9d23
    downloading http://www.jianshu.com/p/cf2edecdba77
    downloading http://www.jianshu.com/p/3b3bca4281aa
    downloading http://www.jianshu.com/p/f382741c2736
    downloading http://www.jianshu.com/p/4ffca0a43476
    downloading http://www.jianshu.com/p/e04bcac99c8d
    downloading http://www.jianshu.com/p/37e927476dfe
    downloading http://www.jianshu.com/p/4981df2eefe7
    downloading http://www.jianshu.com/p/86117613b7a6
    downloading http://www.jianshu.com/p/233ff48d668e
    downloading http://www.jianshu.com/p/13a68ac7afdd
    downloading http://www.jianshu.com/p/aa1121232dfd
    downloading http://www.jianshu.com/p/e99dacbf5c44
    downloading http://www.jianshu.com/p/74042ba10c0d
    downloading http://www.jianshu.com/p/40cc7d239513
    downloading http://www.jianshu.com/p/5a8b8ce0a395
    downloading http://www.jianshu.com/p/59ca82a11f87
    downloading http://www.jianshu.com/p/8266f0c736f9
    downloading http://www.jianshu.com/p/b3e8e9cb0141
    downloading http://www.jianshu.com/p/ae5f78b40f17
    downloading http://www.jianshu.com/p/10b429fd9c4d
    downloading http://www.jianshu.com/p/e76c242c7d6a
    downloading http://www.jianshu.com/p/910662d6e881
    downloading http://www.jianshu.com/p/f68d28d3b862
    downloading http://www.jianshu.com/p/9457100d8763
    downloading http://www.jianshu.com/p/62c0a5122fa8
    downloading http://www.jianshu.com/p/f6420cce3040
    downloading http://www.jianshu.com/p/27a78b2016e0
    downloading http://www.jianshu.com/p/0c007dbbf728
    downloading http://www.jianshu.com/p/f20bc50ad0e8
    downloading http://www.jianshu.com/c/9b4685b6357c?page=1
    downloading http://www.jianshu.com/c/9b4685b6357c?page=2
    downloading http://www.jianshu.com/c/9b4685b6357c?page=3
    downloading http://www.jianshu.com/c/9b4685b6357c?page=4
    downloading http://www.jianshu.com/c/9b4685b6357c?page=5
    downloading http://www.jianshu.com/c/9b4685b6357c?page=6
    downloading http://www.jianshu.com/c/9b4685b6357c?page=7
    downloading http://www.jianshu.com/c/9b4685b6357c?page=8
    downloading http://www.jianshu.com/c/9b4685b6357c?page=9
    downloading http://www.jianshu.com/c/9b4685b6357c?page=10
    downloading http://www.jianshu.com/c/9b4685b6357c?page=11
    downloading http://www.jianshu.com/c/9b4685b6357c?page=12
    downloading http://www.jianshu.com/c/9b4685b6357c?page=13
    downloading http://www.jianshu.com/c/9b4685b6357c?page=14
    downloading http://www.jianshu.com/c/9b4685b6357c?page=15
    downloading http://www.jianshu.com/c/9b4685b6357c?page=16
    downloading http://www.jianshu.com/c/9b4685b6357c?page=17
    downloading http://www.jianshu.com/c/9b4685b6357c?page=18
    downloading http://www.jianshu.com/c/9b4685b6357c?page=19
    downloading http://www.jianshu.com/c/9b4685b6357c?page=20
    downloading http://www.jianshu.com/c/9b4685b6357c?page=21
    downloading http://www.jianshu.com/c/9b4685b6357c?page=22
    downloading http://www.jianshu.com/c/9b4685b6357c?page=23
    downloading http://www.jianshu.com/c/9b4685b6357c?page=24
    downloading http://www.jianshu.com/c/9b4685b6357c?page=25
    downloading http://www.jianshu.com/c/9b4685b6357c?page=26
    downloading http://www.jianshu.com/c/9b4685b6357c?page=27
    downloading http://www.jianshu.com/c/9b4685b6357c?page=28
    downloading http://www.jianshu.com/c/9b4685b6357c?page=29
    downloading http://www.jianshu.com/c/9b4685b6357c?page=30
    downloading http://www.jianshu.com/c/9b4685b6357c?page=31
    the end

image.png

image.png

疑问：

如何获取大数据专题url_seed的地址
链接1：http://www.jianshu.com/c/9b4685b6357c
链接2：http://www.jianshu.com/c/9b4685b6357c?page=1
链接1和链接2都是指向解密大数据专题的首页。但是登陆解密大数据专题时，只会显示链接1，而不会显示链接2。
如何能够得到链接2，或者说链接2是否是在链接1的基础上按照一定的规律组成
将抓取文章数据：作者、标题、文章地址、摘要、缩略图地址、阅读数、平均数、点赞数，存入txt时，为什么没有显示成中文。如上图所示

第四次作业

第四次作业

作业：

第一个作业爬取大数据专题所有文章列表

第二个作业爬取大数据专题所有文章详情

你可能感兴趣的:(第四次作业)