课程作业-爬虫入门04-2-构建爬虫-WilliamZeng-20170812

课堂作业

爬取解密大数据专题所有文章的内容，并保存到数据库中。数据库我选择了MySQL本地搭建的数据库。
要求保存：文章网址，标题，作者，发表时间，文章摘要，文章内容，阅读数，评论数，点赞数和打赏数

参考资料

MySQL官方网页了解安装本地服务器的步骤；
安装一个MySQL客户端工具，HeidiSQL不错，MySQL官方提供的Workbench也能满足要求；
其他的因为这次时间不充分，主要是照搬曾老师GitHub上的代码，做了一些简单修改。遇到问题就是网上搜索答案了。

实现概述

补充了文章的发表时间和内容这2个字段的获取与写入。
参考以前和已有方法把代码修改成获取所有抓取时解密大数据专题的文章信息并写入数据库。

主要问题

mysql.connector的安装在Windows环境不是太友好。不知道其他人有没有遇到问题和如何解决的。用pip install mysql-connector-python会遇到Could not find a version that satisfies the requirement mysql-connector-python (from versions: ) No matching distribution found for mysql-connector-python

后来通过解压官方的zip文件然后命令行安装解决。

2017/9/6更新：发现这个问题是因为我的电脑里同时安装了python官方的2.7和Anaconda里面的python 3.6.1导致的。安装了Anaconda后，pip.exe有2个版本，在系统控制台cmd默认运行的是...\Anaconda3\Scripts\下的pip.exe，而不是我们期望运行的\Python27\Scripts\下的pip.exe，因此会出错。想要运行python2的pip可以使用pip2 install mysql-connector-python。

之前下载官方的msi，系统环境变量PATH有Python2.7的安装目录，我的用户名的环境变量还没有python2.7和Anaconda的安装目录，推测可能是这个原因导致msi无法正常运行，错误提示如下。

课程作业-爬虫入门04-2-构建爬虫-WilliamZeng-20170812_第1张图片

mysql-connector-python-2.1.7-py2.7-windows-x86-64bit.msi安装错误

330篇的文章内容最后有7篇文章的内容不能正常数据库导入。错误提示看可能和字符编码有关，网上也去搜索了这个错误提示，花的时间不多。这里有一篇针对中文环境这类错误的解答，看上去和我们的情况不完全一致。另外就是修改my.ini文件内容。都还没有尝试。如果老师，助教或其他人知道解决这个问题的方法，还望指教。
发布时间的转换有些繁琐。如何更有效地直接抓取和存储时间变量(当地时间)到数据库中需要探索和求教。
exit(-1)是为了检查程序正常运行结束而增加的一行代码？期望最终输出Process finished with exit code -1？

代码部分

commonDB是建立一个数据库连接和实现各种数据库操作方法的类，这次时间不够，基本没有阅读；
paperDao是commonDB的一个子类，根据database里的配置属性建立连接并设置表，主键和表里面的列(字段)；
主程序还是和上次一样包含了导入模块，downlaod，crawl_list和crawl_paper_tag函数。其中crawl_list这次基于以前crawled_links里的方法作了一些修改可以像crawled_links一样爬取专题下所有的文章列表；
每篇文章目标标签信息，包括发表时间和对应文章网址下的文章内容主体都已写入crawl_paper_tag中；
写入数据库通过save_data函数实现，声明一个paperDao的实例，然后调用insertOne方法每次循环插入一条记录。

数据库部分

commonDB和paperDao基本没做修改，paperDao增加了写入'pub_time'这一个字段。代码不列出了。谢谢曾老师提供的建立数据库的代码，我是放到数据库查询(query)窗口里执行的。不知道大家用什么方式建立的。

CREATE TABLE `paper` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增索引',
  `title` varchar(256) NOT NULL DEFAULT '' COMMENT '标题',
  `url` varchar(256) NOT NULL DEFAULT '' COMMENT '文章地址',
  `author` varchar(11) NOT NULL DEFAULT '' COMMENT '作者',
  `abstract` varchar(512) NOT NULL DEFAULT '' COMMENT '文章摘要',
  `read_cnt` int(11) NOT NULL DEFAULT '0' COMMENT '阅读数',
  `comment_cnt` int(11) NOT NULL DEFAULT '0' COMMENT '评论数',
  `like_cnt` int(11) NOT NULL DEFAULT '0' COMMENT '点赞数',
  `reward_cnt` int(11) NOT NULL DEFAULT '0' COMMENT '打赏数',
  `_status` tinyint(1) NOT NULL DEFAULT '1' COMMENT '数据状态',
  `_create_time` datetime NOT NULL DEFAULT '1970-01-01 00:00:00' COMMENT '创建时间',
  `_modify_time` datetime NOT NULL DEFAULT '1970-01-01 00:00:00' ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
  `pub_time` datetime NOT NULL DEFAULT '1970-01-01 00:00:00' COMMENT '发表时间',
  `content` text NOT NULL COMMENT '内容',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=11 DEFAULT CHARSET=utf8;

导入模块

除了曾老师的代码外，抓取的发布时间的格式我自己的理解不是datetime变量，导入一些模块做了一些转换。时间转换方面有更简洁的方法盼望提出。

import os
import time
from datetime import datetime
import urllib2
import urlparse
from bs4 import BeautifulSoup
from paperDao import PaperDao
from time import mktime,strptime

download函数

和以前一样略过。

crawl_list函数

如果有发现写错或不够好的地方请指出，我最后改好的代码只试了一次，感觉跑通了而已。

def crawl_list(url_seed):
    """
    爬取文章列表区块
    :param url_seed:
    :param url_root:
    :return:
    """
    crawled_list_id = set()  # 需要爬取的页面文章列表集ID
    crawled_lists = [] # 需要爬取的页面文章列表集
    page = 1
    flag = True  # 标记是否需要继续爬取
    while flag:
        url = url_seed % page  # 真正爬取的页面
        page += 1  # 下一次需要爬取的页面

        html = download(url)  # 下载页面
        if html == None:  # 下载页面为空，表示已爬取到最后
            break

        soup = BeautifulSoup(html, "html.parser")  # 格式化爬取的页面数据
        lists = soup.find(id='list-container').find('ul', {'class': 'note-list'}).find_all('li') # 获取文章列表集
        if lists.__len__() == 0: # 爬取的页面中已无有效数据，终止爬取
            flag = False

        for list in lists:
            listId = list.get('data-note-id')
            if listId not in crawled_list_id:
                crawled_list_id.add(listId) # 记录未重复的需要爬取的文章列表ID
                crawled_lists.append(list)
            else:
                print 'end'
                flag = False # 结束抓取
    paper_num = crawled_lists.__len__()
    print 'total paper num: ', paper_num
    return crawled_lists

crawl_paper_tag函数

绝大多数老师这次已经写好了。strip函数不错，能提高大块内容中有效信息的抓取效率。文章缩略图网址picUrl的抓取代码保留了，数据库操作里未增加这个字段。时间转换中用到好多函数，希望有更有效的方法。文章内容的抓取和插入数据库操作已包含在这个函数里。

def crawl_paper_tag(lists, url_root):
    """
    获取文章列表详情
    :param list: 要爬取的文章列表
    :param url_root: 爬取网站的根目录
    :return:
    """
    paperList = []  # 文章属性集 list
    # print lists
    for paperTag in lists:
        content = paperTag.find('div', {'class': 'content'}) # 文章信息主体
        author = content.find('div', {'class': 'author'}).text.strip()  # 作者
        title = content.find('a', {'class': 'title'}).text.strip()  # 标题
        paperUrl = content.find('a', {'class': 'title'}).get('href').strip()  # 文章网址
        abstract = content.find('p', {'class': 'abstract'}).text.strip() # 文章摘要
        pic = content.find('a', {'class': 'wrap-img'})  # 文章缩略图
        if pic is not None:
            picSrc = pic.find('img', {'class': 'img-blur-done'}).get('src').strip()
            picUrl = urlparse.urljoin(url_root, picSrc)
        else:
            picUrl = ''
        submitTime = content.find('span', {'class': 'time'}).get('data-shared-at') # 文章发表时间原始格式
        # print submitTime
        # print type(submitTime)
        submitTime2 = datetime.fromtimestamp(mktime(strptime(submitTime[0:19], "%Y-%m-%dT%H:%M:%S"))) # 文章发表时间处理成更友好的时间格式
        # print submitTime2
        # print type(submitTime2)
        metaRead, metaComment, metaLike, metaReward = 0, 0, 0, 0  # 阅读数，评论数，点赞数，打赏数
        meta1 = content.find('div', {'class': 'meta'}).find_all('a')
        meta2 = content.find('div', {'class': 'meta'}).find_all('span')

        for meta in meta1:
            tagClass = meta.find('i').get('class')
            if 'ic-list-read' in tagClass:
                metaRead = meta.text.strip()
            if 'ic-list-comments' in tagClass:
                metaComment = meta.text.strip()

        for meta in meta2:
            tagClass = meta.find('i').get('class')
            if 'ic-list-like' in tagClass:
                metaLike = meta.text.strip()
            if 'ic-list-money' in tagClass:
                metaReward = meta.text.strip()
        html = download(urlparse.urljoin(url_root, paperUrl))
        soup = BeautifulSoup(html, "html.parser")
        rawContent = soup.find('div', {'class': 'show-content'}).text  # 文章内容
        encodedContent = unicode(rawContent).encode('utf-8', errors='ignore')

        paperAttr = {
            'author': author,
            'title': title,
            'url': urlparse.urljoin(url_root, paperUrl),
            'abstract': abstract,
            'content': encodedContent,
            'picUrl': picUrl,
            'read_cnt': metaRead,
            'comment_cnt': metaComment,
            'like_cnt': metaLike,
            'reward_cnt': metaReward,
            'pub_time': submitTime2
        }
        paperList.append(paperAttr)
    return paperList

save_data函数

没有修改曾老师的代码，略过。

调用函数执行页面抓取

url_root = 'http://www.jianshu.com/'  # 网站根目录
url_seed = 'http://www.jianshu.com/c/9b4685b6357c?page=%d'  # 要爬取的页面地址模板

paperList = crawl_list(url_seed)
paperAll = crawl_paper_tag(paperList, url_root)
save_data(paperAll)
exit(-1)

Python Console的输出结果如下

downloading:  http://www.jianshu.com/c/9b4685b6357c?page=1
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=2
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=3
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=4
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=5
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=6
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=7
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=8
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=9
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=10
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=11
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=12
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=13
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=14
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=15
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=16
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=17
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=18
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=19
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=20
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=21
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=22
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=23
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=24
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=25
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=26
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=27
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=28
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=29
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=30
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=31
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=32
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=33
downloading:  http://www.jianshu.com/c/9b4685b6357c?page=34
total paper num:  330
downloading:  http://www.jianshu.com/p/756aef98265d
downloading:  http://www.jianshu.com/p/bbcf8beac941
downloading:  http://www.jianshu.com/p/3adadd861937
downloading:  http://www.jianshu.com/p/42ada5f6cef3
downloading:  http://www.jianshu.com/p/56642987c947
downloading:  http://www.jianshu.com/p/53a1ed89ea3c
downloading:  http://www.jianshu.com/p/40690286401a
downloading:  http://www.jianshu.com/p/f9a1e28fa764
downloading:  http://www.jianshu.com/p/6555c7fc55ca
downloading:  http://www.jianshu.com/p/992fb8568344
downloading:  http://www.jianshu.com/p/9b8522aa85fe
downloading:  http://www.jianshu.com/p/c92214b2a164
downloading:  http://www.jianshu.com/p/2c1656991fcb
downloading:  http://www.jianshu.com/p/6cfce4cc2f7f
downloading:  http://www.jianshu.com/p/2b709a43f33d
downloading:  http://www.jianshu.com/p/8bff40bc35b3
downloading:  http://www.jianshu.com/p/958a8a7eb9f3
downloading:  http://www.jianshu.com/p/db232275df93
downloading:  http://www.jianshu.com/p/469196dafa48
downloading:  http://www.jianshu.com/p/48b88a06952e
downloading:  http://www.jianshu.com/p/a1624dfcf27f
downloading:  http://www.jianshu.com/p/364c1db69f38
downloading:  http://www.jianshu.com/p/02c23bef2877
downloading:  http://www.jianshu.com/p/e9fef05094f8
downloading:  http://www.jianshu.com/p/99657d135922
downloading:  http://www.jianshu.com/p/4f38600dae7c
downloading:  http://www.jianshu.com/p/39d6793a6554
downloading:  http://www.jianshu.com/p/87f36332b707
downloading:  http://www.jianshu.com/p/2841c81d57fc
downloading:  http://www.jianshu.com/p/97ff0beca873
downloading:  http://www.jianshu.com/p/8318f5b722cf
downloading:  http://www.jianshu.com/p/887889c6daee
downloading:  http://www.jianshu.com/p/6450dce856fd
downloading:  http://www.jianshu.com/p/d7267956035a
downloading:  http://www.jianshu.com/p/616642976ded
downloading:  http://www.jianshu.com/p/9086d0300d1a
downloading:  http://www.jianshu.com/p/defeeb920c3a
downloading:  http://www.jianshu.com/p/f20bc50ad0e8
downloading:  http://www.jianshu.com/p/456ab3a6ef71
downloading:  http://www.jianshu.com/p/6a91f948b62d
downloading:  http://www.jianshu.com/p/f21f01a92521
downloading:  http://www.jianshu.com/p/de3455ed089c
downloading:  http://www.jianshu.com/p/50a2ba489685
downloading:  http://www.jianshu.com/p/d32d7ab5063a
downloading:  http://www.jianshu.com/p/545581b0c7dd
downloading:  http://www.jianshu.com/p/f6c26ef0f4cc
downloading:  http://www.jianshu.com/p/ed8dc3802927
downloading:  http://www.jianshu.com/p/b4ac4473a55d
downloading:  http://www.jianshu.com/p/6585d58f582a
downloading:  http://www.jianshu.com/p/e76c242c7d6a
downloading:  http://www.jianshu.com/p/f68d28d3b862
downloading:  http://www.jianshu.com/p/92a66ccc8998
downloading:  http://www.jianshu.com/p/13a68ac7afdd
downloading:  http://www.jianshu.com/p/ba4100af215a
downloading:  http://www.jianshu.com/p/8f6ee3b1efeb
downloading:  http://www.jianshu.com/p/3a5975d6ac55
downloading:  http://www.jianshu.com/p/c41624a83b71
downloading:  http://www.jianshu.com/p/5250518f5cc5
downloading:  http://www.jianshu.com/p/0b0b7c33be57
downloading:  http://www.jianshu.com/p/aed64f7e647b
downloading:  http://www.jianshu.com/p/0329f87c9ae4
downloading:  http://www.jianshu.com/p/11b3dbb05c39
downloading:  http://www.jianshu.com/p/ed9ec88e71e4
downloading:  http://www.jianshu.com/p/a425acdaf77e
downloading:  http://www.jianshu.com/p/8f7102c74a4f
downloading:  http://www.jianshu.com/p/a36e997b8e59
downloading:  http://www.jianshu.com/p/a91c54f96ded
downloading:  http://www.jianshu.com/p/ba00a9852a02
downloading:  http://www.jianshu.com/p/f89c4032a0b2
downloading:  http://www.jianshu.com/p/e9f6b15318be
downloading:  http://www.jianshu.com/p/59e0da43136e
downloading:  http://www.jianshu.com/p/e0ae002925bd
downloading:  http://www.jianshu.com/p/1099c3a74336
downloading:  http://www.jianshu.com/p/cfaf85b24281
downloading:  http://www.jianshu.com/p/acc47733334f
downloading:  http://www.jianshu.com/p/233ff48d668e
downloading:  http://www.jianshu.com/p/ad10d79255f8
downloading:  http://www.jianshu.com/p/e121b1a420ad
downloading:  http://www.jianshu.com/p/5e4a86f8025c
downloading:  http://www.jianshu.com/p/5a8b8ce0a395
downloading:  http://www.jianshu.com/p/be0192aa6486
downloading:  http://www.jianshu.com/p/29e304a61d32
downloading:  http://www.jianshu.com/p/fa7dd359d7a8
downloading:  http://www.jianshu.com/p/ed93f7f344d0
downloading:  http://www.jianshu.com/p/aa1121232dfd
downloading:  http://www.jianshu.com/p/4a8749704ebf
downloading:  http://www.jianshu.com/p/ed499f4ecdd1
downloading:  http://www.jianshu.com/p/29062bca16aa
downloading:  http://www.jianshu.com/p/e0bd6bfad10b
downloading:  http://www.jianshu.com/p/c0f3d36d0c7a
downloading:  http://www.jianshu.com/p/10b429fd9c4d
downloading:  http://www.jianshu.com/p/98cc73755a22
downloading:  http://www.jianshu.com/p/2ccd37ae73e2
downloading:  http://www.jianshu.com/p/9ee12067f35e
downloading:  http://www.jianshu.com/p/486afcd4c36c
downloading:  http://www.jianshu.com/p/9457100d8763
downloading:  http://www.jianshu.com/p/90914aef3636
downloading:  http://www.jianshu.com/p/bf5984fb299a
downloading:  http://www.jianshu.com/p/0c007dbbf728
downloading:  http://www.jianshu.com/p/f0436668cb72
downloading:  http://www.jianshu.com/p/0c0e3ace0da1
downloading:  http://www.jianshu.com/p/ae5f78b40f17
downloading:  http://www.jianshu.com/p/70b7505884e9
downloading:  http://www.jianshu.com/p/7b2e81589a4f
downloading:  http://www.jianshu.com/p/7e2fccb4fad9
downloading:  http://www.jianshu.com/p/b7eef4033a09
downloading:  http://www.jianshu.com/p/4b4e0c343d3e
downloading:  http://www.jianshu.com/p/605644d688ff
downloading:  http://www.jianshu.com/p/88d0addf64fa
downloading:  http://www.jianshu.com/p/1fe31cbddc78
downloading:  http://www.jianshu.com/p/4981df2eefe7
downloading:  http://www.jianshu.com/p/b3e8e9cb0141
downloading:  http://www.jianshu.com/p/673b768c6084
downloading:  http://www.jianshu.com/p/a1a2dabb4bc2
downloading:  http://www.jianshu.com/p/a8037a38e219
downloading:  http://www.jianshu.com/p/926013888e3e
downloading:  http://www.jianshu.com/p/9e7cfcc85a57
downloading:  http://www.jianshu.com/p/1ea730c97aae
downloading:  http://www.jianshu.com/p/d1acbed69f45
downloading:  http://www.jianshu.com/p/faf2f4107b9b
downloading:  http://www.jianshu.com/p/f7354d1c5abf
downloading:  http://www.jianshu.com/p/1a2a07611fd8
downloading:  http://www.jianshu.com/p/cf2edecdba77
downloading:  http://www.jianshu.com/p/7cb84cfa56fa
downloading:  http://www.jianshu.com/p/99fd951a0b8b
downloading:  http://www.jianshu.com/p/7e556f17021a
downloading:  http://www.jianshu.com/p/e50c863bb465
downloading:  http://www.jianshu.com/p/356a579062aa
downloading:  http://www.jianshu.com/p/5057ab6f9ad5
downloading:  http://www.jianshu.com/p/40cc7d239513
downloading:  http://www.jianshu.com/p/71c02ef761ac
downloading:  http://www.jianshu.com/p/888a580b2384
downloading:  http://www.jianshu.com/p/f26085aadd47
downloading:  http://www.jianshu.com/p/ada67bd7c56f
downloading:  http://www.jianshu.com/p/9f9fb337be0c
downloading:  http://www.jianshu.com/p/9632ba906ca2
downloading:  http://www.jianshu.com/p/fd9536a0acfb
downloading:  http://www.jianshu.com/p/f6420cce3040
downloading:  http://www.jianshu.com/p/62c0a5122fa8
downloading:  http://www.jianshu.com/p/333dacb0e1b2
downloading:  http://www.jianshu.com/p/5a6c4b8e7700
downloading:  http://www.jianshu.com/p/394856545ab0
downloading:  http://www.jianshu.com/p/17f99100525a
downloading:  http://www.jianshu.com/p/46e82e4fe324
downloading:  http://www.jianshu.com/p/e08d1a03045f
downloading:  http://www.jianshu.com/p/3c71839bc660
downloading:  http://www.jianshu.com/p/e1b28de0a1e4
downloading:  http://www.jianshu.com/p/ea40c6da9fec
downloading:  http://www.jianshu.com/p/601d3a488a58
downloading:  http://www.jianshu.com/p/59ca82a11f87
downloading:  http://www.jianshu.com/p/ff772050bd96
downloading:  http://www.jianshu.com/p/729edfc613aa
downloading:  http://www.jianshu.com/p/366c2594f24b
downloading:  http://www.jianshu.com/p/bd9a27c4e2a8
downloading:  http://www.jianshu.com/p/41b1ee54d766
downloading:  http://www.jianshu.com/p/8fbe3a7b4764
downloading:  http://www.jianshu.com/p/d43125a4ff44
downloading:  http://www.jianshu.com/p/afa17bc391b7
downloading:  http://www.jianshu.com/p/649167e0e2f4
downloading:  http://www.jianshu.com/p/76238014a03f
downloading:  http://www.jianshu.com/p/68423bfc4c4e
downloading:  http://www.jianshu.com/p/c28207b3c71d
downloading:  http://www.jianshu.com/p/5eb037498c48
downloading:  http://www.jianshu.com/p/c6591991d1ca
downloading:  http://www.jianshu.com/p/e04bcac99c8d
downloading:  http://www.jianshu.com/p/74ef104a9f45
downloading:  http://www.jianshu.com/p/f75128ec3ea3
downloading:  http://www.jianshu.com/p/3a95a09cda40
downloading:  http://www.jianshu.com/p/b1a9daef3423
downloading:  http://www.jianshu.com/p/8a0479f55b21
downloading:  http://www.jianshu.com/p/b4670cb9e998
downloading:  http://www.jianshu.com/p/0565cd673282
downloading:  http://www.jianshu.com/p/819a202adecd
downloading:  http://www.jianshu.com/p/b5c292e093a2
downloading:  http://www.jianshu.com/p/6920d5e48b31
downloading:  http://www.jianshu.com/p/d2dc5aa9bf8f
downloading:  http://www.jianshu.com/p/cbfab5db7f6f
downloading:  http://www.jianshu.com/p/af4765b703f0
downloading:  http://www.jianshu.com/p/761a73b7eea2
downloading:  http://www.jianshu.com/p/cb4f8ab1b380
downloading:  http://www.jianshu.com/p/bb736600b483
downloading:  http://www.jianshu.com/p/1d6fc1a9406b
downloading:  http://www.jianshu.com/p/8448ac374dc1
downloading:  http://www.jianshu.com/p/01385e2dd129
downloading:  http://www.jianshu.com/p/062b8dfca144
downloading:  http://www.jianshu.com/p/4ffca0a43476
downloading:  http://www.jianshu.com/p/1a935c2dc911
downloading:  http://www.jianshu.com/p/ce0e0773c6ec
downloading:  http://www.jianshu.com/p/4b944b22fe83
downloading:  http://www.jianshu.com/p/856c8d648e20
downloading:  http://www.jianshu.com/p/8baa664ea613
downloading:  http://www.jianshu.com/p/77876ef45ab4
downloading:  http://www.jianshu.com/p/d234a015fa90
downloading:  http://www.jianshu.com/p/1b42a12dac14
downloading:  http://www.jianshu.com/p/bfd9b3954038
downloading:  http://www.jianshu.com/p/979b4c5c1857
downloading:  http://www.jianshu.com/p/c1163e39a42e
downloading:  http://www.jianshu.com/p/6233788a8abb
downloading:  http://www.jianshu.com/p/f1ef93a6c033
downloading:  http://www.jianshu.com/p/b223e54fe5ee
downloading:  http://www.jianshu.com/p/f7dc92913f33
downloading:  http://www.jianshu.com/p/e492d3acfe38
downloading:  http://www.jianshu.com/p/8f6b5a1bb3fa
downloading:  http://www.jianshu.com/p/148b7cc83bcd
downloading:  http://www.jianshu.com/p/7acf291b2a5e
downloading:  http://www.jianshu.com/p/3b3bca4281aa
downloading:  http://www.jianshu.com/p/8982ad63eb85
downloading:  http://www.jianshu.com/p/8ca88a90ea17
downloading:  http://www.jianshu.com/p/3b47b36cc8e8
downloading:  http://www.jianshu.com/p/71b968bd8abb
downloading:  http://www.jianshu.com/p/bb4a81624af1
downloading:  http://www.jianshu.com/p/9c7fbcac3461
downloading:  http://www.jianshu.com/p/872a67eed7af
downloading:  http://www.jianshu.com/p/f756bf0beb26
downloading:  http://www.jianshu.com/p/41c14ef3e59a
downloading:  http://www.jianshu.com/p/e5d13e351320
downloading:  http://www.jianshu.com/p/dffdaf11bd4c
downloading:  http://www.jianshu.com/p/733475b6900d
downloading:  http://www.jianshu.com/p/6dbadc78d231
downloading:  http://www.jianshu.com/p/9258b0495021
downloading:  http://www.jianshu.com/p/1292d7a3805e
downloading:  http://www.jianshu.com/p/e2ee86a8a32b
downloading:  http://www.jianshu.com/p/74db357c7252
downloading:  http://www.jianshu.com/p/23144099e9f8
downloading:  http://www.jianshu.com/p/e99dacbf5c44
downloading:  http://www.jianshu.com/p/d6243f087bd9
downloading:  http://www.jianshu.com/p/b6359185fc26
downloading:  http://www.jianshu.com/p/05c15b9f16f1
downloading:  http://www.jianshu.com/p/83cc892eb24a
downloading:  http://www.jianshu.com/p/eb01f9002091
downloading:  http://www.jianshu.com/p/210aacd31ef7
downloading:  http://www.jianshu.com/p/6a7afc98c868
downloading:  http://www.jianshu.com/p/14967ec6e954
downloading:  http://www.jianshu.com/p/3dfedf60de62
downloading:  http://www.jianshu.com/p/81819f27a7d8
downloading:  http://www.jianshu.com/p/bab0c09416ee
downloading:  http://www.jianshu.com/p/bd78a49c9d23
downloading:  http://www.jianshu.com/p/289e51eb6446
downloading:  http://www.jianshu.com/p/2e64c2045be5
downloading:  http://www.jianshu.com/p/e5475131d03f
downloading:  http://www.jianshu.com/p/565500cfb5a4
downloading:  http://www.jianshu.com/p/d578d5e2755f
downloading:  http://www.jianshu.com/p/2364064e0bc9
downloading:  http://www.jianshu.com/p/c9e1dffad756
downloading:  http://www.jianshu.com/p/b9407b2c22a4
downloading:  http://www.jianshu.com/p/542f41879879
downloading:  http://www.jianshu.com/p/df7b35249975
downloading:  http://www.jianshu.com/p/1729787990e7
downloading:  http://www.jianshu.com/p/67ae9d87cf3c
downloading:  http://www.jianshu.com/p/296ae7538d1f
downloading:  http://www.jianshu.com/p/ee43c55123f8
downloading:  http://www.jianshu.com/p/d882831868fb
downloading:  http://www.jianshu.com/p/ec3c57d6a4c7
downloading:  http://www.jianshu.com/p/23a905cf936b
downloading:  http://www.jianshu.com/p/56967004f8c4
downloading:  http://www.jianshu.com/p/799c51fbe5f1
downloading:  http://www.jianshu.com/p/3f06c9f69142
downloading:  http://www.jianshu.com/p/020f0281f1df
downloading:  http://www.jianshu.com/p/087ce1951647
downloading:  http://www.jianshu.com/p/cc3b5d76c587
downloading:  http://www.jianshu.com/p/7b946e6d6861
downloading:  http://www.jianshu.com/p/5d8a3205e28e
downloading:  http://www.jianshu.com/p/81d17436f29e
downloading:  http://www.jianshu.com/p/7107b67c47bc
downloading:  http://www.jianshu.com/p/f382741c2736
downloading:  http://www.jianshu.com/p/09b19b8f8886
downloading:  http://www.jianshu.com/p/2c02a7b0b382
downloading:  http://www.jianshu.com/p/27a78b2016e0
downloading:  http://www.jianshu.com/p/5dc5dfe26148
downloading:  http://www.jianshu.com/p/d5bc50d8e0a2
downloading:  http://www.jianshu.com/p/39eb230e6f15
downloading:  http://www.jianshu.com/p/37e927476dfe
downloading:  http://www.jianshu.com/p/13d76e7741c0
downloading:  http://www.jianshu.com/p/02f33063c258
downloading:  http://www.jianshu.com/p/9a9280de68f8
downloading:  http://www.jianshu.com/p/99ae5b28a51f
downloading:  http://www.jianshu.com/p/a32f27199846
downloading:  http://www.jianshu.com/p/11c103c03d4a
downloading:  http://www.jianshu.com/p/910662d6e881
downloading:  http://www.jianshu.com/p/2f7d10b2e508
downloading:  http://www.jianshu.com/p/a9c7970bc949
downloading:  http://www.jianshu.com/p/bc75ab89fac0
downloading:  http://www.jianshu.com/p/d44cc7e9a0a9
downloading:  http://www.jianshu.com/p/9dee9886b140
downloading:  http://www.jianshu.com/p/be384fd73bdb
downloading:  http://www.jianshu.com/p/412f8eab2599
downloading:  http://www.jianshu.com/p/ff2d4eadebde
downloading:  http://www.jianshu.com/p/0ee1f0bfc8cb
downloading:  http://www.jianshu.com/p/dc07545c6607
downloading:  http://www.jianshu.com/p/e71e5d7223bb
downloading:  http://www.jianshu.com/p/b5c31a2eeb8b
downloading:  http://www.jianshu.com/p/6ef6b9a56b50
downloading:  http://www.jianshu.com/p/85da47fddad7
downloading:  http://www.jianshu.com/p/d0de8ee83ea1
downloading:  http://www.jianshu.com/p/f0063d735a5c
downloading:  http://www.jianshu.com/p/a4beefd8cfc2
downloading:  http://www.jianshu.com/p/2c557a1bfa04
downloading:  http://www.jianshu.com/p/cd971afcb207
downloading:  http://www.jianshu.com/p/430b5bea974d
downloading:  http://www.jianshu.com/p/74042ba10c0d
downloading:  http://www.jianshu.com/p/86117613b7a6
downloading:  http://www.jianshu.com/p/0a6977eb686d
downloading:  http://www.jianshu.com/p/4077cbc4dd37
downloading:  http://www.jianshu.com/p/13840057782d
downloading:  http://www.jianshu.com/p/ba43beaa186a
downloading:  http://www.jianshu.com/p/c0c0a3ed35d4
downloading:  http://www.jianshu.com/p/217a4578f9ab
downloading:  http://www.jianshu.com/p/873613065502
downloading:  http://www.jianshu.com/p/90efe88727fe
downloading:  http://www.jianshu.com/p/4931d66276c3
downloading:  http://www.jianshu.com/p/8088d1bede8d
downloading:  http://www.jianshu.com/p/a015b756a803
downloading:  http://www.jianshu.com/p/4dda2425314a
downloading:  http://www.jianshu.com/p/45df7e3ecc78
downloading:  http://www.jianshu.com/p/8ca518b3b2d5
downloading:  http://www.jianshu.com/p/8266f0c736f9
downloading:  http://www.jianshu.com/p/460a8eed5cfa
downloading:  http://www.jianshu.com/p/75fc36aec98e
downloading:  http://www.jianshu.com/p/169403f7e40c
downloading:  http://www.jianshu.com/p/7c54cd046d4b
downloading:  http://www.jianshu.com/p/b5165468a32b
downloading:  http://www.jianshu.com/p/4a3fbcb06981
downloading:  http://www.jianshu.com/p/c88a4453dd6d
downloading:  http://www.jianshu.com/p/b4e2e5e31154
downloading:  http://www.jianshu.com/p/7240db1ba0af
downloading:  http://www.jianshu.com/p/62e127dbb73c
downloading:  http://www.jianshu.com/p/1fa23219270d
downloading:  http://www.jianshu.com/p/e72c8ef71e49
downloading:  http://www.jianshu.com/p/4b57424173a0
downloading:  http://www.jianshu.com/p/6f4a7a1ef85c

下面是数据库写入出错的部分

sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\xA4\x93\xE5\xA5...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x82\xF0\x9F...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x8B\xE6\x95...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\xA4\x97\xE4\xB8...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x93\xE2\x80...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x9D\xE4\xB9...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
sql process error!||errmsg:1366 (HY000): Incorrect string value: '\xF0\x9F\x98\xADn=...' for column 'content' at row 1||sql:INSERT paper(`reward_cnt`,`title`,`url`,`_modify_time`,`abstract`,`author`,`content`,`_create_time`,`pub_time`,`read_cnt`,`comment_cnt`,`like_cnt`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)

抓取的结果导入数据库后如下：

课程作业-爬虫入门04-2-构建爬虫-WilliamZeng-20170812_第2张图片

导入数据库的结果