爬虫小试牛刀（知乎用户关注列表爬取 python+requests+json）

基于环境：python3.6 + requests + json 模块
调试浏览器：chrome浏览器

环境安装

1. python可自行到官网下载
2. requests模块和json模块安装，直接pip

   pip install requests
   pip install json

目标分析

1. 目标用户
   廖雪峰:https://www.zhihu.com/people/liaoxuefeng

2. 按f12开启调试工具

3. 将文件类型 定位到 XHR，并点击 关注者 如下图：
   请求返回的数据一般能够在XHR 或者 JS 中可以找到，命名前缀都会与我们要的东西相关，在左边列表中找到followers开头的文件，点击，在右边的Preview下可以看到用户的关注用户数据
   

4. 我们从 Preview 切到 Headers 可以看到请求信息和响应信息，第一个便是请求 url

   Request URL: https://www.zhihu.com/api/v4/members/liaoxuefeng/followers?include=data[*].answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge[%3F(type%3Dbest_answerer)].topics&offset=0&limit=20

   先测试一下：

   import requests
   
   url = "https://www.zhihu.com/api/v4/members/liaoxuefeng/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset=0&limit=20"
   response = requests.get(url)
   
   print(response.status_code)
   print(response.content)
   

   打印返回信息：
   

   返回了400，这是因为服务器的 反爬虫机制 ，所以我们需要传入一些请求参数，模拟浏览器访问
   让我们回到浏览器的调试页面，查看 Headers 下的参数
   
   我们需要的是authority, cookie, user-agent, 除了这些，还可以传入其他的参数来控制获取的用户条数
   Headers 信息拉到最下面可以看到
   
   参数代码如下：

   headers = {
       "authority": "www.zhihu.com",
       "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
       "cookie": '_zap=54e71ae8-a40f-4f19-937a-4ff6dca65560; d_c0="AEDCETXw0wyPTlgyJj5cZ531pR83mHU-s6Y=|1513161369"; __DAYU_PP=qverFMZIi3babY3j3uBb2564ad3aa576; z_c0=Mi4xUC1tREJ3QUFBQUFBUU1JUk5mRFREQmNBQUFCaEFsVk40NVdFWEFEcmNZdEJzZnAxMzFZSHZMQTktY2w4MEFjRWxR|1536640995|ef414cc20542a5c03559cddfc878fa5c1beb1355; __utma=51854390.1221214925.1513161370.1513161370.1536642013.2; __utmz=51854390.1536642013.2.1.utmcsr=zhihu.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmv=51854390.100--|2=registration_date=20180126=1^3=entry_date=20180126=1; q_c1=b6a66e52e9344bfdb659ec03c65b2cab|1537170416000|1534140717000; _xsrf=9e20cf02-3c14-404e-b7c2-3ead27cfdb2e; tst=r; tgw_l7_route=1c2b7f9548c57cd7d5a535ac4812e20e'
   }
   params = {
       "include": "data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics",
       "offset": 0,
       "limit": 20
   }
   

   然后就可以获取到我们想要的东西了（截图部分内容），我用的是jupyter notebook ,输出看起来比较乱，大家可以用其他格式化工具
   
   还有一个就是输出信息得到的用户名无法显示中文，这时候我们需要用到另一个模块 json，使用户名显示中文，代码如下：

   import json
   myjson=json.loads(content)
   newjson=json.dumps(myjson,ensure_ascii=False)
   

   这样就能显示中文了
   

5. 最后，如果想要得到更多的用户数据，只需要修改 params 参数，offset 代表起始参数，limit 代表获取的用户个数。

完整代码：

# -*- coding:utf-8 -*- 
import requests
import json

url = "https://www.zhihu.com/api/v4/members/jia-zhen-86-28/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cgender%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset=0&limit=20"
headers = {
	"authority": "www.zhihu.com",
  	"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
	"cookie": '_zap=54e71ae8-a40f-4f19-937a-4ff6dca65560; d_c0="AEDCETXw0wyPTlgyJj5cZ531pR83mHU-s6Y=|1513161369"; __DAYU_PP=qverFMZIi3babY3j3uBb2564ad3aa576; z_c0=Mi4xUC1tREJ3QUFBQUFBUU1JUk5mRFREQmNBQUFCaEFsVk40NVdFWEFEcmNZdEJzZnAxMzFZSHZMQTktY2w4MEFjRWxR|1536640995|ef414cc20542a5c03559cddfc878fa5c1beb1355; __utma=51854390.1221214925.1513161370.1513161370.1536642013.2; __utmz=51854390.1536642013.2.1.utmcsr=zhihu.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utmv=51854390.100--|2=registration_date=20180126=1^3=entry_date=20180126=1; q_c1=b6a66e52e9344bfdb659ec03c65b2cab|1537170416000|1534140717000; _xsrf=9e20cf02-3c14-404e-b7c2-3ead27cfdb2e; tst=r; tgw_l7_route=1c2b7f9548c57cd7d5a535ac4812e20e'
}
params = {
"include": "data[*].answer_count,articles_count,gender,follower_count,is_followed,is_following,badge[?(type=best_answerer)].topics",
"offset": 0,
"limit": 20
}

response = requests.get(url, params = params, headers = headers)
content = response.content.decode("utf8")
myjson=json.loads(content)
newjson=json.dumps(myjson,ensure_ascii=False)