抓取某新闻客户端的数据
注意事项
- 在抓取数据之前先清除缓存，缓存会影响抓包
过滤get请求
通过ping命令获取新闻数据的目标IP地址，封包信息中网址带有163可能就是新闻客户端的url
拿到目标IP地址，就可以通过目标IP地址过滤数据
用浏览器打开抓到的url，即可得到新闻的数据
http://c.m.163.com/nc/article/list/T1467284926140/0-20.html
新闻分类tid
url组织形式http://c.m.163.com/nc/article/list/tid/startindex-count.html
其中tid是新闻类别的id，startindex是数据的起始索引，count是新闻数据的条数
- 体育T1348649079062
- 头条T1467284926140
- 娱乐T1348648517839
- 要闻T1348647909107
新闻Tab标签
http://c.m.163.com/nc/topicset/android/subscribe/manage/listspecial.html
{
"tList": [
{
"template": "normal1",
"topicid": "0001899N",
"hasCover": false,
"weburl": "http://www.163.com/",
"alias": "Top News",
"subnum": "3.2万",
"recommendOrder": 0,
"isNew": 0,
"hashead": 1,
"img": "",
"isHot": 0,
"hasIcon": false,
"cid": "C1348646712614",
"recommend": "0",
"headLine": true,
"hasAD": 1,
"color": "",
"bannerOrder": 0,
"tname": "头条",
"ename": "androidnews",
"showType": "comment",
"special": 0,
"tid": "T1348647909107",
"ad_type": 1
},
{
"template": "normal1",
"topicid": "0001899N;000187QQ",
"hasCover": false,
"alias": "yaowenspecial",
"subnum": "10.6万",
"recommendOrder": 0,
"isNew": 0,
"hashead": 1,
"img": "",
"isHot": 0,
"hasIcon": false,
"cid": "C1348647991705",
"recommend": "0",
"headLine": false,
"hasAD": 1,
"color": "",
"bannerOrder": 0,
"tname": "要闻",
"ename": "yaowenspecial",
"showType": "comment",
"special": 0,
"tid": "T1467284926140",
"ad_type": 1
},
{
"template": "recommend",
"topicid": "00037VVH",
"hasCover": false,
"weburl": "http://ent.163.com/",
"alias": "Entertainment",
"subnum": "超过1000万",
"recommendOrder": 120,
"isNew": 0,
"hashead": 1,
"img": "T1348648517839",
"isHot": 0,
"hasIcon": true,
"cid": "C1348648351901",
"recommend": "1",
"headLine": false,
"hasAD": 1,
"color": "",
"bannerOrder": 0,
"tname": "娱乐",
"ename": "yule",
"showType": "comment",
"special": 0,
"tid": "T1348648517839",
"ad_type": 1
}
]
}
json数据中的tid即为上文中的新闻类别id
新闻详情页
http://c.m.163.com/nc/article/docid/full.html
https://c.m.163.com/news/a/docid.html
例如：http://c.m.163.com/nc/article/CG4A99320001899O/full.html
其中CG4A99320001899O是docid，如图所示
视频
tablist
URL：http://c.m.163.com/recommend/video/tablist
[
{
"categorys": "",
"cname": "推荐",
"ename": "Video_Recom"
},
{
"categorys": "自制剧,翻唱,恶搞吐槽",
"cname": "搞笑",
"ename": "Video_Funny"
},
{
"categorys": "国际新闻,社会新闻,社会实验,交通事故,纪实,街头采访,拍客",
"cname": "现场",
"ename": "Video_Scene"
},
{
"categorys": "娱乐资讯",
"cname": "八卦",
"ename": "Video_Gossip"
},
{
"categorys": "宠物,植物视频,萌宝,治愈系列短片",
"cname": "萌物",
"ename": "Video_Adorable"
},
{
"categorys": "电视剧,综艺,电影",
"cname": "影视",
"ename": "Video_Movies"
},
{
"categorys": "美妆,服饰,情感,男士时尚,奢侈品,星座,旅游,美食攻略,食谱,公益短片,小窍门,DIY,历史,教育科普,公开课,职场教育,保健养生,健身,减肥,消费理财,指数分析,财经节目,宝宝养育,早教,准妈妈,读书,绘画,设计,家居,全景",
"cname": "涨姿势",
"ename": "Video_Knowledge"
},
{
"categorys": "时尚达人,帅哥,美女,车模,秀场",
"cname": "美女",
"ename": "Video_Beauty"
},
{
"categorys": "相声小品",
"cname": "小品",
"ename": "Video_Opusculum"
},
{
"categorys": "猎奇视频,奇葩视频",
"cname": "猎奇",
"ename": "Video_Curious"
},
{
"categorys": "音乐",
"cname": "音乐",
"ename": "Video_Music"
},
{
"categorys": "军事武器,军事节目",
"cname": "军武",
"ename": "Video_Military"
},
{
"categorys": "科学,科普,科技前沿,航空",
"cname": "黑科技",
"ename": "Video_Technology"
},
{
"categorys": "动漫",
"cname": "二次元",
"ename": "Video_Comic"
}
]
搞笑
http://c.m.163.com/recommend/getChanListNews?channel=T1457068979049&subtab=Video_Recom&size=20&offset=0&fn=1&prog=&passport=&devId=ujHjV7eaPdxs%2Bn3auGxVEw%3D%3D&lat=J%2FL6Pf45ENZ4SNmCfL%2B5LA%3D%3D&lon=h9rfYmL2wkY8pQ2vQIj%2BjQ%3D%3D&version=22.1&net=wifi&ts=1491786621&sign=o%2Bla3K%2FxO2acwdCT2LSXdB33jstfpUXvMpvQEmViIqx48ErR02zJ6%2FKXOnxX046I&encryption=1&canal=meizu_store2014_news&mac=6K9Bh52MduyjsI60TBWf%2FMniKsPhqwUPmAMQf86XLBg%3D&open=&openpath=
推荐
http://c.m.163.com/recommend/getChanListNews?channel=T1457068979049&subtab=Video_Recom&size=20&offset=20&fn=1&prog=&passport=&devId=ujHjV7eaPdxs%2Bn3auGxVEw%3D%3D&lat=J%2FL6Pf45ENZ4SNmCfL%2B5LA%3D%3D&lon=h9rfYmL2wkY8pQ2vQIj%2BjQ%3D%3D&version=22.1&net=wifi&ts=1491786625&sign=9ccPnLrAh%2FLToMq4nsZNAqUPO0tEXQ%2FkVEZdkMOqQRZ48ErR02zJ6%2FKXOnxX046I&encryption=1&canal=meizu_store2014_news&mac=6K9Bh52MduyjsI60TBWf%2FMniKsPhqwUPmAMQf86XLBg%3D&open=&openpath=
现场
http://c.m.163.com/recommend/getChanListNews?channel=T1457068979049&subtab=Video_Scene&size=20&offset=0&fn=1&prog=&passport=&devId=ujHjV7eaPdxs%2Bn3auGxVEw%3D%3D&lat=J%2FL6Pf45ENZ4SNmCfL%2B5LA%3D%3D&lon=h9rfYmL2wkY8pQ2vQIj%2BjQ%3D%3D&version=22.1&net=wifi&ts=1491786633&sign=NHvp%2BSaWwNBlhl%2BogIBr16Pe9B7P4hPB4nT4GVQu0%2BF48ErR02zJ6%2FKXOnxX046I&encryption=1&canal=meizu_store2014_news&mac=6K9Bh52MduyjsI60TBWf%2FMniKsPhqwUPmAMQf86XLBg%3D&open=&openpath=