今日头条热点新闻-获取_signature
一、简单说明
热点新闻等资讯信息绕不过今日头条，今天讲解下如何通过拿到_signature参数来获取头条新闻。
二、发现问题
今日头条的热点数据是ajax动态刷新得到的，通过筛选测试有数据的url为https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=Yu0DpgAAOZbIwkMmDCYgkGLtA7，其中_signature是需要得到的关键字参数，在js中搜寻中得到生成文件home_e14d2a0.js,生成流程
调试发现"refresh" === t ? 0 : r.params.max_behot_time_tmp可以直接赋值为0；因为生成_signature的js算法比较复杂，网上有很多这方面的博客，大多失效。我们直接通过selenium来获取数据。
三、获取数据
至于selenium的安装和使用在这里就不作过多描述，直接上代码吧！
'''
今日头条
js破解_signetrue
'''
from selenium import webdriver
import requests
import time
import json
from utils.get_proxy import func_proxy
# 进入浏览器设置
options = webdriver.ChromeOptions()
# 设置中文
options.add_argument('lang=zh_CN.UTF-8')
options.set_headless()
options.add_argument(
'user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"')
# 设置代理
service_args = ['--proxy={}'.format(func_proxy()['http']), '--proxy-type=socks5']
brower = webdriver.Chrome(chrome_options=options, service_args=service_args)
brower.get('https://www.toutiao.com/ch/news_hot/')
sinature = brower.execute_script('return TAC.sign(0)')
print(sinature)
"""获取cookie"""
cookie = brower.get_cookies()
cookie = [item['name'] + "=" + item['value'] for item in cookie]
cookiestr = '; '.join(item for item in cookie)
brower.quit()
print(cookiestr)
header1 = {
'Host': 'www.toutiao.com',
'User-Agent': '"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"',
'Referer': 'https://www.toutiao.com/ch/news_hot/',
"Cookie": cookiestr
}
url = 'https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature={}'.format(
sinature)
print(url)
html = requests.get(url, headers=header1, proxies=func_proxy(), verify=False)
print(html.content.decode('unicode_escape'))
所获取到的数据
{'http': 'http://124.230.0.20:42546'}
ZuU44QAAPZ4TmjxZv6zL.WblOP
UM_distinctid=1656b863a92f8-0e645b22313a8b-71272415-75300-1656b863a93f3; tt_webid=6593235575644423694; __tasessionId=lkyrlacv11535107283377; csrftoken=c50e03d22a62281c7ce646395bf32f17; uuid="w:f37f35cde0e24093943a530d5557d608"; CNZZDATA1259612802=222376684-1535103528-%7C1535103528
https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=ZuU44QAAPZ4TmjxZv6zL.WblOP
C:\Anaconda\lib\site-packages\requests\packages\urllib3\connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
{"has_more": false, "message": "success", "data": [{"chinese_tag": "其它", "is_feed_ad": false, "tag_url": "search/?keyword=%E5%85%B6%E5%AE%83", "title": "台风灾害致山东13市500余万人受灾 潍坊灾情最重", "single_mode": true, "middle_mode": true, "tag": "news", "label": ["气象灾害", "台风"], "behot_time": 1535107286, "source_url": "/api/pc/subject/6592925627836268808/", "source": "专题", "more_mode": false, "article_genre": "article", "image_url": "//p3.pstatp.com/list/190x124/aad60018f524a156bffb", "has_gallery": false, "group_source": 4, "item_id": "6592925627836268808", "comments_count": 34, "group_id": "6592925627836268808", "middle_image": {"url": "http://p3.pstatp.com/list/aad60018f524a156bffb", "width": 640, "url_list": [{"url": "http://p3.pstatp.com/list/aad60018f524a156bffb"}, {"url": "http://pb9.pstatp.com/list/aad60018f524a156bffb"}, {"url": "http://pb1.pstatp.com/list/aad60018f524a156bffb"}], "uri": "list/aad60018f524a156bffb", "height": 300}}, {"chinese_tag": "社会", "media_avatar_url": "//p3.pstatp.com/large/ca400072481685ad43b", "is_feed_ad": false, "tag_url": "news_society", "title": "高铁“霸座”男被处治安罚款200元 记入铁路征信体系", "single_mode": false, "middle_mode": false, "abstract": "人民网济南8月24日电近期在网上“火”了的高铁“霸座”男孙某被处治安罚款200元,并在一定期限内被限制购票乘坐火车。", "tag": "news_society", "label": ["社会"], "behot_time": 1535107285, "source_url": "/group/6593171454974493192/", "source": "人民网", "more_mode": false, "article_genre": "article", "comments_count": 35450, "group_source": 2, "item_id": "6593171454974493192", "has_gallery": false, "group_id": "6593171454974493192", "media_url": "/c/user/50502346173/"}, {"chinese_tag": "财经", "media_avatar_url": "//p1.pstatp.com/large/4d00054b126ceaf920", "is_feed_ad": false, "tag_url": "news_finance", "title": "外交部谈中美经贸磋商：没觉得有必要时不时喊两嗓子，替自己壮胆", "single_mode": false, "middle_mode": false, "abstract": "问：我们注意到，美国政府高层近日在谈到中美经贸磋商时常常要强调，美国的经济状况很好，中国的情况不好。", "tag": "news_finance", "label": ["时政外交", "财经"], "behot_time": 1535107285, "source_url": "/group/6593194458169213448/", "source": "环球网", "more_mode": false, "article_genre": "article", "comments_count": 128, "group_source": 2, "item_id": "6593194458169213448", "has_gallery": false, "group_id": "6593194458169213448", "media_url": "/c/user/5954781019/"}, {"single_mode": true, "abstract": "京东物流西南区域分公司快运部城配运输管理部机构负责人任兵、京东物流华中区域分公司大件物流部安装运营部机构负责人黄川等多人已被公安机关刑事拘留。", "middle_mode": true, "more_mode": true, "tag": "news_finance", "label": ["电子商务", "京东", "刑法", "法律", "反腐倡廉"], "comments_count": 385, "tag_url": "news_finance", "title": "京东发布反腐败公告：多位员工被公安机关刑事拘留", "chinese_tag": "财经", "source": "雷帝触网", "group_source": 2, "has_gallery": false, "media_url": "/c/user/6493820122/", "media_avatar_url": "//p6.pstatp.com/large/2c6800053c0c1b76bdfb", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/153508311899482775b997b"}, {"url": "//p99.pstatp.com/list/pgc-image/153508312799647d3847a69"}, {"url": "//p99.pstatp.com/list/pgc-image/1535083138646835745cbd7"}], "source_url": "/group/6593132015963341315/", "article_genre": "article", "item_id": "6593132015963341315", "is_feed_ad": false, "behot_time": 1535107284, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153508311899482775b997b", "group_id": "6593132015963341315", "middle_image": "http://p99.pstatp.com/list/pgc-image/153508311899482775b997b"}, {"single_mode": true, "abstract": "8月24日，网络上流传的一组图片显示，在上海江南造船厂制造的我国国产万吨055型驱逐舰首舰已经开始试航。", "middle_mode": true, "more_mode": true, "tag": "news_military", "label": ["海军", "驱逐舰", "武器", "江南造船厂", "中国军情"], "comments_count": 2570, "tag_url": "news_military", "title": "055型驱逐舰首舰今日首次试航", "chinese_tag": "军事", "source": "观察者网", "group_source": 2, "has_gallery": false, "media_url": "/c/user/2928392215/", "media_avatar_url": "//p3.pstatp.com/large/d2a0015e51ed3a11c15", "image_list": [{"url": "//p9.pstatp.com/list/pgc-image/15350803549188193355432"}, {"url": "//p99.pstatp.com/list/pgc-image/1535080355549a3b71e372e"}, {"url": "//p99.pstatp.com/list/pgc-image/1535080355238e02c474b41"}], "source_url": "/group/6593119924883816973/", "article_genre": "article", "item_id": "6593119924883816973", "is_feed_ad": false, "behot_time": 1535107284, "image_url": "//p9.pstatp.com/list/190x124/pgc-image/15350803549188193355432", "group_id": "6593119924883816973", "middle_image": "http://p9.pstatp.com/list/pgc-image/15350803549188193355432"}, {"single_mode": true, "abstract": "近日，有网友在网上发布了几张王祖贤昔日旧照。其中有一张就是王祖贤和她爸爸妈妈，以及金牌经纪人陈自强的四人合影，她站在中间搂着爸爸和经纪人的胳膊，备受宠爱。", "middle_mode": true, "more_mode": true, "tag": "news_entertainment", "label": ["王祖贤", "陈自强", "篮球", "加拿大", "张曼玉"], "comments_count": 43, "tag_url": "news_entertainment", "title": "王祖贤全家照曝光，父母都是大高个，光看穿着就知道是有钱人家！", "chinese_tag": "娱乐", "source": "猫眼娱乐", "group_source": 2, "has_gallery": false, "media_url": "/c/user/64781639962/", "media_avatar_url": "//p3.pstatp.com/large/2c6b001dd55cf954a3f6", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/153507480604336e5031b95"}, {"url": "//p99.pstatp.com/list/pgc-image/1535074806030f559e9695a"}, {"url": "//p3.pstatp.com/list/pgc-image/1535074806129dbbd6d84d2"}], "source_url": "/group/6593099140371055112/", "article_genre": "article", "item_id": "6593099140371055112", "is_feed_ad": false, "behot_time": 1535107283, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153507480604336e5031b95", "group_id": "6593099140371055112", "middle_image": "http://p99.pstatp.com/list/pgc-image/153507480604336e5031b95"}], "next": {"max_behot_time": 1535107283}}
当再次将cookie添加发送请求可以刷新数据
sinature = "ZuU44QAAPZ4TmjxZv6zL.WblOP"
cookiestr = 'UM_distinctid=1656b863a92f8-0e645b22313a8b-71272415-75300-1656b863a93f3; tt_webid=6593235575644423694; __tasessionId=lkyrlacv11535107283377; csrftoken=c50e03d22a62281c7ce646395bf32f17; uuid="w:f37f35cde0e24093943a530d5557d608"; CNZZDATA1259612802=222376684-1535103528-%7C1535103528'
header1 = {
'Host': 'www.toutiao.com',
'User-Agent': '"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"',
'Referer': 'https://www.toutiao.com/ch/news_hot/',
"Cookie": cookiestr
}
url = 'https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature={}'.format(
sinature)
print(url)
html = requests.get(url, headers=header1, proxies=func_proxy(), verify=False)
print(html.content.decode('unicode_escape'))
刷新的数据为
{'http': 'http://60.179.248.118:29128'}
https://www.toutiao.com/api/pc/feed/?category=news_hot&utm_source=toutiao&widen=1&max_behot_time=0&_signature=ZuU44QAAPZ4TmjxZv6zL.WblOP
C:\Anaconda\lib\site-packages\requests\packages\urllib3\connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
{"has_more": false, "message": "success", "data": [{"single_mode": true, "abstract": "就在刚才，一直对华很不友好的澳大利亚总理特恩布尔终于下台了，可接替他的人却同样不令人放心…昨天，耿直哥曾撰文给大家讲述了澳大利亚政坛这两天发生了一场大地震：该国的公司的股权。", "middle_mode": true, "more_mode": false, "tag": "news_world", "label": ["中澳关系", "政治", "华为", "时政外交", "国际"], "comments_count": 675, "tag_url": "news_world", "title": "澳洲总理突然下台，新上台的昨天才捅了我们一刀！", "chinese_tag": "国际", "source": "环球时报", "group_source": 2, "has_gallery": false, "media_url": "/c/user/5787290902/", "media_avatar_url": "//p2.pstatp.com/large/9935/2486304457", "source_url": "/group/6593151862864609799/", "article_genre": "article", "item_id": "6593151862864609799", "is_feed_ad": false, "behot_time": 1535107391, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/15350876575153d77867b24", "group_id": "6593151862864609799", "middle_image": "http://p99.pstatp.com/list/pgc-image/15350876575153d77867b24"}, {"single_mode": true, "abstract": "日前，外交部前发言人章启月出任中国驻希腊大使。得知将担任新闻发言人，章启月觉得自己并不太适合做这项工作，丈夫刘结一鼓励她说：“你还是试一试，应该去迎接点挑战嘛”。", "middle_mode": true, "more_mode": false, "tag": "news_politics", "label": ["章启月", "刘结一", "外交政策", "印度尼西亚", "时政"], "comments_count": 35, "tag_url": "search/?keyword=%E6%97%B6%E6%94%BF", "title": "外交部前发言人章启月再任大使", "chinese_tag": "时政", "source": "长安街知事", "group_source": 2, "has_gallery": false, "media_url": "/c/user/4327876576/", "media_avatar_url": "//p3.pstatp.com/large/3538/9145332", "source_url": "/group/6593111799187374596/", "article_genre": "article", "item_id": "6593111799187374596", "is_feed_ad": false, "behot_time": 1535107376, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153507834923010ad6116e1", "group_id": "6593111799187374596", "middle_image": "http://p99.pstatp.com/list/pgc-image/153507834923010ad6116e1"}, {"single_mode": true, "abstract": "作为中国新“四大发明”之一的共享单车，不仅改变了我们中国每个人的出行方式，它还走向国外，给外国带去了互联网+自行车交通的新方式。", "middle_mode": true, "more_mode": true, "tag": "news_tech", "label": ["自行车", "英国", "交通", "经济", "科技"], "comments_count": 70, "tag_url": "news_tech", "title": "大量共享单车在英国遭偷盗破坏 摩拜公司愤怒发文警告", "chinese_tag": "科技", "source": "全球每日趣闻", "group_source": 2, "has_gallery": false, "media_url": "/c/user/100722784942/", "media_avatar_url": "//p7.pstatp.com/large/a14900002c28a48e0dcc", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1535008838168ab3417847e"}, {"url": "//p99.pstatp.com/list/pgc-image/1535010175592846a23e90b"}, {"url": "//p3.pstatp.com/list/pgc-image/153501044157296ceeb2798"}], "source_url": "/group/6592825874624545287/", "article_genre": "article", "item_id": "6592825874624545287", "is_feed_ad": false, "behot_time": 1535107361, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/1535008838168ab3417847e", "group_id": "6592825874624545287", "middle_image": "http://p3.pstatp.com/list/pgc-image/1535008838168ab3417847e"}, {"single_mode": true, "abstract": "她忍痛送别的爱人，正是曾写过《血染的风采》《黄土高坡》等经典歌曲的著名作曲家苏越。苏越和安雯也许对现在的“90后”“00后”来说，苏越的名字和他的那些歌曲，过于陌生。", "middle_mode": true, "more_mode": false, "tag": "news_entertainment", "label": ["流行音乐", "音乐", "歌手", "黄格选", "日本"], "comments_count": 728, "tag_url": "news_entertainment", "title": "从内地乐坛教父到诈骗犯，为了面子，他埋葬了自己的人生与爱情", "chinese_tag": "娱乐", "source": "环球人物杂志", "group_source": 2, "has_gallery": false, "media_url": "/c/user/3994535116/", "media_avatar_url": "//p2.pstatp.com/large/2266/1953108337", "source_url": "/group/6592884788338622990/", "article_genre": "article", "item_id": "6592884788338622990", "is_feed_ad": false, "behot_time": 1535107346, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/153502535842658f7bb37af", "group_id": "6592884788338622990", "middle_image": "http://p99.pstatp.com/list/pgc-image/153502535842658f7bb37af"}, {"single_mode": true, "abstract": "人民网北京8月24日电日前,福布斯公布2018年全球收入最高的男演员榜。作为演员、导演、制片和商人多重身份的乔治·克鲁尼以2.39亿美元登顶。", "middle_mode": true, "more_mode": true, "tag": "news_entertainment", "label": ["成龙", "海姆斯沃斯", "乔治·克鲁尼", "小罗伯特·唐尼", "威尔·史密斯"], "comments_count": 25, "tag_url": "news_entertainment", "title": "全球收入最高男演员出炉：成龙第五系唯一上榜中国演员", "chinese_tag": "娱乐", "source": "人民网", "group_source": 2, "has_gallery": false, "media_url": "/c/user/50502346173/", "media_avatar_url": "//p3.pstatp.com/large/ca400072481685ad43b", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/1535092968677eb1e2608f3"}, {"url": "//p99.pstatp.com/list/pgc-image/1535092969914fd64f5615a"}, {"url": "//p99.pstatp.com/list/pgc-image/1535092969609860dcf09ed"}], "source_url": "/group/6593174112720388616/", "article_genre": "article", "item_id": "6593174112720388616", "is_feed_ad": false, "behot_time": 1535107331, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/1535092968677eb1e2608f3", "group_id": "6593174112720388616", "middle_image": "http://p3.pstatp.com/list/pgc-image/1535092968677eb1e2608f3"}, {"single_mode": true, "abstract": "说到“漂”，总避免不了“房租”这个话题。有报告显示，目前中国租赁人口已接近2亿人。根据中国房地产业协会的数据，北京、深圳、上海、杭州、广州这5个城市房租水平居前，其中北京为92.33元/月/㎡。", "middle_mode": true, "more_mode": true, "tag": "news_house", "label": ["租房", "经济", "盘点", "房产"], "comments_count": 36, "tag_url": "search/?keyword=%E6%88%BF%E4%BA%A7", "title": "各大城市房租排行榜出炉！你一个月要交多少钱？", "chinese_tag": "房产", "source": "南方周末", "group_source": 2, "has_gallery": false, "media_url": "/c/user/50371413220/", "media_avatar_url": "//p3.pstatp.com/large/ca2000539631fc1d01d", "image_list": [{"url": "//p99.pstatp.com/list/pgc-image/15350346510653c6d876cd7"}, {"url": "//p3.pstatp.com/list/pgc-image/1535034651276d0f13bda31"}, {"url": "//p3.pstatp.com/list/pgc-image/1535034650096cd059c4f9f"}], "source_url": "/group/6592923629598540301/", "article_genre": "article", "item_id": "6592923629598540301", "is_feed_ad": false, "behot_time": 1535107316, "image_url": "//p99.pstatp.com/list/190x124/pgc-image/15350346510653c6d876cd7", "group_id": "6592923629598540301", "middle_image": "http://p99.pstatp.com/list/pgc-image/15350346510653c6d876cd7"}, {"single_mode": true, "abstract": "虽然是参加婚礼，但是他与奥地利外长卡琳⋅克奈斯尔和奥地利总理塞巴斯蒂安⋅库尔茨还成功进行了“商谈”。", "middle_mode": true, "more_mode": true, "tag": "news_world", "label": ["普京", "婚礼", "外交政策", "政治", "柔道"], "comments_count": 6, "tag_url": "news_world", "title": "因为对普京做出这个动作，奥地利外长被要求辞职", "chinese_tag": "国际", "source": "环球网", "group_source": 2, "has_gallery": false, "media_url": "/c/user/5954781019/", "media_avatar_url": "//p1.pstatp.com/large/4d00054b126ceaf920", "image_list": [{"url": "//p3.pstatp.com/list/pgc-image/153507129660162fbe735e8"}, {"url": "//p99.pstatp.com/list/pgc-image/1535071296031f3dabca397"}, {"url": "//p9.pstatp.com/list/pgc-image/1535071296345e128c495f8"}], "source_url": "/group/6593081018062733837/", "article_genre": "article", "item_id": "6593081018062733837", "is_feed_ad": false, "behot_time": 1535107301, "image_url": "//p3.pstatp.com/list/190x124/pgc-image/153507129660162fbe735e8", "group_id": "6593081018062733837", "middle_image": "http://p3.pstatp.com/list/pgc-image/153507129660162fbe735e8"}], "next": {"max_behot_time": 1535107301}}
至于cookie过期时间自己测试吧！有点凑篇幅的嫌疑