python爬虫框架论文开题报告范文_仿今日头条新闻爬虫设计毕业论文+设计源码+开题报告...

摘要

随着社会日新月异和互联网进入大数据时代，自媒体得到了迅猛的发展，人们获取新闻资讯的方式越来越多，接触和使用新闻信息的方式正在逐渐改变，受众从被动接受信息到按需主动搜索信息，而新闻的种类繁多杂乱，各类人需要的新闻也不尽相同，为此当前以今日头条为代表的各大媒体通过数据分析，数据挖掘等方式，在内容生产上做到精确定位用户需求，着力打造消费痛点等特点。因此，开发一款新闻定制推送系统有助于为人们提供更为优质的新闻信息推送服务，人们完全可以根据自己的喜好定制报纸和杂志，随着人们使用时间的增长，可以做到“机器越来越懂你”。

本文所描述的仿今日头条个性新闻推荐系统其数据来源主要是新浪新闻，首先利用爬虫工具将新闻数据获取到本地数据库中，然后将新闻信息以及新闻分析的结果将在前台网页中进行可视化输出。本文主要进行了以下几个方面的工作：

1)利用Python语言结合HTTPClient开源工具编写了一个针对新浪新闻的可扩展的网络爬虫，该爬虫程序能够按照广度优先的爬行策略对新闻数据(包括图片信息)进行全面的定向抓取以及周期性的增量抓取；

2)采用HTMLParser对获取到的新闻信息进行元数据抽取，将新闻的编号、标题、内容、发布方、发布时间等元数据以及新闻图片等元数据解析出来并存入MySQL数据库中；

通过使用该新闻爬取系统，用户能够实时的更新新闻信息，及时、全面、准确地掌握新闻热点动态，提高对于重大突发事件的处理能力，对于更及时，全面的了解各地的实时信息具有重要意义。

关键词：新闻热点，网络爬虫，元数据抽取，可视化

ABSTRACT

With the rapid development of society and the entry of the Internet into big data's time, since the media has been developing rapidly, people have more and more ways of obtaining news information, and the way of contact and use of news information is gradually changing.From passively receiving information to actively searching for information on demand, the audience has a wide variety of news items and different types of people need different kinds of news. For this reason, the major media, represented by today's headlines, use data analysis, data mining, and so on.In content production to achieve accurate positioning of user needs, efforts to create consumer pain points and other characteristics.Therefore, the development of a customized news push system will help to provide people with better quality News push service, people can customize newspapers and magazines according to their own preferences, as people use time, can achieve "the machine knows you more and more".

The profile news recommendation system described in this paper mainly comes from Sina News. Firstly, it uses crawler tools to get news data into local databases.Then the news information and the results of news analysis will be visualized in the front page.The main work of this paper is as follows:

1) using Python language and HTTPClient open source tool to write an extensible web crawler for Sina News.The crawler program can crawl news data (including picture information) according to the breadth first crawling strategy.

2) HTMLParser is used to extract the news information, and the metadata such as the number, the title, the content, the publisher, the time of release and the news picture are analyzed and stored in the MySQL database.

By using this news crawling system, users can update news information in real time, timely and comprehensively, accurately grasp the news hotspots, improve the handling capacity for major emergencies, and have a great significance for a timely and comprehensive understanding of real-time information around the country.

Key words:News hotspots, web crawlers, metadata extraction, visualization

目录