基于主题和类别的网络新闻采集系统设计与实现

发布时间：2019-01-06 07:01

【摘要】：随着互联网的发展,网络新闻成为人们获取信息的重要来源之一。网络新闻具有传播速度快、影响范围大、社会受众面广等优点,但是也存在一些虚假性、低质量的网络新闻,网络新闻质量的参差不齐降低了用户的阅读体验。另外,网络新闻在某种程度上成为网络舆论的来源和传播途径,因此在海量的网络新闻数据中采集到真实、准确、结构化的网络新闻数据成为网络舆情研究的重点。本文面向主题网络新闻和类别网络新闻,着重解决了网络新闻采集中主题采集、类别采集的问题,并在其基本功能的实现基础上,进一步考虑提高系统的性能。本文引入了主题爬虫和SVM分类器的概念,引入了Xpath和多线程的技术,在以上理论和技术的基础上,设计并实现了一个基于主题和类别的网络新闻采集系统,系统具有采集并存储主题网络新闻和类别网络新闻的功能。在基于主题的网络新闻采集中,本系统通过计算页面相似度的方法形成爬取的优先级队列,然后通过Xpath技术抽取主题网络新闻的标题、网址、发布时间、发布来源、正文等内容,最后将采集到的主题性网络新闻数据存储到系统数据库中。在基于类别的网络新闻采集中,本文引入Libsvm包来实现分类器的训练和构造,然后通过Xpath技术抽取类别新闻的标题、网址、发布时间、发布来源、正文等内容,类别包括社会、娱乐、财经和体育,最后将采集到的类别性网络新闻数据存储到系统数据库中。首先,本文介绍了网络新闻采集的研究背景和意义,着重介绍了国内外对于主题爬虫、分类器的研究工作;其次,本文介绍了网络新闻采集过程中涉及的理论和技术,包括Robots协议、通用网络爬虫、支持向量机、主题爬虫搜索策略、Xpath技术等;然后,本文对系统的需求进行了分析和介绍,对系统的体系结构进行了整体设计,对系统的模块组成进行了详细设计,其中系统的模块包括新闻网站种子注入模块、网页源代码获取模块、网页解析模块、分类模块、主题过滤模块、URL调度模块、URL去重模块、网页信息抽取模块、数据库存储模块;另外,本文在系统整体设计和详细设计的基础上,通过调用ICTCLAS包和Libsvm包,实现了以上设计的诸多模块,进一步地实现了基于主题的网络新闻采集和基于类别的网络新闻采集的功能。最后,本文列举了系统运行所需要的硬件环境和软件环境,对系统的功能和性能分别进行了测试,测试的结果达到系统预期要求,但是还有很多需要改进的地方。本系统采用C#语言在Windows7 32位操作系统环境下对主题采集和类别采集进行了实现。系统的健壮性、高效性、持续性、稳定性等都达到预期要求,能够准确、及时、有效地采集并存储基于主题和基于类别的网络新闻数据。
[Abstract]:With the development of Internet, network news has become one of the important sources for people to obtain information. Network news has the advantages of fast transmission, wide influence, wide social acceptance, but there are some false, low-quality network news, the uneven quality of network news reduces the user's reading experience. In addition, to some extent, network news has become the source of public opinion and the way of dissemination, so collecting real, accurate and structured network news data in the mass of network news data has become the focus of network public opinion research. This paper aims at the topic network news and the category network news, and solves the problem of the topic collection and the category collection in the network news collection emphatically, and on the basis of its basic function realization, further consideration to improve the performance of the system. In this paper, the concepts of topic crawler and SVM classifier are introduced, and Xpath and multithreading techniques are introduced. Based on the above theories and techniques, a network news collection system based on topic and category is designed and implemented. The system has the function of collecting and storing topic network news and category network news. In the network news collection based on topic, this system forms the crawling priority queue by calculating the similarity of the page, then extracts the title, URL, release time, release source of the topic network news by Xpath technology. Finally, the collected thematic network news data is stored in the system database. In the network news collection based on category, this paper introduces Libsvm packet to realize the training and construction of classifier, and then extracts the title, URL, publishing time, publishing source, text and other contents of category news through Xpath technology. Entertainment, finance and sports, and finally the collection of category network news data stored in the system database. First of all, this paper introduces the research background and significance of the network news collection, focusing on the domestic and foreign research on the topic crawler, classifier; Secondly, this paper introduces the theory and technology involved in the process of network news collection, including Robots protocol, general web crawler, support vector machine, topic crawler search strategy, Xpath technology and so on. Then, this paper analyzes and introduces the requirements of the system, designs the architecture of the system as a whole, and designs the module composition of the system in detail. The module of the system includes the seed injection module of the news website. Web source code acquisition module, web page analysis module, classification module, theme filtering module, URL scheduling module, URL de-reduplication module, page information extraction module, database storage module; In addition, on the basis of the overall design and detailed design of the system, by calling the ICTCLAS package and the Libsvm package, this paper realizes many modules of the above design. The functions of subject-based network news collection and category-based network news collection are further realized. Finally, this paper lists the hardware and software environment needed to run the system, and tests the function and performance of the system separately. The results of the test meet the expected requirements of the system, but there are still many areas for improvement. This system uses C # language in Windows7 32-bit operating system environment to realize the subject collection and category acquisition. The robustness, efficiency, persistence and stability of the system can meet the expected requirements, and can accurately, timely and effectively collect and store the network news data based on topic and category.
【学位授予单位】：山东师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【参考文献】