Web垂直信息检索技术及算法的研究与实现

发布时间：2018-06-20 07:33

本文选题：垂直信息检索 + 中文分词　；参考：《广东技术师范学院》2017年硕士论文

【摘要】：伴随着计算机硬件的不断发展,互联网也随着硬件方面的提升而得到了空前的发展,特别是在这个数据爆发的时代,大量信息覆盖着整个社会,随之就出现了大数据以及相关计算机的新技术。在大数据时代里,信息检索系统能够准确无误地找到人们自身所需的数据,其中信息检索系统的定义是用户根据一定的检索关键字或者策略,借助相关的爬虫技术,将互联网上的相关的数据爬取下来,并通过中文分词、网页去重、排序优化等相关技术对爬取下来的数据信息进行处理,最后呈现给用户所需求的信息。其中,中国的百度、360以及国外的Google、Yahoo等最具有代表意义,尽管它们专注于检索领域,但都各有各的特点,成为人们生活中必不可少的工具。由于Google、Yahoo、百度、360等检索的范围大,涉及到信息量广,针对特定的领域检索可能还存在一定的困难。为了克服针对特定的领域的专业检索,人们便引入了垂直信息检索系统的概念。垂直信息检索系统的定义是基于某一专业领域所开发的信息检索系统,如文档垂直信息检索系统、旅游垂直信息检索系统以及购物垂直信息检索系统等等。本项目,主要研究新闻垂直信息检索系统,并在原有技术的基础上进行了相关的优化操作:首先,在Heritrix的原型上进行二次开发,使得优化后的Heritrix爬虫技术爬取网页资源效率更高;然后在获得网页资源的基础上,通过HTMLParser技术将网页格式转化成TXT文本格式,并以IK Analyzer分词技术为基础进行了优化,对TXT文本内容进行分词以及过滤TXT文本内容中的脏数据;接着改进TF-IDF加权算法,有效去除网页中重复的部分;最后,以Struts+Spring+Hibernate为架构,以MySQL为存储数据库,借助PageRank算法改进Lucene的排序算法,创建以及查询索引,实现新闻垂直信息检索系统。
[Abstract]:With the continuous development of computer hardware, the Internet has also been unprecedented development with the improvement of hardware, especially in this era of data explosion, a large amount of information covers the whole society. Then came the new technology of big data and related computers. In the era of big data, the information retrieval system can accurately find the data that people need, and the definition of information retrieval system is that the user can use the relevant crawler technology according to a certain search keyword or strategy. The related data on the Internet is crawled down and processed by Chinese word segmentation, web page removal, ranking optimization and so on. Finally, the information required by users is presented. Among them, China's Baidu 360 and foreign Google Yahoo have the most representative significance. Although they are focused on the search field, they all have their own characteristics and become indispensable tools in people's lives. Because of the wide range of search such as Google Yahoo, Baidu and 360, which involves a wide amount of information, there may still be some difficulties in searching for specific fields. In order to overcome the specialized retrieval in specific fields, the concept of vertical information retrieval system is introduced. The definition of vertical information retrieval system is based on the information retrieval system developed by a professional field, such as document vertical information retrieval system, tourism vertical information retrieval system and shopping vertical information retrieval system. This project mainly studies the news vertical information retrieval system, and carries on the related optimization operation based on the original technology: first, carries on the secondary development on the Heritrix prototype, The optimized Heritrix crawler technology is more efficient in crawling web resources, and then the web page format is transformed into TXT text format by HTML Parser technology and optimized based on IK Analyzer participle technology. The text content of TXT is segmented and the dirty data in TXT text content is filtered. Then the TF-IDF weighted algorithm is improved to effectively remove the duplicate parts of the web page. Finally, the Struts Spring hibernate is used as the framework and MySQL as the storage database. The PageRank algorithm is used to improve Lucene's sorting algorithm, to create and query indexes, and to realize the news vertical information retrieval system.
【学位授予单位】：广东技术师范学院
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【参考文献】