基于改进PageRank算法的医学垂直搜索引擎的研究与实现

发布时间：2018-09-18 09:07

【摘要】：近年来,互联网逐渐成为人们获得医学健康资讯的一个重要平台,其中搜索引擎在查询医学信息的过程中给人们提供了极大的便利。但是现有的医学搜索引擎在主题相似性判断和网页排序算法还是存在不足。因此,本文从主题相似性判断和PageRank算法方面做出改进,构建了一个面向医学领域的垂直搜索引擎。主要研究内容与成果如下:(1)选择初始URL,构建医学领域主题词库,研究空间向量模型。在抓取网页后,分别从超链接、元信息、词库三个方面进行主题相关性判别,高效地去除了与主题不相关的页面,使得搜索引擎的效率大幅度提升。(2)本文对PageRank算法和HITS算法进行了研究与分析。由于PageRank算法的计算效率更高、计算数据量更大,所以,本文采用PageRank算法作为网页排序算法。并针对PageRank算法偏向旧网页、平均分配权值、主题漂移等不足之处,引入时间反馈因子,提高“新”网页的评分;引入权威性反馈因子改进出链网页的权值;引入了主题相关度因子抑制“主题漂移”。(3)基于以上两点研究成果,本文设计了一个面向医学领域的垂直搜索引擎。在设计搜索引擎时,主要分为爬虫模块、检索服务模块。另外,基于Nutch的高扩展性及插件机制,本文加入了IKAnalyzer中文分词器改善搜索引擎处理中文信息的能力。(4)最后对项目进行部署并验证。实验表明,该垂直搜索引擎实现了按词切分,并且分词的准确率达到了90%;对网页经过主题相似性判断后,爬虫效率提高了8%;通过对PageRank算法的改进,垂直搜索引擎的查准率有了明显提高,并且返回给用户的前10条结果的查准率在0.7以上。
[Abstract]:In recent years, the Internet has gradually become an important platform for people to obtain medical health information, in which search engine provides great convenience in the process of searching medical information. However, the existing medical search engines still have some shortcomings in topic similarity judgment and web page sorting algorithms. Therefore, a vertical search engine oriented to medical field is constructed by improving the topic similarity judgment and PageRank algorithm. The main research contents and results are as follows: (1) choose the initial URL, to construct the subject thesaurus of medical field and study the spatial vector model. After crawling the web page, we distinguish the theme correlation from hyperlink, meta-information and thesaurus respectively, and effectively remove the page which is not related to the topic. The efficiency of search engine is greatly improved. (2) the PageRank algorithm and HITS algorithm are studied and analyzed in this paper. Because the PageRank algorithm is more efficient and the amount of computing data is larger, the PageRank algorithm is used as the sorting algorithm for web pages in this paper. Aiming at the shortcomings of PageRank algorithm, such as biased old web pages, average weight distribution, topic drift and so on, time feedback factor is introduced to improve the score of "new" web pages, and authoritative feedback factor is introduced to improve the weights of web pages. The theme correlation factor is introduced to suppress the "topic drift". (3) based on the above two research results, this paper designs a vertical search engine oriented to the medical field. When designing search engine, it is mainly divided into crawler module and retrieval service module. In addition, based on the high extensibility and plug-in mechanism of Nutch, this paper adds IKAnalyzer Chinese word Segmentation to improve the ability of search engine to process Chinese information. (4) finally, the project is deployed and verified. Experiments show that the vertical search engine can segment words by word, and the accuracy of word segmentation reaches 900.The crawler efficiency is improved by 8 percent after the page is judged by the similarity of topic, and the PageRank algorithm is improved. The accuracy of vertical search engine has improved obviously, and the precision rate of the top 10 results returned to users is more than 0.7.
【学位授予单位】：长安大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【参考文献】