改进的内容分析排序算法在搜索引擎中的研究与应用

发布时间：2018-03-07 18:56

本文选题：搜索引擎　切入点：排序算法　出处：《大连交通大学》2013年硕士论文　论文类型：学位论文

【摘要】：随着因特网的不断发展和日益普及,网上的信息量呈爆炸趋势增长,如何在浩瀚如海的信息中迅速地定位到所需的信息,已成为人们不得不考虑的问题,于是信息检索技术应运而生并成为了当今最热门的技术之一。搜索引擎正是基于一定的策略在Web上搜索和发现信息,它的出现虽然只有短暂的十几年时间,但在Web上已经有了确定不移的地位。本文的研究工作主要有以下几方面： 1.就搜索引擎及其发展历史作了简要的叙述,之后则就搜索引擎中重要的几类检索模型进行了相应的介绍,同时信息检索的评价指标,如：查准率、查全率等也在文中有所体现。 2.在Salton发明的向量空间模型的基础上,本文对其经典算法TF-IDF公式进行了改进,提出了词频(TF)和逆文档频率(IDF)的乘积累加求和法和TF-IDF的归一化法,并与词频和逆文档频率的计算方法进行了比较；在基于概率模型方面,本文实现了文档和查询词权重的基础上的BM25排序算法；在语言模型方面,采用集合了最大似然模型和集合模型优势的Jelinek-Mercer平滑方法和绝对折扣平滑方法,之后本文对贝叶斯分析方法和狄利克雷参数分布结合的原狄利克雷(Dirichlet)平滑方法进行了改进,并对其算法作了部分深化。 3.本文深入地研究了Lucene的排序机制,并且将上述几种基于内容的排序算法融入到了Lucene的排序机制中,进一步完善了Lucene对于查询结果的排序功能。最后,采用TREC会议中中英文双语语料进行了实验,通过查询测试和结果评价,作者发现运用不同排序算法对结果文档进行排序时,Lucene返回的结果文档的相关性是不一样的,而改进型的狄利克雷(Dirichlet)平滑方法具有最好的排序效果；而相对于采用词频TF、逆文档频率IDF等的向量空间模型排序效果而言,TF-IDF归一化法的排序效果较好。
[Abstract]:With the continuous development and popularization of the Internet, the amount of information on the Internet is increasing. How to quickly locate the information needed in the vast sea of information has become a problem that people have to consider. Therefore, information retrieval technology emerged as the times require and became one of the most popular technologies. Search engine search engine search and discover information on Web based on certain strategy, although it appeared for only a short period of more than ten years. But in the Web has a certain position. The research work of this paper mainly includes the following aspects:. 1. A brief description of search engine and its development history is given, and then several important retrieval models in search engine are introduced. At the same time, the evaluation index of information retrieval, such as precision rate, is introduced. Recall rate is also reflected in the text. 2. On the basis of the vector space model invented by Salton, the classical algorithm TF-IDF formula is improved in this paper. The product accumulation summation method and the normalization method of TF-IDF are proposed. And compared with word frequency and inverse document frequency calculation method; in the aspect of probability model, this paper realized the BM25 sorting algorithm based on the weight of documents and query words; in the language model, The Jelinek-Mercer smoothing method and absolute discount smoothing method with the advantages of maximum likelihood model and set model are adopted. After that, the original Dirichlet smoothing method which combines Bayesian analysis method with Delikley parameter distribution is improved. The algorithm is partly deepened. 3. This paper deeply studies the sorting mechanism of Lucene, and integrates the above content-based sorting algorithms into the sorting mechanism of Lucene, which further improves the sorting function of Lucene for query results. The experiment is carried out by using bilingual Chinese and English corpus in TREC conference. Through query test and result evaluation, the author finds that the correlation of the result document returned by Lucene is different when different sorting algorithms are used to sort the result document. The improved Dirichlet smoothing method has the best sorting effect, while the TF-IDF normalization method is better than the vector space model with word frequency TFF, inverse document frequency IDF and so on.
【学位授予单位】：大连交通大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】