基于本体概念相似度的主题爬虫中网页排序模型研究
发布时间：2018-10-04 19:47
【摘要】：相比通用搜索引擎，专注于某一具体领域的主题搜索引擎可以带来更高精度的信息采集，为用户带来更好信息检索服务。主题爬虫作为主题搜索引擎的核心模块，提高检索信息的领域相关度就显得尤为重要。 但是由于网络资源规模巨大且呈高度动态的增长，采集结果仍然会存在大量不相关的网页信息，从而导致采集效率下降。针对这种问题，本文通过研究主题爬虫设计中的相关性分析技术，主要是网页排序算法的研究，分析总结目前网页排序算法的优缺点，并结合盐湖领域特点，利用本体在表达语义方面的优势，提出一种新的基于本体概念相似度的网页排序算法，以此提高主题相关性计算准确度。 该方法首先选择出合适网页作为初始领子种子集合，然后通过构建盐湖领域本体获取本体概念集，并对概念集分类且给予权重，，利用概念相似度计算方法计算网页内所有概念与本体概念集中概念的相似度，根据综合得分对网页进行排序，将得分高的网页存放到主题爬虫中，为将来的网页采集做准备。最后通过实验证明，该算法不仅大大减少了不相关的结果，提高了采集网页的主题相关度，而且也提高了检索的准确率。
[Abstract]:Compared with the general search engine, the subject search engine focused on a specific field can bring higher precision information collection and better information retrieval service for users. As the core module of subject search engine, it is very important to improve the relevance of subject crawler. However, due to the large scale and highly dynamic growth of network resources, there will still be a large number of irrelevant web page information, which leads to a decline in the efficiency of collection. In order to solve this problem, this paper analyzes and summarizes the advantages and disadvantages of the current web page sorting algorithm, and combines the characteristics of the salt lake field by studying the correlation analysis technology in the subject crawler design, mainly the research of the web page sorting algorithm. Taking advantage of ontology in expressing semantics, a new web page sorting algorithm based on ontology concept similarity is proposed to improve the accuracy of topic correlation calculation. The method first selects the appropriate web page as the initial collar seed set, then obtains the ontology concept set by constructing the salt lake domain ontology, and classifies the concept set and gives the weight to the concept set. The concept similarity calculation method is used to calculate the similarity between all the concepts in the web page and the concepts in the ontology concept set. According to the comprehensive score, the web pages with high scores are sorted, and the high score pages are stored in the subject crawler to prepare for the future collection of web pages. Finally, the experimental results show that the algorithm not only reduces the irrelevant results, but also improves the retrieval accuracy.
【学位授予单位】：北京信息科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.1
本文编号：2251648
[Abstract]:Compared with the general search engine, the subject search engine focused on a specific field can bring higher precision information collection and better information retrieval service for users. As the core module of subject search engine, it is very important to improve the relevance of subject crawler. However, due to the large scale and highly dynamic growth of network resources, there will still be a large number of irrelevant web page information, which leads to a decline in the efficiency of collection. In order to solve this problem, this paper analyzes and summarizes the advantages and disadvantages of the current web page sorting algorithm, and combines the characteristics of the salt lake field by studying the correlation analysis technology in the subject crawler design, mainly the research of the web page sorting algorithm. Taking advantage of ontology in expressing semantics, a new web page sorting algorithm based on ontology concept similarity is proposed to improve the accuracy of topic correlation calculation. The method first selects the appropriate web page as the initial collar seed set, then obtains the ontology concept set by constructing the salt lake domain ontology, and classifies the concept set and gives the weight to the concept set. The concept similarity calculation method is used to calculate the similarity between all the concepts in the web page and the concepts in the ontology concept set. According to the comprehensive score, the web pages with high scores are sorted, and the high score pages are stored in the subject crawler to prepare for the future collection of web pages. Finally, the experimental results show that the algorithm not only reduces the irrelevant results, but also improves the retrieval accuracy.
【学位授予单位】：北京信息科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.1
【参考文献】
相关期刊论文 前10条
1 张文秀;朱庆华;;领域本体的构建方法研究[J];图书与情报;2011年01期
2 朱礼军,陶兰,刘慧;领域本体中的概念相似度计算[J];华南理工大学学报(自然科学版);2004年S1期
3 马培华;;科学开发我国的盐湖资源[J];化学进展;2009年11期
4 刘玉婷;马志明;;网页排序中的随机模型及算法[J];中国科学:数学;2011年12期
5 孙德才;孙星明;张伟;刘玉玲;;基于匹配区域特征的相似字符串匹配过滤算法[J];计算机研究与发展;2010年04期
6 李荣;杨冬;刘磊;;基于本体的概念相似度计算方法研究[J];计算机研究与发展;2011年S3期
7 蔡国民;王雅琳;;搜索引擎的相关排序算法分析与优化[J];吉首大学学报(自然科学版);2006年05期
8 李学勇,欧阳柳波,李国徽,钟敏娟;网络蜘蛛搜索策略比较研究[J];计算机工程与应用;2004年04期
9 陈杰;蒋祖华;;领域本体的概念相似度计算[J];计算机工程与应用;2006年33期
10 刘文剑;郭宁;金天国;;制造资源本体的相似度计算模型[J];计算机集成制造系统;2010年11期
相关博士学位论文 前1条
1 蔡盈芳;基于本体的航空产品知识库构建研究[D];北京交通大学;2011年
本文编号：2251648
本文链接：https://www.wllwen.com/kejilunwen/sousuoyinqinglunwen/2251648.html
