大豆主题垂直搜索引擎关键技术的研究与设计

发布时间：2018-07-09 20:15

本文选题：大豆主题 + 垂直搜索引擎　；参考：《东北农业大学》2013年硕士论文

【摘要】：随着互联网技术的快速发展，网络信息资源呈现出爆炸性增长态势，如何快速找到满足用户需求的信息成为越来越重要的问题。目前，搜索引擎已成为互联网最重要的应用之一，传统的通用搜索引擎为所有用户提供统一接口，但随着信息量的持续增长，其已不能满足特定领域用户对信息准确性、实时性和深度等多方面的个性化需求，因而，专门用来查询某一学科领域或主题的搜索引擎即“垂直搜索引擎”应运而生，并得到快速的发展和广泛的应用。本课题来源于星火计划项目，立足于粮食主产区农业现实状况，，针对农业信息化中普遍存在的信息资源共享程度低的问题，尤其是大豆产业信息化建设，为从事大豆生产加工、科研及流通工作的人员提供共享数据资源。本文采用垂直搜索技术对互联网上农业领域中的大豆相关信息进行采集、过滤，为以“中国大豆网”为标志的门户网站构建大豆信息库，同时，设计了面向大豆主题的垂直搜索引擎构架，对其关键技术开展研究，并实现了原型系统。本文的主要研究内容如下：（1）首先，明确本文研究目的和意义，分析垂直搜索引擎以及其在农业领域中应用的研究现状和动态；其次，对通用搜索引擎和垂直搜索引擎的发展、结构、原理以及各自的优劣进行分析、比较，并基于大豆主题，对主题搜索引擎的系统结构进行设计。（2）网页信息采集的核心是网络蜘蛛，其自动地在互联网上按照一定的搜索策略进行搜索爬行，并将搜集的信息存储到本地。主题网络蜘蛛与通用网络蜘蛛最大的区别是，前者是有选择地抓取主题相关的页面，而后者则是“见网页就抓”。本文对主题网络蜘蛛的结构、原理、搜索策略以及主题相关度分析算法进行深入研究和分析，考虑链接锚文本和网页标题对相关度的影响以及链接陷阱问题，对已有的链接分析算法进行改进。（3）索引可以提高检索效率，本文索引能够有效提高管理与审核模块加载数据的速度。索引对象是经过中文分词处理的网页文档，中文分词就是将连续的字序列拆分成词序列的过程。本文对已有的分词算法和倒排索引技术以及开源Lucene索引框架的索引过程和搜索过程进行研究，由于Lucene自带的中文分词不够精确，因此，采用基于IKAnalyzer分词的Lucene索引框架。（4）基于上述研究，按照软件工程学的理论对面向大豆主题的垂直搜索引擎原型系统进行实现，主要是对该系统中的网页信息采集、索引和管理与审核模块进行实现，最终为大豆门户网站提供大豆相关数据。综上所述，本文以国内主要大豆网站为初始抓取的目标网站（如中国农产品交易网、中国粮油信息网、黑龙江省农业信息网、天下粮仓等），基于Java技术对面向大豆主题的垂直搜索引擎原型系统进行实现，为大豆门户网站提供数据支撑，同时，为面向大豆主题信息的查询提供了理论基础，本文的研究也可作为其他农业主题搜索引擎的参考。
[Abstract]:With the rapid development of Internet technology, network information resources show an explosive growth trend. How to quickly find information to meet the needs of users has become a more and more important problem. At present, the search engine has become one of the most important applications of the Internet. The traditional general search engine provides a unified interface for all users, but with the letter The continuous growth of interest rates has been unable to meet the personalized needs of users in specific fields such as information accuracy, real-time and depth. Therefore, the search engine called "vertical search engine", which is specially used to query a subject area or subject, has emerged as the times require, and has been rapidly developed and widely used.
This project is based on the project of star fire plan, based on the agricultural reality of the main grain producing area, aiming at the problem of low sharing of information resources in the agricultural informatization, especially in the construction of soybean industry, providing the sharing data resources for the people engaged in soybean production and processing, scientific research and circulation work. Cable technology collects and filters soybean related information in the field of agriculture on the Internet and filters, constructs soybean information base for the portal website marked by "China soybean network". At the same time, it designs a vertical search engine framework for soybean theme, studies its key technologies and implements a prototype system. The main contents of this paper are the main contents of this paper. As follows:
(1) first, make clear the purpose and significance of this study, analyze the research status and dynamics of vertical search engine and its application in agriculture; secondly, analyze the development, structure, principle and the advantages and disadvantages of the general search engine and vertical search engine, and compare the system structure of the subject search engine based on the subject of soybean. Design.
(2) the core of the web information collection is the web spider, which automatically searches and crawls according to a certain search strategy on the Internet, and stores the information to the local. The biggest difference between the theme network spider and the common web spider is that the former is the choice to grab the main questions related pages, and the latter is "see the web page to catch". This paper makes an in-depth study and analysis of the structure, principle, search strategy and topic correlation analysis algorithm of the theme network spider, considering the influence of the link anchor text and the page title on the correlation degree and the link trap problem, and improves the existing link analysis algorithm.
(3) index can improve the efficiency of retrieval. The index can effectively improve the speed of loading data in the management and audit modules. The index object is a web page document processed by Chinese word segmentation. The Chinese word segmentation is the process of splitting the serial word sequence into the word sequence. In this paper, the existing segmentation algorithm and inverted index technique and open source Lucene are used in this paper. The indexing process and search process of the index frame are studied. Because the Chinese word segmentation is not accurate enough for Lucene, the Lucene indexing framework based on IKAnalyzer segmentation is adopted.
(4) based on the above research, the prototype system of vertical search engine for soybean subject is realized in accordance with the theory of software engineering. It is mainly to implement the web information collection, index and management and audit module in the system, and finally provide soybean related data for soybean portal.
To sum up, this article takes the main soybean website in China as the initial target website (such as China's agricultural product trading network, China Grain and oil information network, Heilongjiang agricultural information network, the world grain barn, etc.), based on Java technology to achieve the soybean theme vertical search engine prototype system, providing data support for soybean portal sites. It provides a theoretical basis for the inquiry of soybean theme information. The research in this paper can also serve as a reference for other agricultural topic search engines.
【学位授予单位】：东北农业大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【参考文献】