基于垂直搜索引擎的结构化信息处理技术研究

发布时间：2018-06-10 13:23

本文选题：搜索 + 索引　；参考：《浙江理工大学》2013年硕士论文

【摘要】：随着互联网的发展，搜索引擎在不断满足巨大的信息资源量的需求下，却无法兼顾到信息搜索的准确度和及时性，此时垂直搜索引擎为满足用户需求应运而生，本文通过对垂直搜索引擎进行了深入研究，并针对现有模型及其所存在的问题提出一种改进的垂直搜索引擎模型，依据该模块特点对结构化数据的去重和分类算法进行改进，通过两个改进算法在改进的垂直搜索引擎模型中的实验应用，，得出改进的垂直搜索引擎模型进一步提高了垂直搜索引擎的实时性和准确性。新模型设计的主要方案是对现有模型新增加一个数据二次处理的模块，该模块主要对抽取到的非结构化数据和半结构化数据向结构化数据转换。模块的主要研究内容是对网页信息的去重处理和分类处理。因此本文的主要研究内容和创新点分为以下三点：（1）在参考现有的电子商务领域广泛应用的垂直搜索引擎的基础上，提出一种改进的垂直搜索引擎应用模型，结合本文改进的去重算法和分类算法使用查全率和准确率两个指标评估该模型的实用性和可行性。（2）提出一种新的信息处理技术的网页去重算法，并以时间复杂度、空间复杂度、查全率和准确率作为四个指标分析该算法在改进的垂直搜索引擎模型中的可行性和健壮性，以及对信息检索效率的提高。（3）对现有的一种分类算法进行改进，进而使得适合于本文提出的垂直搜索引擎的结构化数据处理计算，该算法的结构包括词条数组和每个词条的文本链表。词条数组指将所有的训练文本分词，经过特征提取后的所有特征项组成的数组，存储在数组中的是特征项（词条）的ID号。词条数组中的每个词条(ti)有一个指针，指向含有ti的所有文本组成的链表。文本链表由两部分组成，文本的ID和ti在文本中的权重。ti的文本链表生成以后，按照ti在文本中的权重递减排序，然后对其进行进一步的优化进而降低原有算法的查找范围。
[Abstract]:With the development of the Internet, the search engine is not able to take into account the accuracy and timeliness of information search in order to meet the needs of users. In this paper, the vertical search engine is deeply studied, and an improved vertical search engine model is put forward in view of the existing model and its existing problems. According to the characteristics of this module, the algorithm of removing and classifying structured data is improved. Through the experimental application of two improved algorithms in the improved vertical search engine model, it is concluded that the improved vertical search engine model can further improve the real-time and accuracy of the vertical search engine. The main scheme of the new model design is to add a new data secondary processing module to the existing model, which mainly converts the extracted unstructured data and semi-structured data to structured data. The main research content of the module is to dereprocess and classify the web page information. Therefore, the main contents and innovations of this paper can be divided into the following three points: firstly, an improved vertical search engine application model is proposed on the basis of reference to the existing vertical search engine which is widely used in the field of electronic commerce. Combining the improved algorithm and classification algorithm to evaluate the practicability and feasibility of the model by using recall and accuracy. (2) A new information processing algorithm for web pages is proposed, and the complexity of time, space and space are used to evaluate the feasibility of the model. Recall rate and accuracy rate are used as four indexes to analyze the feasibility and robustness of the algorithm in the improved vertical search engine model, and to improve the efficiency of information retrieval. The structure of the algorithm consists of an array of terms and a text list of each term. The term array refers to an array of all the trained text participles and all the feature items extracted by the feature, and the ID number of the feature item (entry) is stored in the array. Each entry in the entry array has a pointer to a list of all text containing ti. The text list consists of two parts: the ID of the text and the weight of ti in the text. After the text list is generated, the text list is sorted according to the decreasing weight of ti in the text, and then it is further optimized to reduce the search range of the original algorithm.
【学位授予单位】：浙江理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【参考文献】