基于词典与统计结合的中文分词方法研究及全文检索系统设计

发布时间：2018-02-22 05:55

本文关键词： 全文检索 Lucene 中文分词 HMM模型资源库　出处：《华中师范大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着信息时代的飞速发展,信息量越来越多,如何从众多的信息中找出自己想要的信息变得越来越重要,如荆楚资源库中的武当山资源库,资源总量达到几百G,各种类型的文档总数有几千个,从众多的文档中找出特定的信息变得比较困难,信息检索技术就是用来解决这个问题。全文检索作为信息检索的一种,正在扮演者越来越重要的角色,很多大型搜索引擎都采用了全文检索技术。中文分词是中文信息处理的第一步,无论是自然语言处理还是全文检索,都离不开中文信息的提取,而信息提取必然涉及到分词。中文由于字与字之间没有空格作为词分隔符且中文语义语境都比较复杂,导致中文分词一直是一个难点,针对中文分词人们提出了各种各样的方法来分词,如词典分词、统计分词、理解分词等。本文分析了全文检索技术的原理并讨论了开源的全文检索框架Lucene,接着针对全文检索必须要使用文本切分提取信息,讨论了中文分词相关原理及技术,针对目前广泛采用的词典分词和统计分词做了详细的讨论,比较了各种分词方法的优点以及缺点,提出了一种基于词典与统计相结合的分词方法。本文所做的工作如下:1.分析了全文检索和中文分词的研究背景和研究现状,并对目前比较常用的全文检索和中文分词技术做了分析和描述;2.分析了常用的分词技术并在比较各种方法的优劣的基础上提出了一种基于词典与统计相结合的分词方法,该方法利用词典分词切分整体效果良好和统计分词具有歧义识别的优点,采用词典分词做粗分和HMM模型做歧义判断,以达到提高分词精度的目的。3.利用Lucene框架结合自定义分析器,进行资源库全文检索系统的设计。
[Abstract]:With the rapid development of the information age, more and more information, how to find the information they want from the numerous information becomes more and more important, such as the Wudang Mountain resource bank in Jingchu resource bank. The total amount of resources is several hundred Gand the total number of documents of various types is several thousand. It becomes more difficult to find specific information from many documents. Information retrieval technology is used to solve this problem. Full-text retrieval is a kind of information retrieval. Chinese word segmentation is the first step of Chinese information processing, whether natural language processing or full-text retrieval, can not be separated from the extraction of Chinese information. Chinese word segmentation is always a difficulty because there is no space between words as word separator and Chinese semantic context is complicated. For Chinese word participle people put forward a variety of methods for word segmentation, such as dictionary participle, statistical word segmentation, This paper analyzes the principle of full-text retrieval technology and discusses the open source full-text retrieval framework Lucene.Then, aiming at the need to use text segmentation to extract information in full-text retrieval, the related principles and techniques of Chinese word segmentation are discussed. This paper makes a detailed discussion on lexical segmentation and statistical participle which are widely used at present, and compares the advantages and disadvantages of various word segmentation methods. A word segmentation method based on the combination of dictionary and statistics is proposed. The work done in this paper is as follows: 1. The research background and status quo of full-text retrieval and Chinese word segmentation are analyzed. This paper also analyzes and describes the popular full-text retrieval and Chinese word segmentation techniques. 2. This paper analyzes the common word segmentation techniques and puts forward a word segmentation method based on the combination of dictionary and statistics on the basis of comparing the advantages and disadvantages of various methods. In this method, the overall effect of word segmentation in dictionaries is good and the statistical segmentation has the advantages of ambiguity recognition. The rough segmentation of dictionary segmentation and the HMM model are used to judge ambiguity. In order to improve the accuracy of word segmentation. 3. The design of full-text retrieval system of resource base is carried out by using Lucene framework and self-defined analyzer.
【学位授予单位】：华中师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【参考文献】