基于领域特殊性和统计语言知识的新词抽取方法

发布时间：2018-10-31 07:49

【摘要】：近年来,随着经济社会的快速发展,大量新词出现在人们生活中。在自然语言处理领域,许多研究方向等都离不开新词的自动抽取。作为语言信息处理领域的一项基础技术,新词抽取技术具有巨大的研究价值和实际应用前景。本文提出了一种新颖的新词抽取方法,主要工作如下:1.提出了一个基于领域特殊性和统计语言知识的新词抽取方法。通过观察、分析语料的特点,采用基于领域特殊性的垃圾串过滤方法过滤垃圾串,得到候选新词列表;然后基于统计语言知识(包括词频、内部结合紧密性)对新词进行抽取。实验验证了该方法的有效性。2.新词抽取方法的优化,从两个方面对新词抽取方法进行了优化:优化内部结合紧密性,采用EMI来衡量,替换PMI;引入上下文外部特征,采用左熵和右熵来衡量词语的自由度。并从多方面采用多种方法评估比较该方法的效果,评估不同统计特征的结合以及调整参数。实验结果显示,相比未优化前的方法,新词抽取的效果得到大大提升,准确率最大提升39%,召回率最大提升63%。3.新词抽取方法的应用验证,将抽取的新词应用在分词系统中,实验结果显示,在含有新词的语料上,分词效果提升了10%;另外,新词抽取方法能够应用在英文领域词典的构建上。实验验证了本文方法可扩展性和语言独立性的特点。基于领域特殊性和统计语言知识的新词抽取方法是一种无监督的方法,它不需要训练语料,不需要定义规则,克服了传统方法的缺点。此外,本文方法具有很强的可扩展性和语言独立性,能够抽取大量的新词和领域词语。
[Abstract]:In recent years, with the rapid development of economy and society, a large number of new words appear in people's lives. In the field of natural language processing, many research directions are inseparable from the automatic extraction of new words. As a basic technology in the field of language information processing, neologism extraction technology has great research value and practical application prospect. In this paper, a novel new word extraction method is proposed. The main work is as follows: 1. A new word extraction method based on domain particularity and statistical language knowledge is proposed. By observing and analyzing the characteristics of the corpus, the garbage string filtering method based on domain particularity is used to filter the garbage string, and the list of candidate new words is obtained, and then the new words are extracted based on the knowledge of statistical language (including word frequency, internal compactness). The experimental results show that the method is effective. 2. This paper optimizes the neologism extraction method from two aspects: optimizing the internal compactness, using EMI to measure, replacing PMI; introducing the external features of context, and using left entropy and right entropy to measure the degree of freedom of words. Various methods are used to evaluate and compare the effect of the method, to evaluate the combination of different statistical characteristics and to adjust the parameters. The experimental results show that, compared with the unoptimized method, the effect of neologism extraction is greatly improved, the accuracy is increased by 39 and the recall rate is increased by 63. 3. The new words extraction method is applied to the word segmentation system. The experimental results show that the segmentation effect has been improved by 10% in the corpus containing new words. In addition, neologism can be applied to the construction of English domain dictionaries. Experiments verify the extensibility and language independence of this method. The new word extraction method based on domain particularity and statistical language knowledge is an unsupervised method. It does not require training corpus and does not need to define rules, which overcomes the shortcomings of traditional methods. In addition, this method has strong extensibility and language independence, it can extract a large number of new words and domain words.
【学位授予单位】：北京理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.1

【相似文献】