基于CHI和KNN的文本特征选择与分类算法的研究

发布时间：2018-05-08 00:12

本文选题：文本分类 + 特征选择　；参考：《北京工业大学》2016年硕士论文

【摘要】：随着文本信息的快速膨胀,从庞杂的信息中获取对自己有用的信息越来越困难。文本自动分类作为一种有效组织和管理文本信息的数据挖掘方法,解决了文本信息杂乱的问题,在信息检索、信息过滤、邮件过滤、数字化图书馆等领域有着非常广泛的应用前景。研究出一套分类精度高、效率高、稳定性好的文本分类方法具有重要的学术意义和实用价值。特征选择、特征加权及分类是文本分类过程中较重要的环节,本文以此为主要内容展开研究。首先简要说明了文本分类的研究背景和意义,阐述了文本分类的国内外研究现状,概括了本文的研究内容和章节安排。然后介绍了文本分类过程的关键技术和分类性能评价指标,为后期研究打好基础。最后,本文针对CHI特征选择、TFIDF特征加权、KNN分类方法进行了深入研究并加以改进,主要研究内容如下:(1)针对CHI模型未考虑特征项词频,及放大了在指定类中很少出现但在其他类中普遍存在的特征项权重的缺点,本文基于传统CHI算法,提出了一种自适应的特征选择方法,该方法引入了自适应比例因子,可自动调节与类别呈正、负相关特征项的比重,消除了人为选取比例因子带来的误差,又引入了词频因子和类间方差,使最终选择出的特征项在特定类中出现频数多而在其他类中分布较少。结合KNN方法进行实验,实验结果表明,无论在平衡语料集上还是在非平衡语料集上,所提方法都有较好的分类效果,尤其提高了在非平衡语料集上的分类效果。(2)针对TFIDF方法未考虑特征项类间、类内分布因素的不足,本文提出了一种基于CHI统计和信息熵的改进型TFIDF特征加权方法,由于特征项的2?统计量和类内信息熵分别反映了特征项的类间、类内分布情况,因此,在传统TFIDF方法的基础上,引入了2?统计量调节因子和类内分布熵因子,弥补了TFIDF方法的缺陷,提高了特征项权重计算的准确性。结合KNN方法进行实验,实验结果表明,所提方法用于权重计算后提高了分类器性能,同时证明了该方法具有较好的稳定性。(3)针对KNN方法训练样本增多而分类速度变慢的问题,本文提出了一种基于K-Medoids和隶属度的改进型KNN文本分类方法,在传统KNN算法的基础上,采用改进K-Medoids聚类算法删掉对KNN分类贡献小的训练样本,减少了分类过程中相似度的计算量,并引入了隶属度,实现了有差别地处理与测试文本最近邻的K个文本。实验结果表明,在保证较高文本分类精度的前提下,所提方法提高了KNN方法的分类效率,同时验证了综合采用本文所提三种方法的有效性。
[Abstract]:With the rapid expansion of text information, it is becoming more and more difficult to obtain useful information from complex information. As an effective data mining method for organizing and managing text information, automatic text classification solves the problem of text information clutter, such as information retrieval, information filtering, mail filtering. Digital library and other fields have a very broad application prospects. It is of great academic significance and practical value to develop a set of text classification methods with high classification accuracy, high efficiency and good stability. Feature selection, feature weighting and classification are important links in the process of text classification. Firstly, the research background and significance of text classification are briefly explained, the current research situation of text classification at home and abroad is expounded, and the research content and chapter arrangement of this paper are summarized. Then it introduces the key technology of text classification process and classification performance evaluation index, which lays a good foundation for the later research. Finally, this paper makes an in-depth study and improvement on the classification method of CHI feature selection / TFIDF feature weighted KNN. The main research contents are as follows: (1) for the CHI model, the word frequency of feature items is not taken into account. In this paper, an adaptive feature selection method is proposed based on the traditional CHI algorithm, in which the adaptive scaling factor is introduced. It can automatically adjust the proportion of the positive and negative characteristic items, eliminate the error caused by the artificial selection of the proportion factor, and introduce the word frequency factor and inter-class variance. The final selected feature items appear more frequently in certain classes than in other classes. The experimental results with KNN method show that the proposed method has a good classification effect both on the balanced corpus and on the non-balanced corpus. In particular, the classification effect on the non-equilibrium corpus is improved. (2) aiming at the deficiency of the TFIDF method which does not take into account the inter-class and intra-class distribution factors of the feature items, an improved TFIDF feature weighting method based on CHI statistics and information entropy is proposed in this paper. Because of the 2? Statistics and intra-class information entropy reflect the distribution of feature items between classes and within classes respectively. Therefore, based on the traditional TFIDF method, we introduce 2? The statistical adjustment factor and the distribution entropy factor make up the defect of TFIDF method and improve the accuracy of calculating the weight of the feature term. The experimental results with KNN method show that the proposed method improves the performance of classifier after weight calculation, and proves that the proposed method has good stability. (3) aiming at the problem that the training sample of KNN method increases and the classification speed becomes slower, In this paper, an improved KNN text classification method based on K-Medoids and membership degree is proposed. Based on the traditional KNN algorithm, the improved K-Medoids clustering algorithm is used to delete the training samples that contribute little to the KNN classification, thus reducing the amount of computation of similarity in the classification process. The membership degree is introduced to deal with K text that is nearest to the test text. The experimental results show that the proposed method improves the classification efficiency of the KNN method and verifies the effectiveness of the three methods proposed in this paper.
【学位授予单位】：北京工业大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.1

【相似文献】