文本分类中基于词频再排序的特征选择方法研究

发布时间：2018-03-08 05:01

本文选题：特征选择　切入点：文本分类　出处：《南京大学》2016年硕士论文　论文类型：学位论文

【摘要】：文本自动分类作为数据挖掘技术的分支之一,是处理和组织海量信息的关键技术。通过文本自动分类技术,取代传统的、低效率的人工分类方法,对文本信息进行快速并且准确地分类,使文本信息得以有效的组织,由此提高利用效率。目前,文本自动分类技术已经被广泛应用,所以说文本自动分类技术不仅有重要的理论意义,还有很高的实用价值。文本分类过程主要包括文本预处理、特征降维、分类模型训练、分类模型性能评估等,以及不均衡数据集的处理问题,这些方面都得到了相关学者的深入研究,并且取得的丰富的研究成果,但也存在一些不足之处。本文先对整个文本分类的流程以及一些经典的文本分类算法进行一定的研究,主要是文本预处理的过程,以及朴素贝叶斯、kNN、SVM分类算法的研究。特征降维是文本分类的研究重点之一,其主要目的是依据某一种或某些评价标准对特征全集进行降维处理,从而生成一个较低维度的特征子集,为之后的分类提高速度、准确率与效率。本文对现有的一些经典特征选择算法(文档频率、信息增益、互信息、卡方统计量)进行一定的研究。然后考虑不均衡数据集问题的基础上,提出一种基于词频再排序的特征选择方法,提高分类器分类性能。基于词频再排序的特征选择方法的思想是,在进行常用特征选择方法后,初步选择出一部分特征,然后对各个类别再根据特征在其类别中的词频,进行再一步的筛选,最终构成特征子集。本文选用Reuters-21578数据集,通过在IG、CHI、MI三种特征选择方法上的应用,以及在朴素贝叶斯、kNN、SVM分类器上的实验,证明了该方法的有效性。
[Abstract]:As one of the branches of data mining technology, automatic text classification is the key technology for processing and organizing mass information. Text information can be classified quickly and accurately so that text information can be organized effectively, thus improving the utilization efficiency. At present, automatic text classification technology has been widely used. Therefore, the automatic text classification technology not only has important theoretical significance, but also has high practical value. The text classification process mainly includes text preprocessing, feature dimensionality reduction, classification model training, classification model performance evaluation, etc. And the problem of dealing with unbalanced data sets, these aspects have been deeply studied by relevant scholars, and the rich research results have been obtained. However, there are some shortcomings. Firstly, this paper studies the whole text classification process and some classical text classification algorithms, mainly the process of text preprocessing. Feature dimensionality reduction is one of the key points in text classification. The main purpose of feature reduction is to reduce the dimension of feature complete set according to some evaluation criteria. Thus, a lower dimensional feature subset is generated, which improves the speed, accuracy and efficiency of the later classification. In this paper, some classical feature selection algorithms (document frequency, information gain, mutual information, etc. Then considering the problem of unbalanced data set, a feature selection method based on word frequency reordering is proposed. The idea of the feature selection method based on word frequency reordering is to select a part of the feature after carrying out the common feature selection method, and then select the word frequency of each category according to the feature. In this paper, we select the Reuters-21578 data set, through the application of the three feature selection methods, and the experiment on naive Bayesian KNNN SVM classifier, the effectiveness of this method is proved.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.1

【相似文献】