LDA模型下不同分词方法对文本分类性能的影响研究

发布时间：2018-07-25 16:48

【摘要】：通过定义类别聚类密度、类别复杂度以及类别清晰度三个指标,从语料库信息度量的角度研究多种代表性的中文分词方法在隐含概率主题模型LDA下对文本分类性能的影响,定量、定性地分析不同分词方法在网页和学术文献等不同类型文本的语料上进行分类的适用性及影响分类性能的原因。结果表明:三项指标可以有效指明分词方法对语料在分类时产生的影响,Ik Analyzer和ICTCLAS分词法分别受类别复杂度和类别聚类密度的影响较大,二元分词法受三个指标的作用相当,使其对于不同语料具有较好的适应性。对于学术文献类型的语料,使用二元分词法时的分类效果较好,F1值均在80%以上;而网页类型的语料对于各种分词法的适应性更强。尝试通过对语料进行信息度量而非单纯的实验来选择提高该语料分类性能的最佳分词方法,以期为网页和学术文献等不同类型的文本在基于LDA模型的分类系统中选择合适的中文分词方法提供参考。
[Abstract]:By defining three indexes of clustering density, class complexity and class clarity, the paper studies the influence of several representative Chinese word segmentation methods on text classification performance under the implicit probability topic model (LDA) from the perspective of corpus information measurement. This paper quantitatively and qualitatively analyzes the applicability of different word segmentation methods in the corpus of different types of texts, such as web pages and academic documents, and the reasons that affect the classification performance. The results show that the three indexes can effectively indicate the influence of word segmentation on the classification of corpus. Ik Analyzer and ICTCLAS participle are influenced by the class complexity and cluster density respectively, while the binary participle is equally affected by the three indexes. So that it has better adaptability to different corpus. For the corpus of academic literature, the classification effect of binary participle method is better than 80%, while the data of web page type is more adaptable to all kinds of word segmentation methods. This paper attempts to choose the best word segmentation method to improve the classification performance of the corpus by measuring the information of the corpus rather than simply experimenting. It is expected to provide a reference for different types of text such as web pages and academic documents to select suitable Chinese word segmentation methods in the classification system based on LDA model.
【作者单位】：武汉大学信息管理学院;武汉大学信息资源研究中心;
【基金】：国家社会科学基金资助项目(15BTQ066)
【分类号】：TP391.1

【相似文献】