面向内容的网页分类方法研究与实现

发布时间：2018-05-28 05:39

本文选题：网页分类 + 特征选择　；参考：《南京邮电大学》2017年硕士论文

【摘要】：在当今信息化时代中,互联网逐步渗透到社会生活的各个方面,从而使得浏览网页成为人们获取信息的主要方式。然而,随着互联网的飞速发展,互联网中的网页数量急速增长。面对海量且复杂的网页,人们通常无法迅速地获得自己所需的内容而导致其用户体验较差。基于此,本文主要研究如何对网页按照不同主题进行分类,分类过程包括两个步骤:网页数据预处理与文本分类。在获取到网页数据之后,首先需要对其进行预处理,网页数据预处理的过程大致包括网页去噪、中文分词、特征选择以及文本表示;其次,将得到的数值型数据输入分类模型中进行文本分类。本文主要针对特征选择以及文本分类进行改进。首先根据网页的内容与结构特性,结合并改进布隆过滤器和TF-IDF算法,提出一种基于降维的特征选择算法,以便剔除冗余的特征项;其次,本文对能有效地处理较大规模数据的支持向量机算法进行改进,通过分析核函数的原理和性质,构建了一种全新的混合核函数,并利用遗传算法的交叉验证搜索方式来对新核函数寻求最佳参数,由此提出了一个兼具学习能力与泛化能力的支持向量机,接着通过仿真实验对比验证其在文本分类中的优越性能。最后实现了一个基于该改进算法的网页分类系统,将系统用于对网页内容的分类,实验结果表明,本文提出的算法在大部分情况下能取得较好的分类效果,该算法在网页分类领域的应用中有一定的实用价值。
[Abstract]:In today's information age, the Internet has gradually penetrated into all aspects of social life, thus making browsing the web page the main way for people to obtain information. However, with the rapid development of the Internet, the number of web pages in the Internet is growing rapidly. In the face of massive and complex web pages, people usually can not get the content they need quickly, resulting in poor user experience. Based on this, this paper mainly studies how to classify web pages according to different topics. The classification process includes two steps: page data preprocessing and text classification. After getting the web page data, we need to preprocess it first. The process of page data preprocessing includes web page denoising, Chinese word segmentation, feature selection and text representation. The obtained numerical data is input into the classification model for text classification. This paper mainly focuses on feature selection and text classification. First of all, according to the content and structure of the web page, combining and improving the Bron filter and TF-IDF algorithm, a feature selection algorithm based on dimensionality reduction is proposed in order to eliminate redundant feature items. In this paper, the support vector machine (SVM) algorithm, which can deal with large scale data effectively, is improved. By analyzing the principle and properties of kernel function, a new hybrid kernel function is constructed. A support vector machine with both learning ability and generalization ability is proposed to search for the best parameters of the new kernel function by using the cross-validation search method of genetic algorithm. Then the superior performance in text classification is verified by simulation experiments. Finally, a web page classification system based on the improved algorithm is implemented. The experimental results show that the proposed algorithm can achieve a good classification effect in most cases. The algorithm has some practical value in the field of web page classification.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092;TP391.1

【参考文献】