Web文本分类方法研究与系统实现

发布时间：2018-12-07 20:55

【摘要】： 近年来,Web已经飞速发展成为了世界上数据量最大的公共信息源。如何使Web用户能够在浩瀚的信息资源中方便、快捷的定位到所需要的信息,已经成为迫切需要解决的问题。Web文本的正确分类正是其中的核心问题。Web文本分类源自于自动分类技术,是Web文本挖掘的重要组成部分。它不仅可以有效提高用户的搜索效率,帮助用户快速、准确的定位到目标知识,而且还可以获取到不同用户的类别兴趣特征,为满足用户的个性化服务要求提供参考。目前的分类研究多把文档类别看成是平面化的、不相交的,没有考虑到类别间的层次关系。当类别数目较多时,平面分类学习得到分类器的时间开销大,而且在对未知文档分类时,需要与全部类模型进行比较,这显然很不恰当。本文在对Web文本挖掘及自动分类技术进行深入研究的基础上,结合类别间的层次关系,实现了一个多层次的Web文本分类系统。本文创新点和关键技术如下: 1.建立了层次化的训练和分类模型:本文针对网页内容丰富、涉及多领域的多个类别的特征,分析了平面分类方法在多类别情况下存在的问题,提出了层次分类的思想,建立了层次化的训练和分类模型。 2.设计并实现了Web文本的自动抽取器:Web网页中掺杂的广告、超链接等噪声给Web文本分类带来了极大困扰。本文实现了一个Web文本自动抽取器,使Web页面经过处理变为较纯净的包含标题和正文内容的纯文本。 3.提出了一种适合于Web网页的关键词提取方法:网页中不同位置和不同词性的词语对表达网页内容所起的作用也有所不同,针对这一特点,本文提出了基于词性、位置和词频信息加权的关键词提取方法来进一步过滤掉网页噪声词,取得了较好的效果。 4.提出了一种基于χ2统计量加权的分类方法:χ2统计量能够很好的反映特征和类别间的相关性。本文创新性的将χ2统计量应用于文本分类,不但简化了分类过程,而且在实际应用中得到了较好的分类速度和准确度。本论文根据Web文本的特点提出了一套针对大规模、多类别的Web文本进行分类的实施方案,设计了一个Web文本的多层次分类系统。结果表明,本系统在实践中的分类性能优于一般的平面分类器。
[Abstract]:In recent years, Web has developed rapidly into the largest public information source in the world. How to enable Web users to locate the needed information conveniently and quickly in the vast information resources, The correct classification of Web text is the core problem. Web text classification is derived from automatic classification technology and is an important part of Web text mining. It not only can effectively improve the search efficiency of users, help users to locate the target knowledge quickly and accurately, but also can obtain the interest characteristics of different users, and provide a reference to meet the personalized service requirements of users. Most of the current classification studies regard document categories as flat, disjoint, and do not take into account the hierarchical relationship between categories. When the number of categories is large, the time cost of learning classifier by plane classification is very large, and when classifying unknown documents, we need to compare them with all class models, which is obviously not appropriate. Based on the in-depth study of Web text mining and automatic classification technology, this paper implements a multi-level Web text classification system based on the hierarchical relationship between categories. The innovations and key technologies of this paper are as follows: 1. A hierarchical training and classification model is established. Aiming at the features of many kinds of web pages which are rich in content and involving many fields, this paper analyzes the problems existing in the method of plane classification in the case of multiple categories, and puts forward the idea of hierarchical classification. A hierarchical training and classification model is established. 2. An automatic Web text extractor is designed and implemented. The noise such as ads and hyperlinks in Web pages brings great trouble to Web text classification. In this paper, an automatic Web text extractor is implemented, which makes the Web page become pure text containing title and text. 3. In this paper, a keyword extraction method suitable for Web web pages is proposed. Different positions and different parts of speech in web pages play different roles in the expression of web pages. In view of this characteristic, this paper proposes a new method based on part of speech. Position and word frequency information weighted keyword extraction method to further filter out the page noise words, and achieved good results. 4. A classification method based on the weighting of 蠂 2 statistics is proposed. 蠂 2 statistics can well reflect the correlation between features and categories. This paper innovatively applies 蠂 2 statistics to text classification, which not only simplifies the classification process, but also obtains better classification speed and accuracy in practical application. According to the characteristics of Web texts, this paper proposes a set of implementation schemes for large-scale, multi-class Web text classification, and designs a multi-level classification system for Web texts. The results show that the classification performance of this system is better than that of general plane classifier in practice.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2010
【分类号】：TP391.1

【参考文献】