级联式低消耗大规模网页分类在线获取方法

发布时间：2018-06-09 03:47

本文选题：大规模网页数据获取 + 网页分类　；参考：《计算机应用》2017年04期

【摘要】：针对海量网页在线自动高效获取网页分类系统设计中如何更有效地平衡准确度与资源开销之间的矛盾问题,提出一种基于级联式分类器的网页分类方法。该方法利用级联策略,将在线与离线网页分类方法结合,各取所长。级联分类系统的一级分类采用在线分类方法,仅利用锚文本中网页标题包含的特征预测其分类,同时计算分类结果的置信度,分类结果的置信度由分类后验概率分布的信息熵度量。若置信度高于阈值(该阈值采用多目标粒子群优化算法预先计算取得),则触发二级分类器。二级分类器从下载的网页正文中提取特征,利用预先基于网页正文特征训练的分类器进行离线分类。结果表明,相对于单独的在线法和离线法,级联分类系统的F1值分别提升了10.85%和4.57%,并且级联分类系统的效率比在线法未降低很多(30%左右),而比离线法的效率提升了约70%。级联式分类系统不仅具有更高的分类能力,而且显著地减少了分类的计算开销与带宽消耗。
[Abstract]:Aiming at the problem of how to balance the contradiction between accuracy and resource cost more effectively in the design of online automatic and efficient web page classification system for massive web pages, a web page classification method based on cascade classifier is proposed. This method combines online and offline web page classification methods with concatenation strategy. The first level classification of cascaded classification system adopts online classification method. Only the features contained in the page title in anchor text are used to predict the classification, and the confidence degree of the classification result is calculated at the same time. The confidence of classification results is measured by the information entropy of the posteriori probability distribution. If the confidence level is higher than the threshold (the threshold is obtained by multi-objective particle swarm optimization algorithm in advance), the two-level classifier is triggered. The second level classifier extracts features from the downloaded text of a web page and uses a classifier based on the training of page text features to carry out offline classification. The results show that the F _ 1 value of the cascade classification system increases by 10.85% and 4.57% respectively compared with the single on-line method and offline method, and the efficiency of the cascade classification system is less than that of the on-line method by about 30%, and the efficiency of the cascade classification system is increased by about 70% compared with the offline method. The cascade classification system not only has a higher classification capability but also significantly reduces the computational overhead and bandwidth consumption of the classification.
【作者单位】：成都信息工程大学软件工程学院;广东省气象台;
【基金】：国家自然科学基金资助项目(61501063,61501064) 四川省科技计划项目(2016JY0240) 四川省教育厅科研基金资助项目(15ZB0177)~~
【分类号】：TP393.092

【相似文献】