CCDet:一种高效的大规模中文重复网页检测方法

发布时间：2018-02-21 01:10

本文关键词： CCDet算法重复网页检测中文句号特征索引剪切　出处：《计算机研究与发展》2013年S2期 　论文类型：期刊论文

【摘要】：重复文档检测是信息检索领域中一个非常重要的问题.由于网页结构和内容的复杂性,现有方法在网页查重上没有达到很好的准确性,且只有少量工作用于处理包含关系网页检测问题;同时,由于网页数量的巨大,重复网页检测处理时需要考虑大规模数据的并行化算法.提出一种基于句号特征的大规模重复中文网页检测方法CCDet.CCDet采用了一种基于中文句号特征来完成重复文档的相似性比对方法,与现有的主要重复网页检测算法相比,CCDet大幅提高了检测具有重复关系网页和具有包含关系网页的准确性,并拥有较高的检测效率.同时,为了适应大规模新闻网页的查重处理,使用MapReduce编程框架实现了并行化的CCDet算法,使之能够并行化地进行重复网页检测.实验结果表明,并行化的CCDet算法具有较好的检测效果和计算性能,并具有良好的可扩展性.
[Abstract]:Duplicate document detection is a very important problem in the field of information retrieval. Due to the complexity of web page structure and content, the existing methods do not achieve good accuracy in page retrieval. And only a small amount of work is used to deal with the problem of containing relational page detection; at the same time, because of the large number of pages, The parallelization algorithm of large scale data should be taken into account in the detection of duplicate pages. A large scale Chinese page detection method based on period feature is proposed in this paper. CCDet.CCDet adopts a Chinese period feature to complete duplicate documents. Similarity comparison method, Compared with the existing algorithms, the CCDet greatly improves the accuracy and efficiency of the detection of web pages with repeatable relationship and contains relation pages. At the same time, in order to adapt to the reprocessing of large scale news pages, CCDet improves the accuracy of detecting web pages with duplicate relationship and contains relational pages. The parallel CCDet algorithm is implemented by using the MapReduce programming framework, which makes it possible to detect repeated web pages in parallel. The experimental results show that the parallel CCDet algorithm has good detection effect and computational performance, and has good scalability.
【作者单位】：南京大学计算机科学与技术系;
【基金】：国家自然科学基金项目(61072152) 江苏省工业支撑计划基金项目(BE2011172)
【分类号】：TP393.092;TP391.3

【共引文献】