基于语义分析的垃圾页面检测算法研究

发布时间：2018-05-05 21:22

本文选题：搜索引擎 + 垃圾网页检测　；参考：《大连理工大学》2013年硕士论文

【摘要】：垃圾网页是针对搜索引擎结果排名设计的,而不是为了提高用户上网体验。为了能使某些页面在搜索引擎的检索排名中比较靠前,垃圾网页制作者往往采用欺骗搜索引擎的手段。垃圾网页的猖獗使得搜索引擎的检索排名越来越不能反映网页的真实价值,用户的搜索体验不断下降,垃圾网页已经成为搜索引擎面临的最大挑战之一。因此,研究有效的垃圾网页检测算法具有深远的意义。本文首先对搜索引擎的工作原理进行了简单描述,并介绍了几种常用的垃圾网页检测技术,对他们的研究状况做了简单的分析。接着说明了这些技术的基本原理以及他们的特点,总结了一系列反搜索引擎技术。本文针对垃圾网页的特点,围绕特征提取和分类器设计等关键问题进行了研究,并用机器学习的方法,系统设计了垃圾网页检测框架。核心思想是提取网页中具有代表性的内容特征,然后用机器学习算法训练这些特征,构建检测模型,再用生成的检测模型对网页进行分类。这样就把垃圾网页的检测转化成机器学习中的分类问题,采用C4.5决策树算法,将网页分类为正常网页和垃圾网页。之后又加入了Bagging和Boosting方法,进一步提高了分类的准确度。本文在公开的标准测试数据集WEBSPAM-UK2007上进行了实验,实验结果表明用本文的基于语义分析的分类算法可以有效的检测垃圾网页。
[Abstract]:Spam pages are designed for ranking search engine results, not for improving the user's online experience. In order to make some pages rank higher in search engines, spam web page makers often use the means of cheating search engines. The rampant spam pages make the search ranking of search engines more and more unable to reflect the true value of web pages, and the search experience of users is declining. Spam pages have become one of the biggest challenges facing search engines. Therefore, it is of great significance to study the effective spam detection algorithm. In this paper, the working principle of search engine is briefly described, and several common spam page detection techniques are introduced, and their research status is analyzed briefly. Then it explains the basic principle of these technologies and their characteristics, and summarizes a series of anti-search engine technologies. According to the characteristics of garbage pages, this paper studies the key problems such as feature extraction and classifier design, and designs the detection framework of garbage pages by means of machine learning. The core idea is to extract the representative content features from web pages, then train these features with machine learning algorithm, construct the detection model, and then use the generated detection model to classify the web pages. In this way, the detection of garbage pages is transformed into a classification problem in machine learning, and C4.5 decision tree algorithm is used to classify web pages into normal pages and junk pages. Then Bagging and Boosting methods were added to further improve the accuracy of classification. The experimental results on the open standard test data set (WEBSPAM-UK2007) show that the classification algorithm based on semantic analysis can effectively detect garbage pages.
【学位授予单位】：大连理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】