基于海量互联网网页拓扑结构的作弊链接与恶意网页挖掘

发布时间：2019-04-03 20:50

【摘要】：万维网提供了大量的信息,任何人都可以访问它。为了识别网页中大量最有价值的信息,互联网用户主要依赖于搜索引擎。搜索引擎通常对大量网页进行分类,并且根据查询相关性与网页排名给出与用户查询最相关的网页。用户通常访问排名最高的网页,忽略其余部分。所以,为了吸引更多互联网用户点击,每个网页在搜索引擎中获得较高排名是非常重要的。搜索引擎是帮助用户找到所需内容的主要方法。为了针对用户的查询建议并给出最密切相关和最流行的网页,搜索引擎会根据某些算法向每个网页分配排名,该排名通常随着链接到该页面的其他网站的数量和排名而增加。然而,作弊链接攻击者已经开发出几种技术来应对这些算法,并提高其自身网页的排名。这些技术通常基于用于协作链路交换的地下链接,在作弊链接开发者之间建立关系网络以在搜索引擎结果中提高他们的网页排名。本文研究如何在海量互联网节点和边之上识别针对搜索引擎的作弊链接与作弊网页,收集互联网上的网页以及他们之间的超文本链接,构造一个互联网拓扑,研究和分析这些作弊链接构成的子图在整个拓扑结构中的特征,并通过扩展的方式追踪这些作弊链接的指向关系,从而识别互联网上的作弊网页。在本文研究中,我们对作弊网页与作弊链接拓扑结构特征进行了比较全面的分析与总结,预测了作弊链接拓扑结构特征,并且根据作弊网页分类与作弊链接拓扑结构特征提出了基于互联网网页拓扑结构的作弊链接与恶意网页挖掘模型,并在该模型中提出了一种简单但高效的种子节点采集与扩展算法。在扩展种子集时,可以在链接农场中找到一些页面作为种子集,则对于每个新页面,如果页面具有从和到达的多个入站链路和出站链路,则该页面很可能是同一链接农场的一部分种子集。然后可以通过添加新页面来扩展种子集。得到种子集后,需要扩展步骤来在数据集中找到更多的坏页,才能建立作弊链接拓扑结构。进行扩展步骤时,如果一个页面指向一堆坏页面,很可能这个页面本身是坏的。因此,从一个页面扩展到链接页面,尽管这里遵循入站链接而不是出站链接。为了验证本文所提出的模型对于互联网上作弊网页挖掘的性能,本文使用Python爬虫模块进行网页挖掘,实验数据根据爬取时间分为三组,共计9.5万页,这些页面位于8452个不同的域中。其中标记作弊网页数共计6208个,得到的种子节点180个。通过三组实验数据显示,本文所提出的基于互联网网页拓扑结构的作弊链接与恶意网页挖掘模型的综合准确率为83.3%,基本上达到了检测作弊网页与链接农场的目的。并且通过实验数据所绘制的作弊链接拓扑结构与作弊链接拓扑结构特征预测所预测的拓扑结构基本一致,从而证明了本文中所对作弊链接拓扑结构的猜想是基本正确的。进一步,通过跟踪这些作弊链接的指向,找到他们所服务的作弊网页,并将这些网页进行举报或者公示,从而达到降低这些作弊网页在搜索引擎中曝光的几率,维护互联网安全。
[Abstract]:The World Wide Web provides a large amount of information, and anyone can access it. In order to identify a large number of most valuable information in a web page, the Internet users rely primarily on search engines. The search engine typically classifies a large number of web pages and gives the most relevant web page to the user based on the query relevance and the web page ranking. The user usually accesses the highest ranked web page and ignores the rest. Therefore, in order to attract more Internet user clicks, each web page is highly ranked in a search engine. The search engine is the main method to help users find the desired content. In order to suggest and give the most closely related and most popular web pages for the user's query suggestions, the search engine will assign a ranking to each web page in accordance with certain algorithms that generally increase with the number and ranking of other sites linked to the page. However, cheating-linked attackers have developed several techniques to address these algorithms and improve their own web page ranking. These techniques are typically based on an underground link for collaborative link exchange and a relationship network is established between the cheating link developers to improve their web page ranking in search engine results. In this paper, how to identify the cheating link and the cheating web page for the search engine on the mass Internet nodes and edges, to collect the web pages on the Internet and the hypertext links between them, to construct an Internet topology, The characteristics of the subgraph formed by these cheating links in the whole topology structure are studied and analyzed, and the pointing relation of the cheating links is tracked through the expanded mode, so as to identify the cheating webpage on the Internet. In that study of this paper, we make a comprehensive analysis and summary of the topological structure of the cheating link and the cheating link, and predict the topological structure of the cheating link. And a simple but efficient seed node acquisition and expansion algorithm is proposed in the model. When you expand a seed set, you can find some pages as a set of seeds in the linked farm, and for each new page, if the page has multiple inbound and outbound links from and to, the page is likely to be a subset of the seed set on the same linked farm. You can then expand the seed set by adding a new page. After you get the seed set, you need to expand the steps to find more bad pages in the data set before you can establish a cheating link topology. When an expansion step is performed, if a page points to a pile of bad pages, it is possible that the page itself is bad. Therefore, extend from one page to the linked page, although it follows the inbound link rather than the outbound link. In order to verify the performance of the model proposed in this paper for the web page mining on the Internet, this paper uses the Python crawler module to carry out web-page mining. The experimental data is divided into three groups according to the time-climbing time, and the total amount is 9.5 million pages, which are located in 8452 different domains. The number of the marked cheating pages is 6,208, and the resulting seed nodes are 180. Through three groups of experimental data, the comprehensive accuracy rate of the cheating link and the malicious web page mining model, which is based on the Internet web page topology, is 83.3%, and the purpose of detecting the cheating webpage and linking the farm is basically achieved. And the topological structure of the cheating link and the topological structure of the cheating link topological structure are basically consistent with the predicted topological structure, so that the conjecture of the cheating link topological structure in the paper is basically correct. Further, by tracking the points of the cheating links, finding the cheating web pages they serve, and reporting or publishing the web pages, the chances of reducing the exposure of the cheating web pages in the search engine are reduced, and the Internet security is maintained.
【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092

【相似文献】