暗网资源挖掘的关键技术研究

发布时间：2018-03-02 04:19

本文关键词： 暗网查询接口更多链接发现　出处：《西安电子科技大学》2013年硕士论文　论文类型：学位论文

【摘要】：进入21世纪后，科技发展日新月异，互联网技术飞速发展，特别是Web2.0技术的引入使得互联网应用渗透到社会生活的各领域。人们从互联网上获取信息的手段通常是使用浏览器，而浏览器搜索的基础是搜索引擎，搜索引擎是指自动从互联网搜集信息，经相关整理后给用户进行查询的系统。搜索引擎索引的资源通过搜索程序检索信息，这些搜索程序就像蜘蛛一样在网络间爬来爬去，被称为网络爬虫，网络爬虫根据网页中的超链接关系抓取资源，这些资源被称为浅层网络资源。然而，互联网中还存在很多资源无法通过网络爬虫直接获取，这些资源隐藏在Web数据库中，只能通过与数据库相关的查询接口提交查询的方式访问，这些资源被称为暗网资源。如果搜索引擎能够对其进行检索，将对提高搜索引擎的资源覆盖率有重要的现实意义。本文针对暗网(Hidden Web)资源挖掘进行了深入研究，分析暗网资源挖掘中的难点问题并提出了解决这些问题的算法。本文的主要工作包括以下几个部分： (1)对暗网查询接口识别和分类的分析与研究； (2)设计了暗网查询接口的查询构造方法； (3)设计并实现了算法来验证构造的URL链接的有效性，，URL链接的有效性基于该URL对应的页面信息的有效性； (4)设计并实现了更多链接发现算法。
[Abstract]:In 21th century, with the rapid development of science and technology and the rapid development of Internet technology, especially the introduction of Web2.0 technology, Internet applications infiltrate into various fields of social life. People usually use browsers to obtain information from the Internet. The foundation of browser search is search engine. Search engine is a system that automatically collects information from the Internet and queries users after related collation. The resources of search engine index retrieve information through search program. These search programs are like spiders crawling around the web, known as web crawlers, which grab resources based on hyperlinks in a Web page, which are called shallow web resources. There are still many resources in the Internet that can not be directly obtained by the web crawler. These resources are hidden in the Web database and can only be accessed by submitting the query through the query interface related to the database. These resources are called dark web resources. If search engines can search them, it will be of great practical significance to improve the resource coverage of search engines. In this paper, we deeply study the hidden network resource mining, analyze the difficult problems in the dark network resource mining and propose the algorithms to solve these problems. The main work of this paper includes the following parts:. 1) the analysis and research on the identification and classification of the dark network query interface; 2) the query construction method of the dark network query interface is designed. 3) the algorithm is designed and implemented to verify the validity of the constructed URL link. The validity of the URL link is based on the page information corresponding to the URL. 4) more link discovery algorithms are designed and implemented.
【学位授予单位】：西安电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【参考文献】