Web应用安全漏洞扫描器爬虫技术的改进与实现

发布时间：2018-03-05 08:22

本文选题：Web漏洞　切入点：网络安全　出处：《北京邮电大学》2014年硕士论文　论文类型：学位论文

【摘要】：过去的几十年中,Web信息技术飞速发展,Web网络已经成为人们生活中不可或缺的一部分。然而Web系统的广泛应用在给生活带来便利的同时,也带来了很多安全隐患。很多公司开发Web应用程序时不太重视该应用程序可能存在的安全风险。导致应用投入使用后,存在大量安全漏洞,给黑客留下大量可乘之机,时刻威胁着用户的上网安全。论文首先全面分析了目前Web应用安全状况,介绍了常见的Web安全漏洞。接着论文从输入参数覆盖率、爬取能力、认证覆盖率三个方面对国内外漏洞扫描器进行分析对比,提出为提高效率而改进的重点是扫描器的爬取速率以及支持身份认证网页信息的采集。在结合分布式系统的基础上,论文提出了分布式爬虫方法,将爬取节点分布在多台独立的主机上,使得整个爬虫系统具有通用爬虫所不具备的易扩展性和高效性。另外论文详细分析了分布式爬虫系统中存在的一些关键技术及其解决方案,包括URL任务分配、URL去重、DNS解析、基于身份认证的信息采集,使得整个爬虫系统在协调工作、负载均衡方面表现优秀。最后论文从扫描速率和身份认证两个方面对漏洞扫描器进行了测试,实验结果证明,论文中设计的基于分布式爬虫的漏洞扫描器能够提升扫描速率,并且可以采集需要身份认证的网页信息。
[Abstract]:In the past few decades, the rapid development of web information technology has become an indispensable part of people's life. However, the wide application of Web system has brought convenience to our daily life. Many companies don't pay much attention to the potential security risks of Web applications. As a result, there are a lot of security vulnerabilities after the application is put into use, leaving a lot of opportunities for hackers to take advantage of. It is always threatening the security of the user's internet. Firstly, the paper analyzes the current Web application security situation, introduces the common Web security vulnerabilities, and then analyzes and compares the vulnerability scanners at home and abroad from three aspects: input parameter coverage, crawling ability, authentication coverage. In order to improve the efficiency, the emphasis of the paper is the crawling rate of scanner and the collection of identity authentication web pages. Based on the distributed system, this paper proposes a distributed crawler method. The crawler nodes are distributed across multiple independent hosts, It makes the whole crawler system more extensible and efficient than the common crawler system. In addition, some key technologies and their solutions in distributed crawler system are analyzed in detail, including URL task allocation URLs to resolve the problem. The information collection based on identity authentication makes the whole crawler system work well in coordination and load balance. Finally, the paper tests the vulnerability scanner from two aspects: scanning rate and identity authentication. The vulnerability scanner based on distributed crawler designed in this paper can enhance the speed of scanning and collect the information of web pages that require identity authentication.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.08

【参考文献】