基于Web信息大数据分析的信息资产研究与实现

发布时间：2018-10-10 07:09

【摘要】：随着Web技术及数字化校园建设的迅速发展,校园局域网存在的Web及其他信息资源,呈现出数量多、规模大、结构复杂、动态变化快等特点,使得校园网信息资产管理者眼花缭乱。对复杂的校园网信息资产进行管理及运行状态的监控,是一件很繁琐的事情。目前已有的校园信息资产管理系统大部分是通过人工主动监测后手动配置、导入数据等方式进行资产的监控。这使监测具有滞后性,并不能及时、准确、宏观地反映校园网信息资产的实时状态信息。本文从Web信息大数据分析的角度出发,研究了 Web信息采集技术中的多线程爬虫技术、线程池调度技术及IP、端口、服务器等扫描探测技术。研究了信息预处理过程中基于MD5的大规模URL去重及基于Simhash的网页去重算法、基于DOM树的页面解析技术。同时研究恶意URL的检测技术及暗链、漏洞等恶意行为的特征匹配算法。将这些技术和算法运用到项目中,实现了基于Web信息大数据分析的校园网信息资产管理系统。本系统的实现分为两个层次。首先是信息采集预处理层,从一个应用层URL链接出发,通过Web网络爬虫及SNMP探测等网络扫描探测技术,获取校园网内全部的资产信息,包括Web网页信息、域名、IP、端口及服务器信息。然后通过对采集的数据进行去噪、去重等预处理,将有价值的信息资产数据进行整合存储。其次是校园网信息资产的可视化展示层,对存储的有价值的多维数据按需进行特征提取、提供信息查询和管理功能。实现了对校园网Web链接分层信息、硬件服务器信息、服务器系统信息、网站运行状态监控及校园网安全情况的统计管理,能够简洁明了、动态地监控校园网信息资产的运行状态。如监测校园网各类服务器的运行状态和使用情况、校园网的暗链、漏洞或恶意代码等安全隐患信息。在系统设计与实现过程中,首先进行系统需求分析,提炼出系统需要实现的功能。然后在需求分析的基础上,对系统进行总体架构及功能模块的设计,并对每个模块进行详细设计与实现。最后,展现系统实现的成果并进行系统测试和运行分析总结。
[Abstract]:With the rapid development of Web technology and the construction of digital campus, the existence of Web and other information resources in campus LAN has the characteristics of large quantity, large scale, complex structure, fast dynamic change and so on. Make campus network information asset managers dazzled. It is very complicated to manage and monitor the information assets of complex campus network. At present, most of the existing campus information asset management systems are manual configuration after manual active monitoring, data import and other ways to monitor assets. This makes the monitoring lag, and can not reflect the real-time state information of campus network information assets in time, accurately and macroscopically. In this paper, from the point of view of Web information big data analysis, the multi-thread crawler technology, thread pool scheduling technology, IP, port, server and other scanning detection techniques in Web information collection technology are studied. In the process of information preprocessing, the large scale URL de-reduplication algorithm based on MD5 and the web page de-reduplication algorithm based on Simhash, and the page parsing technology based on DOM tree are studied. At the same time, the detection technology of malicious URL and the feature matching algorithm of malicious behavior such as dark chain and vulnerability are studied. These techniques and algorithms are applied to the project to realize the campus network information asset management system based on the analysis of Web information big data. The realization of this system is divided into two levels. First of all, the information collection and preprocessing layer, from an application layer URL link, through the Web network crawler and SNMP detection network scanning detection technology, to obtain all the campus network assets information, including Web web page information. Domain name IP, port and server information. Then the valuable data of information assets are stored by de-noising and re-processing. Secondly, the visual display layer of campus network information assets is used to extract the features of the valuable multidimensional data stored on demand and to provide information query and management functions. The statistical management of campus network Web link layer information, hardware server information, server system information, website running state monitoring and campus network security is realized. Monitor the running status of campus network information assets dynamically. Such as monitoring the running status and usage of various servers of campus network, the hidden chain of campus network, vulnerabilities or malicious code and other security information. In the process of system design and implementation, the system needs to be analyzed firstly, and the functions needed to be realized are extracted. Then on the basis of requirement analysis, the overall architecture and functional modules of the system are designed, and each module is designed and implemented in detail. Finally, show the achievement of the system and carry on the system test and operation analysis summary.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP393.09

【相似文献】