实时数据处理技术在网络安全管理系统中的研究与应用

发布时间：2018-10-14 13:11

【摘要】：随着现代通信技术的不断发展与进步,网络得到了快速的普及与推广,已经成为人们工作和生活中不可或缺的一部分。在网络蓬勃发展的背景下,网络安全事件却频繁发生,引起了广泛的关注。面对这种局面,网络安全管理人员开始在网络中大量使用网络安全设备,并通过网络安全管理系统对设备进行统一管理。然而,网络安全设备功能各异,各司其职,产生了大量信息。这些数目庞大的信息不仅难以利用,还给网络安全管理系统造成了巨大的压力。因此,对海量信息进行实时处理成为了网络安全管理系统的发展趋势。为了对网络中大量的实时信息进行处理与分析,本文将当前应用较为广泛的Storm实时数据处理技术引入网络安全管理系统,重新设计出基于Storm的事件管理模块,力图通过Storm提高系统对实时数据的处理效率。本文的主要工作和成果如下:1.本文将安管系统事件管理模块的数据处理过程划分为5个子过程,分别是事件采集、事件标准化、事件预处理、事件高级处理和事件展示。本文在Storm实时数据处理技术的基础上,重新对事件管理的5个子过程进行了设计与实现,将实际需求与流式数据处理相结合。通过测试,基于Storm的事件处理逻辑较单机Java程序而言在数据处理速度上有了显著的提升。2.在基于Storm对事件预处理进行设计的过程中,针对关联事件较为冗余的问题,本文利用Storm中Bolt允许缓存的机制,采用基于特征字段分类的去重方法,发现数据之间的关联关系,将系统中拥有相同特征的事件合并,有效地清除了系统的冗余数据。3.针对Storm流式数据处理的特点,本文对事件高级处理的事件归并过程进行了改造,将统计归并事件数量的工作交给事件展示逻辑,从而保证了数据的流式处理。而且,由于在事件规则匹配中程序会频繁查询事件规则,本文将事件规则存储在Redis内存数据库,提高事件规则的查询效率。
[Abstract]:With the continuous development and progress of modern communication technology, the network has been rapidly popularized and popularized, and has become an indispensable part of people's work and life. In the background of the vigorous development of network, network security events occur frequently, which has attracted wide attention. Faced with this situation, network security managers began to use a large number of network security equipment in the network, and through the network security management system to manage the equipment unified. However, network security equipment has different functions and functions, and produces a lot of information. This huge amount of information is not only difficult to use, but also a huge pressure on network security management system. Therefore, real-time processing of mass information has become the development trend of network security management system. In order to process and analyze a lot of real-time information in the network, this paper introduces the Storm real-time data processing technology, which is widely used in the network, into the network security management system, and redesigns the event management module based on Storm. This paper tries to improve the efficiency of real-time data processing by Storm. The main work and results of this paper are as follows: 1. In this paper, the data processing process of the event management module is divided into five sub-processes, namely, event collection, event standardization, event preprocessing, event advanced processing and event display. Based on the Storm real-time data processing technology, this paper redesigns and implements the five sub-processes of event management, and combines the actual requirements with the streaming data processing. Through the test, the event processing logic based on Storm has significantly improved the speed of data processing compared with the single Java program. 2. 2. In the process of designing event preprocessing based on Storm, aiming at the problem of redundancy of associated events, this paper uses the mechanism of Bolt in Storm to allow cache, and adopts the method of de-reduplication based on the classification of feature fields to find the correlation relationship between data. Merge the events with the same characteristics in the system, effectively clear the redundant data of the system. 3. According to the characteristics of Storm streaming data processing, the process of event merging in advanced event processing is reformed in this paper, and the work of counting the number of merged events is handed over to the event display logic, thus ensuring the flow processing of data. Furthermore, event rules are frequently queried in the event rule matching, so the event rules are stored in the Redis memory database to improve the query efficiency of the event rules.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP393.08;TP315

【相似文献】