基于大数据分析的网络攻击场景重建系统

发布时间：2019-01-11 09:45

【摘要】：网络攻击事件的频繁发生和入侵手段的多样化,使得入侵检测系统(IDS)每天产生大量误报和冗余的告警信息,是网络安全管理工作的一大难题。研究者们发现通过对孤立的告警信息进行关联分析,可以还原攻击事件的发生过程,帮助管理员发现网络的薄弱环节。本文使用Spark集群和Hadoop生态圈的关键技术,完成了从数据的采集与存储,数据传输到计算和展示的完整架构,能有效应对告警数据海量化带来的问题。在此大数据处理架构上,本文运用告警关联方法对网络告警日志进行分析,设计并实现了基于大数据分析的攻击场景重建系统。本文重点研究因果关联方法和概率关联方法的算法流程,并在大数据平台上实现。对于因果关联方法,本文加入基于特征重叠的告警去冗余模块,可以有效去除海量告警日志中重复和冗余的信息。概率关联方法的核心是定义了IP地址、端口和告警类别等属性相似度计算函数,并定义各属性最小相似度期望值和权重,计算告警整体相似度。本文将整个攻击场景重建模型拆分成三个独立的模块予以处理,包括去冗余模块用于去除重复告警,聚合模块用于聚合同一攻击步骤产生的告警,关联模块用于将告警聚合的结果关联形成多步攻击场景。为了验证系统的有效性和实用性,本文使用天津理工大学IDS数据集和DARPA数据集进行实验,结果证明采用本文方法比PengNing的纯因果关联方法更能直观的展现网络攻击步骤,能大量减少IDS的重复告警信息,准确的聚合和关联相似告警信息,有效提高了IDS系统的可用性。
[Abstract]:The frequent occurrence of network attack events and the diversification of intrusion means make intrusion detection system (IDS) produce a large number of false positives and redundant alarm information every day, which is a major problem in network security management. The researchers found that by associating the isolated alarm information, the attack event can be restored and the administrator can find the weak link of the network. Using the key technologies of Spark cluster and Hadoop ecosystem, this paper completes a complete framework from data acquisition and storage, data transmission to calculation and display, which can effectively deal with the problems caused by the sea quantization of alarm data. In this framework of big data, this paper uses the method of alarm correlation to analyze the network alarm log, and designs and implements an attack scene reconstruction system based on big data analysis. This paper focuses on the algorithm flow of causality correlation and probabilistic correlation, which is implemented on big data platform. For causality correlation method, the redundant module based on feature overlap is added to remove redundant and redundant information in massive alarm logs. The core of the probabilistic association method is to define the IP address, port and alarm class and other attribute similarity calculation functions, and define the minimum similarity expectation and weight of each attribute, and calculate the overall similarity of alarm. In this paper, the whole attack scene reconstruction model is divided into three independent modules, including the deredundancy module used to remove the repeated alarm, the aggregation module used to aggregate the alarm generated by the same attack step. The association module is used to correlate the result of alarm aggregation into a multi-step attack scenario. In order to verify the effectiveness and practicability of the system, this paper uses the IDS data set and DARPA dataset of Tianjin University of Science and Technology for experiments. The results show that the proposed method can show the network attack steps more intuitively than the pure causality correlation method of PengNing. It can greatly reduce the repeated alarm information of IDS, accurately aggregate and correlate similar alarm information, and effectively improve the usability of IDS system.
【学位授予单位】：天津理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.08;TP311.13

【相似文献】