基于Mahout分类器的异常流量检测技术的研究

发布时间：2018-06-16 15:16

本文选题：大数据分布式 + 网络异常流量　；参考：《天津理工大学》2017年硕士论文

【摘要】：近年来,互联网带给人们丰富的共享信息资源,方便了人们的工作与生活,人们越来越离不开网络,网络在现代人们的生活中扮演者越来越重要的角色。另一方面,很多人针对网络进行恶意攻击,从中获取利益。对于大部分普通网民来说,网络环境日益复杂,人们不仅需要加强自身保护意识,更加需要网络安全人员对网络流量进行维护与监管,检测异常网络流量,从而保证网络使用者的上网安全。因此,网络异常流量检测技术具有重大意义,也是文本的主要研究课题。本文针对网络中数据源多维化的特点,将数据源多维数据的信息熵投影到不同的分类支撑向量,由于机器学习容易产生过度训练或训练不足这些问题,本文对比EWMA、Entropy、K-means、GMM、SVDD等异常流量检测方法,选择支持多维数据统计的非监督式SVDD分类方法,训练可采用的数据集。针对训练分类器成本消耗大与分类检测效果佳的冲突问题,本文选择一种异常实时响应训练的学习模式,只在有异常检测点加入时,才进行重新训练。不仅如此,对于训练数据集中异常点的选取,采用贝叶斯网络模型推理预测下一节点的异常概率,随即判断是否将异常点加入到训练集中继续训练,以此整体提高异常流量检测的效率与精度。本文主要针对的问题以及研究的创新点如下:(1)针对研究过程中,异常数据集来源困难、数据分析过程复杂的客观条件,本文选择大数据分布式平台环境进行数据的处理分析。为分析分类器异常检测能力的表现效果,本文对比EWMA、Entropy、K-means、GMM、SVDD等主流的异常检测方法,最终实验选定在多维信息熵构建支撑向量基础上,由SVDD方法对异常流量进行检测,能够有优于其他几种方法的检测效果。(2)针对分类器数据量、训练地与实际应用地点的不同,原始的训练集检测效果不能很好地检测新数据,且当前重新训练所需要的资源消耗量大等等这些问题,本文采用异常实时响应训练方法。只有在加入异常点时,才进行训练集的重训练,将检测窗口平行移动至异常点,剔除最开始的检测点,增加当前适用的检测点。这种方法既能提高训练集的适应性与准确率,还能将物质消耗,资源消耗降低。(3)针对检测统计量时间相关性、异常样本对精度影响的问题,本文采用贝叶斯网络模型推理预测该异常节点可能会出现异常的概率,对检测精度进行优化,重新组织加入训练集的异常点。针对异常检测模型检测出的异常,提出应急响应方法,不仅能够检测异常,更加能够处理异常。
[Abstract]:In recent years, the Internet has brought people a rich share of information resources, convenient for people's work and life, people are becoming more and more inseparable from the network, the network plays a more and more important role in the life of modern people. On the other hand, a lot of people have malicious attacks on the network to gain benefits. For most ordinary netizens, The network environment is increasingly complex, people not only need to strengthen their own awareness of protection, more need network security personnel to maintain and supervise network traffic, detect abnormal network traffic, so as to ensure the network users' Internet security. Therefore, the network anomaly traffic detection technology is of great significance and is also the main research topic of the text. In view of the multi-dimensional characteristics of data sources in the network, the information entropy of data source multidimensional data is projected to different classification support vectors. Because machine learning is easy to produce excessive training or lack of training, this paper compares the abnormal flow detection methods such as EWMA, Entropy, K-means, GMM, SVDD and so on, and chooses the non supervision of multi-dimensional data statistics. SVDD classification method is used to train data set which can be used. Aiming at the conflict between the high cost of training classifier and the better effect of classification detection, this paper selects a learning model of abnormal real-time response training. It is only retrained when the exception detection point is added, not only for the selection of abnormal points in the training data set, but also for the selection of abnormal points in the training data set. The Bayesian network model is used to predict the abnormal probability of the next node, and then to judge whether the anomaly points are added to the training set to continue training to improve the efficiency and accuracy of the anomaly traffic detection. The main problems and the innovation points in this paper are as follows: (1) in the process of research, the number of abnormal data sets is difficult and the number of data is difficult. According to the analysis of the complex objective conditions, this paper selects the large data distributed platform environment for data processing and analysis. In order to analyze the performance effect of the classifier anomaly detection ability, this paper compares the EWMA, Entropy, K-means, GMM, SVDD and other mainstream anomaly detection methods. The final test is selected on the basis of the multidimensional information entropy construction support vector, and the S is based on the support vector. The VDD method is better than the other several methods to detect the abnormal traffic. (2) in view of the classifier's data quantity, the training ground is different from the actual application location, the original training set detection effect can not detect the new data well, and the current retraining needs a large amount of resource consumption and so on. The training method of abnormal real-time response. Only when the exception point is added, the training set is carried out, the detection window is moved parallel to the exception point, and the first detection point is eliminated and the current detection point is increased. This method can not only improve the adaptability and accuracy of the training set, but also reduce the consumption of material and the consumption of resources. (3) (3) In this paper, we use Bayesian network model to predict the abnormal probability of the anomaly node, optimize the detection precision and reorganize the exception points of the training set. The emergency response method is proposed for the anomaly detected by the anomaly detection model. Only to be able to detect exceptions and to be able to handle exceptions more.
【学位授予单位】：天津理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.06

【相似文献】