面向大规模网络日志的主动故障检测方法的研究

发布时间：2018-05-03 16:43

本文选题：大数据 + 网络日志　；参考：《东北师范大学》2017年硕士论文

【摘要】：随着网络服务和元素的增加,其产生的网络日志被网络服务者视为监控网络健康和故障排除的重要数据源之一。在大型生产网络中,直接分析网络日志进行主动故障检查已成为一个具有挑战性的任务。这是因为以下两个原因:首先,日志的非结构化:日志消息是非结构化文本消息,不同的供应商或者不同的操作系统提供的文本消息格式不同;其次,日志的多样化:在大型生产网络中包含着各式各样的网络设备,它们所发生的网络事件会生成各种各样的日志信息。在本文中,我将通过构建两个新颖的模型一同协作完成故障的自动检测任务:基于原始日志的模板提取模型和基于日志模板的分类故障检测模型。其中,第一个模型的目标是基于聚类的思想,从非结构化的日志中直接地、自动地提取日志模板;第二个模型的目标是基于日志模板建立一个故障分类器模型,从而可以判定当前新增的日志块是否与故障有关。即本文的主动故障检测模型是将原始日志作为输入,判断日志是否与故障有关作为输出,从而快速并主动地完成检测任务,帮助网络维护者进行预防性的维护操作以及止损操作。本文首先分析出来原始日志的最小结构,然后在不需要领域知识的前提下,根据日志的模板词与参数词理论,从三个不同的角度进行日志模板提取,并对提取日志模板模型进行了优化;然后从日志模板中提取四个特征并自动表征日志模板序列的模式,采用支持向量机与高斯核函数进行监督机器学习,分析出当前状态是否可能导致故障;最后使用了实习公司中的实际生产数据,对两个模型进行优化和准确率的验证,验证了模型的实用性。
[Abstract]:With the increase of network services and elements, the network log generated by the network services is regarded as one of the important data sources for monitoring network health and troubleshooting. In large production networks, direct analysis of network logs for active fault checking has become a challenging task. This is due to two reasons: first, the unstructured nature of the log: the log message is an unstructured text message, and the format of the text message provided by different vendors or different operating systems is different; second, Log diversification: a large production network contains a wide variety of network devices, which occur network events that generate a variety of log information. In this paper, I will construct two novel models to work together to complete the automatic fault detection task: the template extraction model based on the original log and the classification fault detection model based on the log template. The goal of the first model is to extract the log template directly and automatically from the unstructured log based on the idea of clustering, and the goal of the second model is to build a fault classifier model based on the log template. This can determine whether the current new log block is related to the failure. That is, the active fault detection model in this paper takes the original log as the input, determines whether the log is related to the fault as the output, and thus completes the detection task quickly and actively. Assist network maintainers in preventive maintenance and stop loss operations. This paper first analyzes the minimum structure of the original log, and then extracts the log template from three different angles according to the theory of template words and parameter words of the log without the need of domain knowledge. The model of extracting log template is optimized, then four features are extracted from log template and the pattern of log template sequence is represented automatically. Support vector machine and Gao Si kernel function are used to supervise machine learning. Finally, the actual production data of the internship company is used to optimize and verify the accuracy of the two models, which verifies the practicability of the model.
【学位授予单位】：东北师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP393.06

【参考文献】