基于机器学习的EMC存储设备日志分析与Bug预测技术研究

发布时间：2018-05-04 08:11

本文选题：机器学习 + 日志分析　；参考：《南京大学》2014年硕士论文

【摘要】：EMC是一家以存储设备、服务和解决方案为主营业务的信息存储咨询科技公司,随着公司业务的发展,越来越多的企业或个人选择购买EMC的存储设备来进行数据存储和备份。存储设备每天将运行日志上传到EMC数据中心,一旦设备发生异常,这些日志就是支持工程师用来维护和定位用户设备中Bug的唯一途径。而随着日志文件数量的不断增加,大大增加了支持工程师的人工维护成本和Bug定位的难度,并且设备Bug的产生也大大影响了用户对EMC产品的满意度。由于日志反映了用户机器设备基本信息和每天的运行情况,在一个Bug发生之前,一定可以在日志中找到相关数据的展示或波动预示Bug的产生。为此,论文项目提出了基于机器学习的日志分析和Bug预测,使用机器学习的方法找出Bug产生的规律来主动发现用户机器设备中存在的问题,帮助支持工程师及时发现并解决Bug,提高处理效率并且减少设备故障率。论文选定有关存储磁盘的Bug作为研究对象,分成两个部分完成对存储磁盘Bug的预测。第一个部分进行日志内容的提取和存储,把日志中有关于存储段落的数据使用相关的正则表达式进行匹配和提取,存储到Greenplum数据库中；第二个部分对提取出的日志数据使用Weka进行机器学习,建立Bug预测模型。论文选择Weka中决策树和贝叶斯的相关算法对数据进行学习建模,对比不同模型的优劣,最终选定一种算法建立预测模型对存储磁盘的Bug做出预测。论文主贡献包括以下几点：(1)提出了用于日志提取和Bug预测的解决方案,对存储磁盘Bug相关的日志提取和预测需求进行准确的描述和定义。(2)设计并实现了对日志内容进行提取和存储的框架,可以支持对日志内容根据需求进行提取和存储的扩展。(3)使用Weka进行机器学习模型的训练,最终选定贝叶斯网络建立Bug预测模型,达到88%的预测正确率和低于13%的漏报率。
[Abstract]:EMC is an information storage consulting technology company with storage devices, services and solutions as its main business. With the development of its business, more and more enterprises or individuals choose to purchase EMC storage devices for data storage and backup. Storage devices upload daily running logs to the EMC data center, which is the only way to support engineers to maintain and locate Bug in user devices in the event of device exceptions. With the increase of the number of log files, the cost of supporting engineers' manual maintenance and the difficulty of Bug positioning are greatly increased, and the production of equipment Bug greatly affects the user's satisfaction with EMC products. Because the log reflects the basic information of the user's machine and equipment and the daily running condition, before a Bug occurs, the display or fluctuation of the relevant data can be found in the log to predict the production of the Bug. For this reason, the paper puts forward log analysis and Bug prediction based on machine learning, using machine learning method to find out the rule of Bug production to actively discover the problems existing in user's machine equipment. Help support engineer to detect and solve Bug in time, improve handling efficiency and reduce equipment failure rate. In this paper, the Bug of storage disk is selected as the research object, and the prediction of storage disk Bug is completed in two parts. In the first part, the contents of the log are extracted and stored, and the data about the stored paragraphs are matched and extracted by using the relevant regular expressions in the log, and stored in the Greenplum database. In the second part, Weka is used to learn the extracted log data, and the Bug prediction model is established. This paper selects the decision tree and Bayesian algorithms in Weka to model the data, compares the advantages and disadvantages of different models, and finally selects an algorithm to build a prediction model to predict the Bug of the storage disk. The main contributions of this paper are as follows: 1) A solution for log extraction and Bug prediction is proposed. The log extraction and prediction requirements related to storage disk Bug are described and defined accurately. 2) the framework of extracting and storing log contents is designed and implemented. It can support the extension of extracting and storing log content according to the requirement. It can use Weka to train the machine learning model. Finally, Bayesian network is selected to establish the Bug prediction model, and the prediction accuracy is 88% and the false report rate is less than 13%.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP181;TP333

【相似文献】