基于主机的日志大数椐分析及安全性检查

发布时间：2018-03-10 05:27

本文选题：大数据　切入点：hadoop　出处：《大连理工大学》2015年硕士论文　论文类型：学位论文

【摘要】：在当今大数据及云计算时代,各种设备如手机,智能手环,智能眼镜甚至垃圾桶等等通过连接网络产生了丰富的应用,极大的方便了人们的生活。但与此同时生成大量的系统日志文件,这些日志中包含着用户登陆地点,使用设备,使用时间甚至用户当前的运动状态,血压及心跳等大量的信息。根据这些信息,能够分析出目标的行为习惯,身体状态等等。如何有效快速的处理如此大量的日志并从中挖掘出有用的信息已经成为全球学术界探索的热点之一。本文在大数据的背景下,对目前主流的日志分析及挖掘技术进行了研究。首先学习了目前常见的几种日志并行处理方法,同时深入研究目前已有的大数据处理分析方法,进而设计出一套大数据日志分析用户及系统安全性检查的解决方案。本文利用IBM智慧城市中的智慧物流系统作为日志来源,在主机上采用hadoop作为并行日志分析的计算框架,对大规模日志进行并行处理,将hadoop统计的结果作为SVM(支持向量机)的数据源,对已有数据进行分类,对用户的习惯进行分析,总结出不同用户的习惯规律,在用户产生异常行为时,系统及时发出警告,保证系统安全。同时,采用了一种改进的主动学习算法,结合支持向量机对不平衡数据进行训练和分类,该算法通过高斯核函数分析已有数据集的分布状况,从而更加智能的从已有训练集样本未覆盖的区域选择新的样本,克服了训练集样本分布过度集中,信息表达不充分的缺点。利用该算法对未登录访问请求日志进行深层分析,分类器可以通过较小的代价学习更多的样本信息。同时,对未授权的访问请求进行区分处理,单独进行主动学习,从而提高整个系统的性能和安全性。
[Abstract]:In the age of big data and cloud computing, a variety of devices, such as mobile phones, smart bracelets, smart glasses and even trash cans, have produced rich applications through the Internet. It greatly facilitates people's life. But at the same time, a large number of system log files are generated, which include user login location, use of equipment, usage time and even the user's current state of motion. A lot of information, such as blood pressure and heartbeat. Based on this information, we can analyze the behavior habits of the target. How to effectively and quickly process such a large number of logs and find useful information from them has become one of the hot spots in the global academic world. This paper studies the current mainstream log analysis and mining techniques. Firstly, we study several common parallel log processing methods, and at the same time, deeply study the existing big data processing and analysis methods. Then a set of big data log analysis user and system security check solution is designed. This paper uses the intelligent logistics system in the intelligent city of IBM as the log source, and adopts hadoop as the parallel log analysis computing framework on the host computer. The large scale log is processed in parallel, and the result of hadoop statistics is taken as the data source of SVM (support Vector Machine). The existing data are classified, the habits of users are analyzed, and the habit rules of different users are summarized. When the user produces abnormal behavior, the system sends out the warning in time to ensure the security of the system. At the same time, an improved active learning algorithm is used to train and classify the unbalanced data with support vector machine. The algorithm uses Gao Si kernel function to analyze the distribution of existing data sets, so as to select new samples more intelligently from the regions not covered by the existing training set samples, which overcomes the over-distribution of training set samples. The algorithm is used to deeply analyze the log of unlogged access request, and the classifier can learn more sample information at a lower cost. At the same time, the unauthorised access request is distinguished and processed. Independent active learning to improve the performance and security of the entire system.
【学位授予单位】：大连理工大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP309;TP311.13

【相似文献】