基于并行计算的海量日志分析系统实现

发布时间：2018-04-26 20:07

本文选题：分布式计算 + 分布式文件系统　；参考：《安徽大学》2013年硕士论文

【摘要】：日志是关于系统或者软件状态的相关记录,使用日志文件可以帮助用户更好的了解系统及软件当前的运行状态。日志文件通过记录程序在运行的状态,便于在部署之后进行系统安全的维护、系统性能的优化、系统故障的排查。传统的日志工具在技术方面存在诸多缺陷,处理效率低下,无法满足日益增长的海量数据。基于并行计算的海量日志分析系统应运而生,在深入研究不同类型日志文件特点的基础上,设计并实现了一个基于并行计算,分布式存储,实时和非实时分析的海量日志文件分析系统,具有较高的研究价值。本文首先对国内外在海量日志研究领域的研究情况和发展趋势做了充分的调研和分析,在此基础上提出了一个基于并行计算技术的海量日志文件解决方案模型。该模型采用集群方式并行的收集日志、分布式文件系统进行存储、并行计算MapReduce模型针对日志进行非实时处理、ElasticSearch模型针对日志文件进行实时检索,实现了日志采集,分析的完全自动化处理,有效的解决了传统方式下日志文件处理过程中存在的问题。系统基于Hadoop海量数据的处理模型,针对自身业务即网络会议系统的需求进行设计,搭建了通用的系统框架,同时系统对外提供的服务调用,弥补了传统日志分析技术上的缺陷,用户可以针对自身的需求进行二次开发,具有很大的灵活性,进一步贴近了实际的应用场景。基于并行计算的海量日志分析系统有效解决了海量日志数据处理过程中存在的主要问题,结合并行计算模型有效的提高了日志分析的效率,在当前数据量的爆炸式增长的大环境下具有很强的现实意义和应用价值。
[Abstract]:Log is a record of system or software status. The use of log files can help users better understand the current state of the system and software. The log file records the running state of the program, which is convenient for system security maintenance, system performance optimization and system fault checking after deployment. The traditional logging tools have many technical defects, and the processing efficiency is low, which can not meet the increasing amount of data. The massive log analysis system based on parallel computing emerges as the times require. On the basis of deeply studying the characteristics of different types of log files, a distributed storage system based on parallel computing is designed and implemented. Real-time and non-real-time analysis of massive log file analysis system has high research value. Firstly, this paper makes a full investigation and analysis of the research situation and development trend in the field of mass log research at home and abroad, and then puts forward a solution model of massive log file based on parallel computing technology. This model uses the cluster mode to collect logs in parallel, distributed file system to store, parallel computing MapReduce model to carry out non-real-time processing of logs Elastic search model for log files real-time retrieval, achieve log collection. The fully automated processing of analysis effectively solves the problems existing in the process of log file processing in the traditional way. The system is based on the processing model of Hadoop magnanimity data, aiming at the demand of its own business, that is, the network conference system, and builds a general system framework, and at the same time, the system provides the service call to the outside, which makes up for the shortcomings of the traditional log analysis technology. Users can develop second time according to their own needs, which has great flexibility and closer to the actual application scenario. The magnanimous log analysis system based on parallel computing effectively solves the main problems in the process of processing massive log data, and improves the efficiency of log analysis effectively by combining the parallel computing model. It has strong practical significance and application value under the circumstance of explosive growth of data volume.
【学位授予单位】：安徽大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.6

【参考文献】