日志数据分析系统的设计与实现

发布时间：2018-05-20 00:06

本文选题：日志分析 + 仪表盘　；参考：《北京交通大学》2017年硕士论文

【摘要】：随着企业信息化的不断发展,运行的信息系统不断增加,大型企业内部部署的各种软硬件资源不断增加。在企业系统维护的过程中,数据处理的压力会越来越大,系统维护的工作量和难度不断增加。路由器,交换机,PC服务器,阿里云服务器,各种大型应用系统,数据库等计算机网络设备产生的网络日志,系统日志,各种应用程序日志不断增加。在日志分析方面,传统的技术在处理大量复杂日志时就会出现非常大的计算瓶颈。日志,作为对系统应用,网络设备,用户操作的记录工具,在保障系统安全性,排除系统故障,监控系统和网络运营情况等很多方面都起到举足轻重的作用。通过日志分析可以总结查看出系统故障出处,用户操作行为,设备运行状态等很多信息。如通过查看用户注册登录记录,用户事件记录即可发现用户是否为正常状态。对网络配置,设备参数,接口变动等行为的记录分析,就能获取网络运行状态的信息。如果能对实时产生的日志做到实时全量采集,就可以进行数据实时分析或设置数据告警来对系统运行状态进行监控。通过数据仪表盘展示分析结果,是企业极易接受的一种数据展现形式,更能清晰的体现出数据的价值。本文所描述的项目是作者在北京久其智通数据科技有限公司实习过程中参与设计开发的实际项目。本人在项目中参与了系统需求分析,设计开发及测试部署的整个过程,主要包括系统前期需求调研,日志数据接入系统、日志检索、可视化仪表盘和告警共四个模块的设计,以及除了日志检索模块之外的开发工作。实现了对接入到系统的日志数据实时采集,实时检索,定时监控告警,日志分析结果可视化等功能。本论文首先根据实际业务需求对系统进行需求分析,确定了系统的功能需求和非功能需求。在此基础上完成系统的概要设计,使用阿里云服务器作为开发的测试环境,并将系统分为检索模块,入数据到中间层模块,中间层数据接入到ElasticSearch索引模块,仪表盘模块,告警模块。完成数据库设计后,在系统详细设计与实现部分,对负责实现模块进行功能描述和类设计,并对负责的重点功能的实现逻辑及实现过程中各方法的调用关系和实际运行效果进行了详细的说明。系统上线后,针对接入系统的日志数据的实时快速检索,大大提升了运维人员的工作效率,也节省了工作人员的时间成本。通过监控告警和仪表盘的应用,使日志数据中蕴含的丰富资源得以展现,也大大提升了被监控系统和设备的运行稳定性,能够对系统中出现的问题迅速定位。
[Abstract]:With the continuous development of enterprise information, the number of information systems is increasing, and the software and hardware resources deployed in large enterprises are increasing. In the process of enterprise system maintenance, the pressure of data processing will be more and more, and the workload and difficulty of system maintenance will increase. Routers, switches, PC servers, Ali Cloud servers, a variety of large application systems, databases and other computer network equipment generated network logs, system logs, various application logs continue to increase. In the aspect of log analysis, the traditional technology will have a very large computing bottleneck when dealing with a large number of complex logs. Log, as a recording tool for system application, network equipment and user operation, plays an important role in ensuring system security, troubleshooting system, monitoring system and network operation. Through log analysis, we can summarize and view a lot of information, such as system fault source, user operation behavior, equipment running state and so on. If you view the user login record, the user event record can find out whether the user is in normal state. The information of network running state can be obtained by analyzing the behavior of network configuration, device parameters, interface change and so on. If the log generated in real time can be collected in real time, the data can be analyzed in real time or alarm can be set to monitor the running state of the system. Display the analysis result through the data dashboard, is a kind of data display form that the enterprise is easy to accept, can manifest the value of the data more clearly. The project described in this paper is a practical project in which the author participated in the design and development of the intern in Beijing Jiuqi Zhitong data Technology Co., Ltd. In the project, I participated in the whole process of system requirement analysis, design, development and test deployment, mainly including the design of four modules: system pre-requirement investigation, log data access system, log retrieval, visual dashboard and alarm. In addition to log retrieval module and the development work. The functions of real-time log data acquisition, real-time retrieval, timing monitoring and alarm monitoring, and visualization of log analysis results are realized. In this paper, the functional and non-functional requirements of the system are determined by analyzing the requirements of the system according to the actual business requirements. On this basis, the outline design of the system is completed, and the system is divided into retrieval module, data into the middle layer module, middle layer data access to ElasticSearch index module, instrument board module, using Ali cloud server as the test environment. Alarm module. After the completion of database design, in the system detailed design and implementation part, responsible for the implementation module functional description and class design, The realization logic of the key functions and the relation between the methods in the process and the actual running effect are explained in detail. After the system goes online, the real-time and fast retrieval of log data for the access system can greatly improve the efficiency of the operation and maintenance personnel, and also save the staff time cost. Through the application of monitoring alarm and dashboard, the rich resources contained in log data can be displayed, and the stability of the monitored system and equipment can be greatly improved, and the problems in the system can be quickly located.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【参考文献】