基于实时日志系统的海量日志服务平台设计与实现

发布时间：2018-04-13 03:25

本文选题：日志收集 + 实时流数据　；参考：《中国科学院大学(中国科学院工程管理与信息技术学院)》2017年硕士论文

【摘要】：在大数据时代,如果把数据的价值比作黄金,记录用户行为、系统运行状态的日志无疑是提炼宝藏的矿石。除此之外,日志也是开发、测试和运维人员问题调查、系统诊断的主要参考来源。然而随着业务发展,系统规模不断扩大,各种服务也越来越复杂,日志收集这个看似简单的任务也变得越来越具有挑战性。在传统的企业应用系统中,日志数据都是按文件分散存储。这样的日志已经无法满足实时的日志监测和全量的数据挖掘需求。所以,当前需要开发一个日志服务平台来对日志数据进行统一的实时收集、集中存储以及分析。本文完成的主要内容包括:(1)结合企业的实际情况,充分考虑使用便捷性、可靠性以及高效性的基础上确定了系统架构,规划了日志服务平台的框架,设计了系统功能功能模块,存储结构和用户交互界面;(2)基于Kafka消息队列,完成了日志的统一存储,实现了日志实时消费和日志投递模块,对存储方式进行了优化,提高存储资源的利用率;(3)合理设计了日志接入服务,提供了高吞吐量的日志传输管道,支持了海量的实时日志写入;(4)搭建了海量日志服务管理平台,方便用户对日志进行维护和管理。海量日志服务服务平台是针对日志类数据的一站式服务系统,用户无需开发就能快捷的完成数据采集、消费、投递及查询分析等功能,帮助提升公司运维和运营效率,赋能大数据时代的海量日志处理能力。
[Abstract]:In the era of big data, if the data value to gold, record the user behavior, the running state of the system log is undoubtedly treasure ore refining. In addition, the log is to investigate the problem of development, testing and maintenance personnel, the main source of reference for system diagnosis. However, with the development of business, the system continues to expand the scale of services the log collection is more and more complex, this seemingly simple task has become increasingly challenging. In the traditional enterprise application system, log data are stored by file. This log log data has been unable to meet the real-time monitoring and total demand of mining. Therefore, the need to develop a real-time collection the log service platform for unified log data, centralized storage and analysis. The main work of this paper include: (1) according to the actual situation of enterprises, fully consider the use of Convenience, reliability and efficiency based on system architecture determines the planning framework log service platform, the design of the system function module, storage structure and user interface; (2) Kafka message queue based on the complete unified storage log, realizes real-time log consumption and log delivery module for storage the way was optimized to improve the utilization of storage resources; (3) to design a reasonable log access service, provides a log transmission pipeline of high throughput, support real-time log massive write; (4) to build a massive log service management platform, convenient maintenance and management of the massive log. The log service platform is the one-stop service system log data, users can quickly complete without the development of data acquisition, consumption, delivery of query and other functions, help enhance the company's operation and maintenance Efficiency, the ability to deal with massive log processing in the big data age.

【学位授予单位】：中国科学院大学(中国科学院工程管理与信息技术学院)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】