基于可视化技术的分布式系统监控数据分析

发布时间：2018-10-09 14:04

【摘要】：分布式技术是目前互联网企业搭建平台不可或缺的技术之一,庞大的分布式系统往往每天都会产生大量的数据。分布式系统监控数据的可视化可以帮助运维工程师更好地了解网络结点的状态,挖掘出监控数据中所蕴含的信息,更迅速准确地制定运维策略。因此,如何将可视化技术应用于分布式系统对互联网领域有着重要意义和价值。本论文针对如何将可视化技术应用于分布式系统的监控数据进行了研究,解决了监控数据的采集,数据处理和映射,可视化展示,监控数据分析等问题,搭建了一个完整的数据可视化系统。本文选用Logstash,Redis,Elasticsearch,OpenTSDB等技术实现数据采集和存储,采用D3框架实现ThemeRiver,Event Timeline,Circos,Stack等数据可视化模块,并载入真实数据来进行可视化分析实践,以验证本文所实现的可视化模块的可行性和有效性。论文的主要研究工作有:1.监控数据的采集和存储。针对目前市面上存在的服务器监控技术进行了对比和分析,从原理,适应性,实现优化等角度考虑选取合适的技术来采集和存储结点监控数据,为可视化提供数据来源。2.可视化模型的设计和实现。本文针对网络结点数据可视化的需求进行了调研和分析,选取负载数据,网络通信量数据,日志数据作为可视化数据,设计并实现了ThemeRiver,Event Timeline,Circos,Stack等数据可视化模块,并添加简单的交互来增强可视化模型的实用性。3.基于可视化模型的监控数据分析实践。载入真实数据来进行可视化分析实践,通过分析监控数据来验证本文所实现的可视化模型的有效性和可行性。将可视化技术应用于分布式系统监控数据的分析,可以帮助运维人员尽早发现网络结点中潜在的问题,及时调整结点状态,预防服务器奔溃等灾难性事故的发生,为制定有效的运维策略提供参考。
[Abstract]:Distributed technology is one of the indispensable technologies for Internet enterprises to build platforms. Large distributed systems often produce a large amount of data every day. The visualization of monitoring data in distributed systems can help operational engineers better understand the status of network nodes, dig out the information contained in monitoring data, and make operational and maintenance policies more quickly and accurately. Therefore, how to apply visualization technology to distributed systems is of great significance and value to the field of Internet. This paper studies how to apply the visualization technology to the monitoring data of distributed system, and solves the problems of data acquisition, data processing and mapping, visualization display, monitoring data analysis and so on. A complete data visualization system is built. In this paper, Logstash,Redis,Elasticsearch,OpenTSDB and other technologies are used to realize data acquisition and storage, D3 framework is used to realize ThemeRiver,Event Timeline,Circos,Stack and other data visualization modules, and real data is loaded to carry out visualization analysis practice, to verify the feasibility and effectiveness of the visualization module implemented in this paper. The main research work of this paper is 1: 1. Monitor data acquisition and storage. This paper compares and analyzes the existing server monitoring technology in the market, and considers the appropriate technology to collect and store node monitoring data from the aspects of principle, adaptability, optimization and so on, so as to provide the data source for visualization. Design and implementation of visual model. This paper investigates and analyzes the demand of network node data visualization, selects load data, network traffic data and log data as visualization data, and designs and implements data visualization module such as ThemeRiver,Event Timeline,Circos,Stack. And add simple interaction to enhance the usability of the visual model. 3. Practice of monitoring data analysis based on visual model. The real data is loaded to carry on the visualization analysis practice, and the validity and feasibility of the visualization model realized in this paper are verified by analyzing the monitoring data. The application of visualization technology to the analysis of monitoring data in distributed systems can help operators discover potential problems in network nodes as early as possible, adjust the status of nodes in time, and prevent catastrophic accidents such as the collapse of servers. It provides a reference for the formulation of effective operation and maintenance strategy.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP277

【相似文献】