分布式流计算框架节点管理功能的设计与实现

发布时间：2018-02-21 01:28

本文关键词： 流计算分布式海量数据节点管理　出处：《北京邮电大学》2013年硕士论文　论文类型：学位论文

【摘要】：当今时代,信息技术的迅猛发展带来了数据量的飞速增长。这些数据不仅包含数据库中的结构化数据,还含有海量来自于电子邮件、传感器、在线视频等非结构化、互不兼容的、实时的数据。面对极大的实时数据流量,传统计算模型不再能满足需求。如何有效处理海量非结构化实时数据成为焦点,这促进了处理实时数据的分布式流计算框架的蓬勃发展。目前比较典型的分布式流计算框架有IBM的InfoSphere Steams和Yahoo!公司的S4(Distributed Stream Computing Platform)[1]系统。InfoSphere Steams是一款比较成熟的流计算产品,遗憾的是,该产品需要付费并且不开源,不能进行研究和改进。后者是Yahoo!公司开发开源分布式流计算框架,目前高速发展,是Apache旗下的子项目。 S4是一个分布式流计算平台,它有良好的可扩展性,具有部分容错能力,能够支持插件并且较为通用。S4系统有诸多优势,但仍然存在节点管理不完善,不能动态增加,删除节点的缺陷,同时S4系统管理员不能直观地使用管理界面对分布式流计算框架中各节点进行全面的管理和监控。本文改进S4在节点管理方面的不足,首先介绍流计算和S4的研究发展情况,重点描述S4在节点管理方面的不足。接下来针对S4在节点管理方面的不足,提出分布式流计算框架节点管理总体需求,包括动态管理节点需求及Web节点管理需求。之后针对总体需求,设计相关方案和架构。针对动态管理节点需求,设计并实现两阶段映射的节点管理方案；针对Web节点管理需求,提出采集层、分析整合层、表示层、用户层的分层架构,并对各层次进行设计与实现。接下来,对分布式流计算框架节点管理各项功能进行全面的功能和性能测试,验证了本文设计实现的分布式流计算框架节点管理在功能和性能方面的优势。最后对本文设计实现的分布式流计算框架节点管理提出建议和下一步研究方向。
[Abstract]:Nowadays, the rapid development of information technology has brought about the rapid growth of data volume, which contains not only structured data in databases, but also massive amounts of unstructured data from email, sensors, online video, etc. Incompatible, real-time data. In the face of huge real-time data flow, the traditional computing model can no longer meet the requirements. How to effectively deal with massive unstructured real-time data becomes the focus. This promotes the development of distributed stream computing framework for processing real-time data. At present, the typical distributed stream computing framework is IBM's InfoSphere Steams and Yahoo! The company's S4 distributed Stream Computing platform [1] system .InfoSphere Steams is a mature streaming computing product that, unfortunately, needs to be paid and is not open source and can't be researched and improved. The latter is Yahoo! The company develops open-source distributed stream computing framework, which is a sub-project of Apache. S4 is a distributed stream computing platform, which has good extensibility, partial fault tolerance, can support plug-in and has many advantages over general S4 system, but it still has imperfect node management and can not be dynamically increased. At the same time, the S4 system administrator can not directly use the management interface to manage and monitor the nodes in the distributed flow computing framework. In this paper, the shortcomings of S4 in node management are improved. Firstly, the research and development of flow calculation and S4 are introduced, and the shortcomings of S4 in node management are emphatically described. The overall requirements of node management in distributed flow computing framework, including dynamic management node requirements and Web node management requirements, are proposed, and then the related schemes and frameworks are designed for the overall requirements, and the dynamic management node requirements are addressed. Design and implement a two-stage mapping node management scheme, propose a collection layer, analyze the integration layer, presentation layer, user layer, and design and implement each layer for Web node management requirements. A comprehensive functional and performance test of the node management functions of the distributed stream computing framework is carried out. The advantages of node management in function and performance of the distributed flow computing framework designed and implemented in this paper are verified. Finally, suggestions and further research directions are given for the node management of the distributed flow computing framework designed and implemented in this paper.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.8

【相似文献】