一种基于Actor模型的弹性可伸缩的流处理框架

发布时间：2018-01-22 17:37

本文关键词： 流处理 Actor模型云计算弹性可伸缩层路由机制　出处：《计算机研究与发展》2017年05期 　论文类型：期刊论文

【摘要】：流处理是一种重要的大数据应用模式,在金融、广告、物联网、社交网络等众多领域得到了广泛应用.在流处理场景中,流数据的产生速度往往变化剧烈且不容易预测.这时,如果数据流量峰值超过处理系统的承载能力,可能使得系统运行缓慢甚至崩溃,导致处理作业失效;如果为了应对数据流量峰值而过度配置资源,则可能在系统轻载时产生不必要的浪费.为了解决流处理中负载和资源的匹配问题,流处理系统应该具有弹性可伸缩的能力,一方面以高效的方式组织运算资源;另一方面能根据数据流量的实时变化自动地调整资源使用量.然而,现有的流处理框架对于弹性可伸缩的支持尚很初步.介绍了一种基于Actor模型的弹性可伸缩的流处理框架eSault.eSault首先基于Actor模型将批量的处理单元进行分层管理,通过2层路由机制实现了对伸缩性的支持;在此基础上,设计一个基于数据处理延迟的过载判断算法和基于数据处理速度的轻载判断算法来指导系统对资源的有效使用,进而实现弹性可伸缩的流处理.实验结果表明:eSault具有较好的性能,而且能够很好地实现弹性可伸缩.
[Abstract]:Stream processing is an important big data application model, which has been widely used in many fields, such as finance, advertising, Internet of things, social network and so on. In this case, if the peak value of data flow exceeds the carrying capacity of the processing system, it may make the system run slowly or even collapse, resulting in the failure of processing operations. If resources are over-configured to cope with peak data traffic, unnecessary waste may occur when the system is under light load. In order to solve the problem of load and resource matching in flow processing. The flow processing system should be flexible and scalable. On the one hand, it can organize the computing resources in an efficient way. On the other hand, the resource usage can be automatically adjusted according to the real-time change of data traffic. The support of current flow processing framework for elastic scalability is very preliminary. This paper introduces an elastic scalable flow processing framework based on Actor model eSault.eSault based on Acto at first. R model delaminate the batch processing units. Through the two-layer routing mechanism, the scalability support is realized. On this basis, we design an overload judgment algorithm based on data processing delay and a light-load judgment algorithm based on data processing speed to guide the effective use of resources. The experimental results show that: e Sault has good performance and is able to achieve elastic scalability.
【作者单位】：高可信软件技术教育部重点实验室(北京大学);北京大学(天津滨海)新一代信息技术研究院;
【基金】：国家“八六三”高技术研究发展计划基金项目(2015AA01A202) 国家“九七三”重点基础研究计划基金项目(2011CB302604) 国家自然科学基金项目(61272154,61421091) 百度云服务开放平台示范项目(2015年)~~
【分类号】：TP311.13
【正文快照】： This work was supported by the National High Technology Research and Development Program of China(863 Program)(2015AA01A202),the National Basic Research Program of China(973Program)(2011CB302604),the National Natural ScienceFoundation of China(G61272154,

【相似文献】