分布式数据流上的高性能分发策略

发布时间：2018-10-09 21:09

【摘要】：随着大数据应用的普及,高效可扩展的数据流操作在实时分析处理中扮演着越来越重要的角色.分布式并行处理架构是应对大流量、低延时数据流处理任务的一种有效解决方案.然而在Key-based分组并行处理中,由于数据的倾斜分布及数据流本身的实时、动态和数据规模不可预知等特性,使得数据流分布并行处理系统存在持续且动态负载不均衡现象,这会造成系统时效性降低、硬件资源浪费等问题.现有的研究工作处理均衡负载有两种方案:(1)基于key粒度的迁移,使得并行处理节点负载达到均衡;(2)基于元组粒度级别的拆分,采用随机分发使系统均衡.前者将系统调整至给定的均衡容忍范围内,类似于一维装箱的NP问题;后者对key的拆分势必带来新的为维护Key-based操作的正确性而增加的额外代价,如内存及网络通信成本.综合两种方法,提出对key按需拆分、尽量合并的方法,通过轻量级均衡调整算法以及保证Key-based操作特性的拆分方法,使系统既能达到后者的均衡,又能减少细粒度均衡所带来的额外代价.
[Abstract]:With the popularity of big data application, efficient scalable data stream operations play an increasingly important role in real-time analysis and processing. Distributed parallel processing architecture is an effective solution to deal with large traffic and low delay data flow. However, in Key-based packet parallel processing, because of the skew distribution of data and the real-time, dynamic and unpredictable data flow, the distributed parallel processing system of data stream has the phenomenon of persistent and dynamic load imbalance. This will cause the system to reduce timeliness, hardware resources waste and so on. There are two schemes for balancing load in existing research work: (1) the parallel processing node load is balanced based on the migration of key granularity; (2) based on the partitioning of tuple granularity, the system is balanced by random distribution. The former adjusts the system to a given equilibrium tolerance, which is similar to the NP problem of one-dimensional packing, while the latter brings additional cost to maintain the correctness of Key-based operations, such as memory and network communication costs. By synthesizing the two methods, this paper puts forward the method of splitting key according to demand and combining it as far as possible. By means of lightweight equalization adjustment algorithm and the split method of ensuring the operation characteristics of Key-based, the system can achieve the balance of the latter. It can also reduce the extra cost of fine-grained equilibrium.
【作者单位】：华东师范大学计算机科学与软件工程学院;上海市高可信计算重点实验室(华东师范大学);
【基金】：国家高技术研究发展计划(863)(2015AA015307) 国家自然科学基金(61232002,61332006,61572194)~~
【分类号】：TP311.13

【相似文献】