流式处理系统的动态数据分配技术

发布时间：2018-03-17 16:52

本文选题：数据流　切入点：流式处理　出处：《计算机工程与科学》2014年10期 　论文类型：期刊论文

【摘要】：流式数据处理中,数据倾斜等原因易导致计算节点的负载不均衡,降低系统处理能力。传统的负载均衡方法,比如算子分配、算子迁移和负载脱落等技术因为相对较高的性能代价,在流式处理系统中没有得到广泛的应用。针对流式处理系统的特点,提出一种新的负载均衡方法。在该方法中,计算单元的数据被划分为若干分区,并且数据分区可以在计算单元中动态分配和迁移,在较少干扰系统运行的情况下,通过动态调整各计算单元的分区,平衡各个计算单元的输入流和利用率,以此达到负载平衡的目的。在此基础上,设计并实现了流式处理系统的负载均衡算法和数据在线迁移技术。实验结果表明,该方法能够显著减少数据处理的平均延迟,提高系统吞吐量。
[Abstract]:In flow data processing, data skew and other reasons can easily lead to the calculation node load imbalance, reduce the system processing ability. Traditional load balancing methods, such as operator allocation, The techniques of operator migration and load shedding have not been widely used in the flow processing system because of the relatively high performance cost. A new load balancing method is proposed according to the characteristics of the flow processing system. The data of the computing unit is divided into several partitions, and the data partition can be dynamically allocated and migrated in the computing unit, and the partition of each computing unit can be dynamically adjusted with less interference with the operation of the system. In order to achieve the purpose of load balancing, the load balancing algorithm and data online migration technology of flow processing system are designed and implemented. The experimental results show that, This method can significantly reduce the average delay of data processing and improve system throughput.
【作者单位】：北京航空航天大学计算机学院;泰山学院物理与电子工程学院;
【基金】：国家973计划资助项目(2014CB340300)
【分类号】：TP338.8

【共引文献】