分布式流处理系统操作共享优化算法研究

发布时间：2018-04-08 07:12

本文选题：流数据　切入点：分布式流处理　出处：《吉林大学》2016年硕士论文

【摘要】：随着社交网络、物联网和无线通信网的出现以及快速发展,流数据成为一种重要的大数据类型,在股票交易、电子商务、社交网络和监控等领域应用广泛。流数据相较于普通的大数据,具有持续性、动态性和实时性的特点,对流处理提出新的要求:(1)数据持续到达需要在线处理。流处理作业长时间运行,采用One Pass处理模式,无法留存数据,仅保存中间结果;(2)数据在线变化要求作业拓扑结构动态调整;(3)需要实时或近实时的处理。现有相关工作针对流处理的系统结构和相关技术开展了大量研究,形成分布式流处理技术,实现了大规模流的高速处理。但对分布式流处理系统资源分配问题的研究存在不足,未能充分利用流处理作业间操作可共享的特点,导致数据处理冗余,形成资源浪费。流作业是典型的工作流模式,由多个数据处理操作组成。多个流作业间的基本数据处理操作有重叠,例如filter,join等。因此多个作业间存在操作共享的可能。目前,针对操作共享的相关研究主要从作业拓扑结构角度考虑共享的可能性,建立操作共享方案。对共享操作资源分配问题的研究并不充分,主要体现在:(1)缺乏系统模型,描述流处理作业操作共享过程中的资源约束变化;(2)对流处理作业操作共享后的输出带宽资源分配问题研究不足;(3)流数据动态变化导致作业拓扑结构变化,需要进一步研究资源分配方案快速调整问题。本文提出支持分布式流处理的操作共享优化算法,主要工作如下:(1)建立流处理作业操作共享模型。支持不同作业之间相同操作的逻辑共享,满足不同的资源约束条件,包括计算资源和物理网络带宽资源。(2)研究多资源约束下的资源分配问题。利用作业操作间的相似性,提出共享判定算法,找到逻辑上可共享的操作。将资源分配过程抽象成一个装箱问题,考虑操作共享后的资源使用情况,与不共享的情形相比,共享操作占有的计算资源合并,输出带宽累加。根据资源分配算法,在最小数目的计算节点上完成所有操作的共享任务,保证正常的数据传输。(3)研究快速资源映射调整问题。新作业的动态加入导致整体拓扑结构改变,共享操作的资源需求在线变化,根据当前的资源配置对共享的操作进行筛选,选择共享受益最大的操作,使用较少的调整消耗,快速地满足新的资源需求,保证流处理实时性和动态性。(4)建立模拟实验平台。验证作业操作共享能够有效节省计算资源,并且满足不同系统资源对共享操作数目的约束;验证操作共享算法的有效性,快速调整算法优于资源重新分配算法。
[Abstract]:With social networking, networking and wireless communication networks and the rapid development of data stream is becoming a important data type, the electronic commerce in the stock exchange, and social networking applications and monitoring areas widely. Compared to the common data stream data, with a persistent, dynamic and real-time. The convection process put forward new requirements: (1) continuous arrival of data need to be handled online. Flow operations for a long time operation, using One Pass model, can not only save the data retention, intermediate results; (2) online data changes require dynamic topology adjustment operations; (3) require real-time or near real-time processing system. The structure and related technology of existing related work aiming at cluster processing carried out extensive research, the formation of distributed stream processing technology, to achieve high-speed processing flow. But for large-scale distributed stream processing system of resource allocation problems. The problems, make full use of stream processing operations between operating characteristics can be shared to lead to redundant data processing, the formation of a waste of resources. The flow assignment is typical workflow model, composed of a plurality of data processing operations. A basic data flow between activity processing operations overlap, such as filter, join and so on. So many there may be sharing operation. At present, the related research on operation sharing mainly consider the possibility of shared operation from the perspective of topological structure, establish the operation scheme. To study the operating resource allocation problem sharing is not sufficient, mainly reflected in: (1) the lack of system model, describes the stream processing operations change shared resource constraints in the process of research; (2) processing operations after sharing the distribution output bandwidth of convection is insufficient; (3) data flow dynamic changes leading to topology changes, the need for further The rapid adjustment of resource allocation problems. This paper supports distributed stream processing sharing optimization algorithm, the main work is as follows: (1) the establishment of stream processing operations. Support sharing model between different operations of the same operation logic sharing, meet different resource constraints, including computing resources and physical network bandwidth (2. Study) resource allocation problem under multiple resource constraints. The similarity between the proposed operation, sharing decision algorithm, find a logical sharing operation. Resource allocation process is abstracted into a packing problem, considering the sharing of resource usage, compared with no sharing, sharing operation of possession the computing resources combined, cumulative output bandwidth. According to the resource allocation algorithm to complete all operations of the shared task in the calculation of the minimum number of nodes, ensure the normal data transmission. (3). The rapid adjustment of dynamic resource mapping. The new operation leads to change the overall topology, sharing resource demand changes according to the online operation, the current resource allocation of the shared operation screen, choose to share the biggest benefit of the operation, use less adjustment of consumption, to meet the new demand for resources, ensure the real-time stream processing anddynamics. (4) the establishment of the simulation platform. Verify operation sharing can effectively save computing resources, and to satisfy the constraints of different system resources sharing operation number; validation operation sharing algorithm, fast adjustment algorithm is better than the resource re allocation algorithm.

【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP301.6

【相似文献】