一种正交分解大数据处理系统设计方法及实现

发布时间：2018-10-30 16:58

【摘要】：MapReduce等计算框架的出现开启了大数据处理新纪元,以Hadoop,Spark为代表的大数据处理系统具有大吞吐率、跨平台、高可扩展的优势,并得到广泛应用.然而,为避免与具体的操作系统、硬件平台绑定,这些系统的设计与优化集中在计算模型、调度算法等方面,无法充分利用底层平台的优势.提出了一种基于正交分解的大数据处理系统设计与优化方法,将系统分解为松耦合的多个功能正交的模块,使存储、处理功能分离出来,交给能够利用底层平台操作系统甚至硬件资源的存储、执行引擎,原大数据系统退化为调度平台;进而,提出基于锁无关机制的存储底层优化策略和基于指令超级优化的执行引擎底层优化策略.以此为指导,以Hadoop作为兼容和改进的对象,实现了原型大数据处理系统Arion.Arion既能保持Hadoop的跨平台、高可扩展的优势,又能消除任务执行的瓶颈,其本地化的设计与优化手段对非Hadoop平台同样有效.通过在原型系统上的实验证明,Arion能够提升大数据处理任务的执行效率,最高达7.7%.
[Abstract]:The emergence of MapReduce and other computing frameworks has opened a new era for big data processing. Big data processing system, represented by Hadoop,Spark, has the advantages of large throughput, cross-platform, high scalability, and has been widely used. However, in order to avoid binding with specific operating systems and hardware platforms, the design and optimization of these systems are focused on computing models, scheduling algorithms and so on, which can not make full use of the advantages of the underlying platform. In this paper, a design and optimization method of big data processing system based on orthogonal decomposition is proposed. The system is decomposed into loosely coupled multi-functional orthogonal modules so that storage and processing functions can be separated out. To be able to use the underlying platform operating system or even hardware resources storage, execution engine, the original big data system degraded into a scheduling platform; Furthermore, the storage optimization strategy based on lock independent mechanism and the execution engine optimization strategy based on instruction super optimization are proposed. Taking Hadoop as the compatible and improved object, the prototype big data processing system Arion.Arion can not only maintain the advantages of cross-platform, high scalability of Hadoop, but also eliminate the bottleneck of task execution. Its localization design and optimization methods are equally effective for non-Hadoop platforms. The experiments on the prototype system show that Arion can improve the efficiency of big data's task execution, up to 7. 7%.
【作者单位】：中国科学院计算技术研究所;北方工业大学计算机学院;
【基金】：国家自然科学基金项目(61202061,61202413) 中国科学院计算技术研究所创新课题项目(20146080)~~
【分类号】：TP311.13

【相似文献】