内存集群计算:交互式数据分析

发布时间：2018-11-17 10:05

【摘要】：本文围绕大数据分类中决策数据的管理和分析进行展开.重点分析了大数据时代关于商务智能(Business Intelligence,BI)技术新的应用需求;讨论了计算机硬件和体系结构的发展为决策数据管理和分析带来的挑战和机遇;通过对新兴典型应用的分析和相关技术和系统特点的总结,说明了基于内存计算的高性能数据管理和分析技术是当前亟待解决的问题,具有广阔的应用前景.在全内存式(in-memory)数据管理环境下,网络通讯将成为整个系统的主要瓶颈.结合内存的特点(数据易失性、内存墙瓶颈),设计针对高性能服务器的无共享分布式内存系统拓扑结构;研究面向异构、多层次缓存和内存结构的分布式数据布局与索引策略,跨核、跨处理器、跨服务器的多粒度并行处理框架,缓存感知、内存感知的分布式数据一致性维护等关键技术,轻量级面向按列存储的数据压缩机制及压缩感知的数据处理机制,将是基于内存计算的高性能数据管理与分析技术的重点研究内容,并将最终实现实时交互式分析处理.
[Abstract]:This paper focuses on the management and analysis of decision data in big data classification. This paper analyzes the new application requirements of business intelligence (Business Intelligence,BI) technology in big data's time, discusses the challenges and opportunities brought by the development of computer hardware and architecture for decision data management and analysis. By analyzing the typical emerging applications and summarizing the characteristics of related technologies and systems, it is shown that the high performance data management and analysis technology based on memory computing is an urgent problem to be solved and has a broad application prospect. In the in-memory data management environment, network communication will become the main bottleneck of the whole system. Combined with the characteristics of memory (data volatility, memory wall bottleneck), the topology of non-shared distributed memory system for high performance server is designed. This paper studies distributed data layout and index strategy for heterogeneous, multi-level cache and memory structure, multi-granularity parallel processing framework for cross-core, cross-processor and cross-server, cache awareness. Key technologies such as memory aware distributed data consistency maintenance, lightweight data compression mechanism for column storage and data processing mechanism for compression awareness, It will be the key research content of high performance data management and analysis technology based on memory computing, and will finally realize real-time interactive analysis and processing.
【作者单位】：电子科技集团第三十二研究所;华东师范大学数据科学与工程研究院;
【基金】：国家自然科学重点项目(61332006)
【分类号】：TP333

【共引文献】