赛博—物理空间中数据分布式聚合与大规模处理关键技术研究

发布时间：2018-01-10 09:34

本文关键词：赛博—物理空间中数据分布式聚合与大规模处理关键技术研究　出处：《国防科学技术大学》2016年博士论文　论文类型：学位论文

【摘要】：赛博-物理空间作为信息空间和物理空间的融合,可以使得具有信息感知、信息交换以及计算能力的实体根据应用需求互相关联。这种实体间的关联关系涵盖了从物理域、信息域到社会域多维度的信息,具有大规模、分布、动态等多种特性。在这样一种融合空间中开展、开发应用越来越成为科学工作者关注的话题。在赛博-物理空间中,从信息感知的角度来讲,需要面对大规模实体信息及实体间关系的分布式发现及聚合的挑战。这里实体间关系包括了根据应用需求从不同维度进行描述的信息,例如实体在物理空间中的相对位置关系、由和用户相关的社会属性所表达的逻辑关系等。而从信息聚合的角度来讲,需要面对实体及实体间关系动态性所带来的挑战。这里动态性包括实体状态变化、实体位置变化等引起的实体属性值的动态性,以及由此引起的实体间关联关系的动态性等。而从信息处理的角度来讲,需要面对大规模异构数据高效处理带来的挑战,包括特定类型大规模数据处理的设计,以及多样化业务模式对大规模数据处理平台需求方面的设计等。本文从赛博-物理空间应用中如何进行有效的信息聚合和高效的大规模数据处理两个角度展开,研究如何高效地以分布式方式对实体间关系的发现,面向赛博-物理空间中实体及实体间的动态性,研究如何满足应用对实体稳定关系发现的需求,即如何有效构建稳定子网络,针对数据处理需求如何进行特定类型数据处理模型的设计,以及面向多样化业务模式如何进行数据处理平台融合和任务调度优化。本文工作主要分为以下四个方面:1.基于分布式聚合模型的低负载实体关系发现机制:通过建立赛博-物理空间中分布式的关系发现模型,考虑到多重关系同时发现的需求,设计了基于生成树的分布式关系发现算法MSTRD,使得赛博-物理空间中实体间关系从传统的集中式发现转变为分布式发现,同时保证关系发现过程的高效性,即降低关系发现过程的数据传输量和时间耗费。实验结果表明MSTRD方法不管是从关系发现过程中数据传输总量还是时间耗费上来讲都比集中式的处理方式性能更优,最大数据传输量可节省1/3左右,同时在极端情况下可节省近一半的时间耗费。2.面向分布式动态聚合网络的稳定子网构建机制:通过建立面向赛博-物理空间中应用的分布式实体模型,设计了分布式的实体集合选择算法MRMC,通过该分布式实体集合选择算法,可以在实体集合规模增长有限的情况下提高实体间关系保持不变的时间RUT,即稳定子网络。大量的模拟实验对MRMC算法进行了正确性和有效性验证,结果表明,MRMC算法可以对实体间关系保持不变时间RUT的提升最大达一倍左右。3.基于矩阵抽象的赛博-物理空间大规模数据处理框架:本文设计了面向矩阵运算的通用计算框架Matrix Map,构造了横跨内存和外存的可扩展的数据结构块式键值矩阵BKM,设计了针对算法并行性的数据重组和异步计算的优化策略,以及针对通用矩阵运算的一元、二元操作符。实验结果表明,针对一些常见的图处理算法,Matrix Map比Spark等流行系统在处理速度上最多可以快10倍左右。4.面向赛博-物理空间应用多样化业务模式的大数据平台融合技术:针对大数据处理平台的多样性和繁杂性,通过分析当前主流大数据处理平台的优劣,提出了对不同大数据处理平台进行融合的方案,设计了针对不同数据处理任务在融合系统内的调度框架,设计了基于遗传算法的任务分簇和调度优化策略MMRC,进而提高数据处理效率、降低资源耗费。模拟实验结果表明,和单一系统处理相比,MMRC方法可以对任务的处理时间减少10%左右,资源耗费仅占所有节点资源30%左右。通过上述四个方面的研究,我们可以在很大程度上对赛博-物理空间中应用提供基础性支持,可以为赛博-物理空间中应用的拓展提供有效地支撑。
[Abstract]:Cyber - physical space fusion as information space and physical space, can be made with information awareness, information exchange and computing entities according to the application requirements related to each other. The relationship between entities ranging from physical domain, information domain to the social domain and multi dimension information, has the scale, distribution, dynamic and other characteristics. In such a space to carry out the integration, development and application are becoming more and more scientific workers pay close attention to the topic. In cyber - physical space, from the perspective of information perception, need to face the large-scale distributed entity information and the entity relation discovery and aggregation challenge. Here the relationship between entities including the description of information from different the dimension according to the application requirements, such as the entities in the physical space of the relative position, expressed by the social attribute and user related logical relations. From the letter Information aggregation perspective, need to face the entities and the relationship between dynamic challenge. Here include dynamic entity state changes, dynamic changes caused by the physical location of entity attribute values, and the resulting entity relationship between dynamic. From the information processing perspective, the need for efficient the challenges facing the large-scale processing of heterogeneous data, including the design of specific types of large-scale data processing, and the diversified business model for large scale data processing platform needs the design. This article from the cyber physical space in the application of how to carry out effective information aggregation and efficient large-scale data processing of two angles, to study how to efficiently distributed on relations between entities, cyber - oriented dynamic entities in the physical space between the research on how to satisfy the application of real Find the relationship between body stable demand, that is how to build a stable network effectively, according to data processing needs how to design a specific type of data processing model, as well as for the diversified business model to the platform integration and task scheduling optimization of data processing. This paper is mainly divided into the following four aspects: 1. low load entity relationship model of distributed polymerization based on the discovery mechanism: through the establishment of distributed cyber - physical space discovery model, considering the multiple relations also found the demand, design the MSTRD algorithm found distributed relation based on spanning tree, the relationship between entities in cyber - physical space from the traditional centralized discovery into distributed, while ensuring the efficiency of the process of relation discovery that is, to reduce the amount of data transmission between the discovery process and time consuming. The experimental results show that the MSTRD method is not tube From the relationship found in the process of data transmission or total time speaking than the centralized processing performance better, the maximum amount of data transmission can save about 1/3, while steady stator net construction mechanism saves half of the time spent in.2. oriented distributed dynamic aggregation network in extreme cases: through the establishment of cyber - oriented distributed entities the model used in the physical space, the design of a distributed entity set selection algorithm MRMC selection algorithm through the distributed object set, can be set in the scale of growth is limited to improve the relationship between entity entity constant time RUT, stable sub networks. Extensive simulation experiments are carried out to verify the correctness and validity. The results show that the MRMC algorithm, MRMC algorithm can keep lifting up one time RUT times.3. matrix Abstract Based on race relations between entities Bo - physical space of large scale data processing framework: This paper designs a general computational framework for Matrix Map matrix, to construct scalable data structure block key matrix BKM across the memory and storage, the design optimization strategies for the parallel algorithm of data reorganization and asynchronous computation, as well as a general matrix element two yuan, operator. Experimental results show that for some common image processing algorithms, the Matrix Map Spark system can be the most popular than about 10 times faster than.4. for cyber - physical space application of diversified business model platform big data fusion technology in processing speed for large data processing diversity and complexity of the platform. Through the analysis of the current mainstream big data processing platform advantages and disadvantages, proposed fusion scheme for different data processing platform, designed for different tasks in data processing Integrated scheduling framework in the system design and optimization strategy of MMRC cluster scheduling genetic algorithm based on the task, and then improve the efficiency of data processing, reduce the consumption of resources. The simulation results show that the single system and compared, MMRC method can deal with time on task decreases about 10%, resource consumption accounted for only 30% of all nodes right. Through the above four aspects, we can largely to provide basic support application of cyber - physical space, can provide effective support for the development and application of cyber - physical space.

【学位授予单位】：国防科学技术大学
【学位级别】：博士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】