一体化信息基础设施中面向MapReduce的递归层次结构数据中心网络研究

发布时间：2018-05-30 21:26

本文选题：军事信息系统 + 一体化信息基础设施　；参考：《国防科学技术大学》2012年博士论文

【摘要】：从底层的硬件架构到上层的数据处理方式，数据中心网络和MapReduce共同构成数据中心与云计算的核心体系，同时也成为一体化信息基础设施快速处理大量军情信息和数据的关键技术，是未来网络中心战中夺取制信息权的必备条件。近年来，各种新的服务需求的不断涌现对数据中心网络提出了更高的结构性要求。针对这些要求，研究者们设计了一些新的数据中心网络结构。其中递归层次结构相对其它结构，具有更强的自组织性、更高的可靠性、更好的可拓展性等优点，在军用和民用领域都具有重要研究价值和广泛应用前景。但研究者们仅仅从结构本身来考虑如何提高网络性能，却忽略了数据中心网络设计的实用性要求，特别是与MapReduce的数据处理机制相适应的问题。本文针对一体化信息基础设施中递归层次结构数据中心网络与MapReduce的匹配这一议题展开研究。主要研究工作和创新点如下： 1）提出了递归层次结构数据中心网络的可靠性分析方法出于军事需求考虑，提出了分析判断递归层次结构数据中心网络可靠性的系统方法。从拓扑设计的角度分析了递归层次结构数据中心网络可靠性的评价指标，包括连通性、聚合性、以及敏感性。基于递归层次结构数据中心网络的形式化描述，针对每个可靠性评价指标，研究了具体的量化评价分析方法。利用DCell、FiConn和BCube这三种目前最为典型的递归层次结构为案例，检验了方法的可行性和有效性。通过案例分析发现，虽然FiConn的敏感性最好，但其聚合性和连通性最差，综合三个评价指标的分析结果，得出BCube的可靠性最高。 2）提出了MapReduce程序设计的合理性分析方法基于对象Petri网，提出了一种全面分析验证MapReduce程序设计合理性的系统方法。总结出分析MapReduce程序设计合理性的具体目的，并基于这些目的研究给出MapReduce程序的合理性指标，包括具有逻辑上可执行的工作流程、不存在Straggler和Map冲突、具有合理的运行时间，以此判断MapReduce程序是否存在设计不合理问题。由于对象Petri网能够很好地描述复杂MapReduce程序的内在关系，准确无误的模拟MapReduce程序中各个步骤的执行情况，并且在模拟过程中不需要人工干预，因此利用对象Petri网模拟MapReduce的数据处理步骤。针对各个合理性指标采取不同的方法分析对象Petri网的运行过程和结果判断MapReduce程序是否存在设计不合理的问题，以达到不通过在数据中心网络上运行MapReduce程序就能验证其合理性的目的。通过针对每条合理性指标的实验分析和结果，，证明了方法的有效性。 3）设计了一种支持MapReduce的递归层次结构基于BCube和Fat-tree结构，设计了一种支持MapReduce的递归层次结构——Hyper-Fat-tree Network（HFN）。HFN的构建方法基于BCube的递归规律，即以一个低层的网络拓扑作为一个递归单元，多个这样的递归单元按照超立方体（hypercube）的节点连接关系构成一个高一层的网络拓扑。但不同于BCube的是，HFN的最小递归单元采用类似Fat-tree的冗余构造，并依据MapReduce的执行控制过程，明确了主服务器和工作服务器的相对位置和连接关系，以适应MapReduce的数据处理机制和提高利用MapReduce进行分布式数据处理的可靠性。由于结合了超立方体和Fat-tree的优点，HFN具有连通性高、直径小、可靠性好的特点。HFN还具有较好的可拓展性，HFN可连接的服务器数量是BCube中服务器数量的数倍，有效满足了一体化信息基础设施建设发展对数据中心网络服务器数量不断增长的要求。 4）提出了在一般递归层次结构数据中心网络上组织维护数据文件的方法基于分布式哈希表的基本原理，提出了一种在一般递归层次结构数据中心网络上组织维护数据文件的方法。针对递归层次结构中服务器的互联方式，确定了服务器在组织维护数据文件过程中的作用。利用分布式哈希表的基本原理，研究了在一般递归层次结构数据中心网络上实现数据存储、读取和维护的方法。介绍了这些方法中涉及的分布式哈希表的键表结构，以及基于此结构进行数据文件维护操作的路由方法。针对服务器故障，给出了在一般递归层次结构数据中心网络上组织和维护数据文件的容错方法。通过实验分析比较了在HFN和BCube上进行数据操作的平均路径长度和考虑节点故障时数据操作的成功率，证明了此方法能够将各种数据操作请求信息迅速有效地发送给相应的目的服务器，利用此方法在一般递归层次结构数据中心网络上组织和维护数据文件是可行且有效的。 5）提出了在一般递归层次结构数据中心网络上运行MapReduce的方法基于分布式哈希表的基本原理，提出了一种在一般递归层次结构数据中心网络上运行MapReduce的方法。针对MapReduce的基本数据处理机制，研究了在一般递归层次结构数据中心网络上如何区分主服务器和工作服务器，以及分配Map和Reduce任务、传输中间数据的方法。介绍了这些方法中涉及的分布式哈希表的键表结构，以及基于此结构进行任务分配和中间数据传输的路由方法。研究了MapReduce在一般递归层次结构数据中心网络上的容错方法，包括容错路由以及针对服务器故障的容错方法。通过实验分析比较了HFN与BCube在利用此方法运行MapReduce时的网络性能，包括负载平衡、吞吐量和带宽，证明了此方法能够将整个数据中心网络的负载平均分配给各台服务器，并且在节点故障率较高的情况下依然能够满足带宽需求量大的MapReduce应用和服务。
[Abstract]:From the underlying hardware architecture to the data processing on the upper level, the data center network and the MapReduce constitute the core of the data center and cloud computing. At the same time, it is also the key technology for the integration of information infrastructure to quickly deal with a large number of military information and data. It is a necessary condition for the acquisition of information right in the future network center war. Over the years, the growing demand for new services has made a higher structural requirement for the data center network. In response to these requirements, researchers have designed some new data center network structures, in which the recursive hierarchy is more self-organized, more reliable, and more expansibility, and so on. It has important research value and wide application prospects in both military and civil fields. But researchers only consider how to improve the network performance from the structure itself, but ignore the practical requirements of the data center network design, especially the problem that adapts to the data processing mechanism of MapReduce. This paper aims at the integrated information infrastructure. The issue of the matching between the recursive hierarchical data center network and MapReduce is studied.
The main research work and innovation are as follows:
1) the reliability analysis method of recursive hierarchical data center network is proposed.
A systematic method of analyzing and judging the reliability of the recurrent hierarchical structure data center network is proposed for the consideration of military needs. The evaluation indexes of the reliability of the recursive hierarchical structure data center network are analyzed from the point of view of topology design, including connectivity, aggregation, and sensitivity. The feasibility and effectiveness of the three most typical recursive hierarchical structures of DCell, FiConn and BCube are examined. The case analysis shows that although the sensitivity of FiConn is the best, its aggregation and connectivity is the worst, Combining the analysis results of three evaluation indexes, it is concluded that the reliability of BCube is the highest.
2) put forward the rationality analysis method of MapReduce programming.
Based on object Petri net, this paper presents a systematic method to comprehensively analyze and verify the rationality of MapReduce program design, and summarizes the specific purpose of analyzing the rationality of MapReduce program design. Based on these purposes, the rationality index of the MapReduce program is given, including the logically executable workflow, and there is no Straggler and Map The conflict, with a reasonable running time, can judge whether the MapReduce program has the problem of unreasonable design. Because the object Petri network can describe the internal relationship of the complex MapReduce program well, accurately simulate the execution of each step in the MapReduce program, and do not need manual intervention in the simulation process. The object Petri network simulates the data processing steps of the MapReduce. According to each reasonable index, it takes different methods to analyze the operation process and the result of the object Petri net to judge whether the MapReduce program has the unreasonable design problem, so as to achieve the purpose of verifying its rationality without running the MapReduce program on the data center network. The effectiveness and effectiveness of the method are proved by analyzing the results of each reasonable index.
3) a recursive hierarchical structure supporting MapReduce is designed.
Based on the structure of BCube and Fat-tree, a recursive hierarchical structure supporting MapReduce, Hyper-Fat-tree Network (HFN).HFN, is designed based on the recursive law of BCube, that is, a low-level network topology is used as a recursive unit, and many such recursion units are connected by the nodes of the hypercube (hypercube). It is a high level network topology. But unlike BCube, the minimum recursive unit of HFN uses redundant constructs similar to Fat-tree. According to the execution control process of MapReduce, the relative position and connection relationship between the main server and the work server is defined to adapt to the MapReduce data processing mechanism and improve the use of MapReduce. The reliability of distributed data processing. Because of the advantages of hypercube and Fat-tree, HFN has high connectivity, small diameter, and good reliability,.HFN also has good expansibility. The number of HFN connected servers is several times the number of servers in BCube, and it is full of integrated information infrastructure development to data. The demand for the growing number of central network servers.
4) a method of organizing and maintaining data files on a general recursive hierarchical data center network is proposed.
Based on the basic principle of distributed hash table, a method of organizing and maintaining data files on a general recursive hierarchical structure data center network is proposed. The application of the server in the process of organizing and maintaining data files is determined. The basic principle of the distributed hash table is studied. The method of data storage, reading and maintenance on the general recursive hierarchical data center network is introduced. The key table structure of the distributed hash table involved in these methods, and the routing method based on this structure for data file maintenance operation are introduced. The general recursive hierarchical structure data center is given for the server failure. The fault tolerance method of organizing and maintaining data files on the network. The average path length of data operation on HFN and BCube and the success rate of data operation considering the node failure are compared by experiment analysis. It is proved that this method can send various data operation request information to the corresponding destination server quickly and effectively, and use this method to use this method. It is feasible and effective to organize and maintain data files on general recursive hierarchical data center networks.
5) a method of running MapReduce on general recursive hierarchical data center network is proposed.
Based on the basic principle of distributed hash table, a method of running MapReduce on a general recursive hierarchical data center network is proposed. Based on the basic data processing mechanism of MapReduce, the paper studies how to distinguish between the master server and the work server on the general recursive hierarchical data center network and the allocation of Map and Reduce. The method of transferring intermediate data. This paper introduces the key table structure of distributed hash table involved in these methods, and the routing method based on this structure for task assignment and intermediate data transmission. It studies the fault tolerant methods of MapReduce in the general recursive hierarchical data center network, including fault-tolerant routing and for the server. Through experimental analysis, the network performance of HFN and BCube when using this method to run MapReduce, including load balance, throughput and bandwidth, proves that this method can allocate the load of the whole data center network to each server, and can still be satisfied when the node failure rate is high. MapReduce applications and services with large bandwidth requirements.
【学位授予单位】：国防科学技术大学
【学位级别】：博士
【学位授予年份】：2012
【分类号】：TP308

【参考文献】