虚拟计算环境下性能可预测编程模型及其支撑技术研究

发布时间：2018-04-21 04:26

本文选题：并行计算 + 编程模型　；参考：《上海大学》2014年博士论文

【摘要】：随着数据规模越来越大，，复杂程度越来越高，建立大规模数据中心来对海量数据进行存储和分析成为一个新的趋势。为了有效地管理系统资源和提高系统可靠性，数据中心大多采用了虚拟化技术。基于虚拟化的数据中心将大量计算、存储和通信等资源整合到一起，以虚拟化的方式提供按需配置资源的服务模式。为了更好地利用资源，需要研究适合虚拟计算环境的并行编程模型，且编程模型应该具有性能可预测性，可以指导开发人员设计应用和配置资源。 BSP模型具有性能可预测、容易编程、能避免死锁等优点，BSP模型不仅适合科学计算，近几年在并行数据库、搜索引擎、大规模图处理等领域获得了广泛应用，但是BSP模型必须与硬件结构相结合，才能充分利用硬件结构，以达到性能最优。到目前为止，在将BSP模型与虚拟计算环境相结合，研究虚拟计算环境下基于BSP模型的大数据处理框架方面的研究还比较欠缺，本文正是基于这个目的，围绕虚拟计算环境下性能可预测并行编程模型及其支撑技术展开，本文主要的研究成果如下： (1)针对目前面向大数据处理的并行编程模型研究中存在的不足，利用BSP模型性能可预测、易于编程、消息传递不产生死锁等优点，将BSP模型与虚拟计算环境相结合，提出一种虚拟计算环境下分布式内存与共享内存混合的并行编程模型BSPCloud。 (2)网络I/O是影响BSPCloud应用性能的一个很重要的因素，在虚拟计算环境下，由于多个虚拟机共享和竞争同一I/O资源，导致虚拟机的I/O性能很难保障。本文提出一种基于I/O请求排队的网络资源调度方法。该方法根据各虚拟机的网络带宽周期性地为其分配相应的额度值，利用该额度值控制I/O请求量。通过实验对该方法进行了性能分析和验证，结果表明该方法能有效保障虚拟机的网络I/O性能。 (3)针对虚拟计算环境下，由于VCPU调度次序的不确定性导致BSPCloud应用性能降低和性能预测能力下降的问题，提出一种基于虚拟域的VCPU协同调度方法，该方法将组调度和虚拟域相结合，可以避免VCPU调度的不确定性，提高BSPCloud应用的性能和预测能力。通过实验对该方法进行了性能分析和验证，结果表明该方法能有效提高BSPCloud应用性能和预测能力。 (4)在虚拟计算环境下，运行不同类型应用(比如计算密集型，I/O密集型)的虚拟机在同一物理平台上运行时，静态的资源分配策略不能充分利用底层物理资源。另外，BSPCloud应用将计算和通信相分离，静态的资源分配策略会导致BSPCloud应用在计算阶段浪费大量网络I/O资源，而在通信阶段却浪费大量的计算资源。为了解决这个问题，本文提出一种资源动态分配方法CRDA，该方法利用虚拟机的资源分配和消耗情况来预测其资源需求，根据各虚拟机的实际需求动态分配资源。实验结果表明该方法可以在混合负载环境下提高资源利用率，从而提高BSPCloud应用的性能。 (5)目前，基于虚拟化的资源整合变得非常流行，BSPCloud应用通常会提交到虚拟计算中心运行，为了可以预测BSPCloud在虚拟计算中心运行时的响应时间，本文对虚拟计算中心进行性能建模，将一个服务请求划分为多个子任务，将每个子任务的处理时间划分为计算和通信两个阶段，并充分考虑虚拟计算中心的资源调度策略和虚拟机之间的资源共享。
[Abstract]:As the scale of data is increasing and the complexity is getting higher and higher, it is a new trend to store and analyze mass data in large scale data center. In order to effectively manage system resources and improve system reliability, most data centers have adopted virtualization technology.
Data centers based on Virtualization will integrate a large number of computing, storage and communication resources together to provide a service mode for configuring resources on demand in a virtualized way. In order to make better use of resources, a parallel programming model for virtual computing environment needs to be studied. The programming model should have performance predictability and can guide development. Personnel are designed to apply and deploy resources.
The BSP model has the advantages of predictable performance, easy programming and avoiding deadlock. The BSP model is not only suitable for scientific computing, but has been widely used in the fields of parallel database, search engine and large scale graph processing in recent years, but the BSP model must be combined with the hardware structure to make full use of the hardware structure to achieve the best performance. So far, combining the BSP model with the virtual computing environment, the research on the large data processing framework based on the BSP model in the virtual computing environment is still relatively short. This paper is based on this purpose. This paper focuses on the performance predictable parallel programming model and its supporting technology under the virtual computing environment. The main research results of this paper are as follows. Below:
(1) in view of the shortcomings of the parallel programming model for large data processing, the advantages of the BSP model can be predicted, easy to program, and the message transfer does not produce the life and death lock. The BSP model is combined with the virtual computing environment, and a parallel programming model, BSP, which is mixed with distributed memory and shared memory under the virtual computing environment, is proposed. Cloud.
(2) network I/O is an important factor affecting the performance of BSPCloud applications. In the virtual computing environment, because of the sharing and competition of the same I/O resources by multiple virtual machines, the I/O performance of the virtual machine is difficult to guarantee. This paper proposes a network resource adjustment method based on I/O request queuing. This method is based on the network bandwidth cycle of each virtual machine. The corresponding quota is allocated for it in a period of time, and the amount of the I/O request is controlled by the amount. The performance analysis and verification of the method is carried out through experiments. The results show that the method can effectively protect the network I/O performance of the virtual machine.
(3) in the virtual computing environment, due to the uncertainty of VCPU scheduling order resulting in the degradation of BSPCloud application performance and the decline of performance prediction ability, a collaborative scheduling method based on virtual domain based VCPU is proposed. This method combines group scheduling and virtual domain to avoid the uncertainty of VCPU scheduling and improve the application of BSPCloud applications. The performance of the method is analyzed and verified through experiments. The results show that the method can effectively improve the performance and prediction ability of BSPCloud applications.
(4) under the virtual computing environment, when running different types of applications (such as computing intensive, I/O intensive) virtual machines running on the same physical platform, the static resource allocation strategy can not make full use of the underlying physical resources. In addition, the BSPCloud application separates the computing and communication, and the static resource allocation strategy will lead to the BSPCloud application A large amount of network I/O resources are wasted in the computing stage, but a lot of computing resources are wasted in the communication stage. In order to solve this problem, a dynamic resource allocation method, CRDA, is proposed in this paper. This method uses the resource allocation and consumption of virtual machines to predict the resource requirements, and the resource allocation is dynamically allocated according to the actual requirements of each virtual machine. The test results show that this method can improve resource utilization in mixed load environment, thereby improving the performance of BSPCloud applications.
(5) at present, resource integration based on virtualization has become very popular. BSPCloud applications are usually submitted to virtual computing centers. In order to predict the response time of BSPCloud in the virtual computing center, the performance modeling of the virtual computing center is made, and a service request is divided into multiple subtasks and each sub task is made. The processing time is divided into two stages of computation and communication, and the resource scheduling strategy of virtual computing center and the resource sharing between virtual machines are fully considered.

【学位授予单位】：上海大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：TP391.9

【参考文献】