大数据平台分布式存储资源自动部署研究

发布时间：2018-06-25 22:04

本文选题：云计算 + 大数据　；参考：《电子科技大学》2017年硕士论文

【摘要】：随着大量的基于互联网的服务与大量的服务托管在云平台上的趋势日益流行,需要更加强大的后端存储系统来支持这些服务。一方面存储系统自身应该要有更强大的处理高并发和高强度的工作负载的能力。分布式存储是当前一种很优秀的解决办法。它将会实现以下特性:伸缩性、可用性、持久性、一致性以及分区容忍性。目前实现一个分布式存储来满足以上所有设想是非常困难的或者说是不可能的。因此,把研究的重点放在限制不同的特性来设计不同的分布式存储解决方案来满足不同的使用场景。另一方面,随着云计算技术的发展为设计分布式存储带来了新的挑战。为了更好的在云计算中使用现收现付的价格模型,一种动态的资源部署中间件被提出来(如弹性控制器)。弹性控制是能够帮助节约应用的程序的配置成本,同时不损耗应用的性能。设计一种弹性控制机制来自动部署分布式存储系统是非常重要的。这个问题将要面对的挑战是对系统的工作负载的正确判断和在伸缩系统的时候的数据迁移开销(即是系统添加或删除实例的时候传输数据消耗的时间和资源)。本课题设计的自动部署系统主要是采用了工作负载预测的方法来实现的。主要是通过监测存储系统的性能指标数据(请求延迟、cpu利用率、实例数量等),通过检测的工作负载数据来预测系统下一个时期的工作负载情况,以及使用检测的数据推出系统满足SLA的最小配置,从而在下一预测窗口开始前重新部署系统规模。在添加或移除实例的时候会做一个数据的迁移,使服务请求更加均匀的访问每一个存储实例。对工作负载的预测是整个系统的核心部分,本课题采用了基于维纳滤波器的原理实现的维纳预测器来预测每一个工作负载预测窗口的工作负载,这也是本课题的创新点。对于预测工作负载本文采用特定类型的工作负载,根据其周期性将工作负载分为特定时长的预测窗口,在每一个预测窗口开始的时候预测下一个预测窗口的工作负载。最后,通过大量的实验和测试对本课题提出的提出的预测算法以及验证分布式系统的性能,证实了基于维纳预测的方案能够在保证SLA的前提下,节约平台资源同时也就节省了配置成本。
[Abstract]:With the increasing popularity of a large number of Internet-based services and a large number of services hosted on cloud platforms, more powerful back-end storage systems are needed to support these services. On the one hand, storage systems themselves should be more powerful in handling high concurrent and high-intensity workloads. Distributed storage is currently an excellent solution. It will implement the following features: scalability, availability, persistence, consistency, and partition tolerance. It is very difficult or impossible to implement a distributed storage to meet all of the above assumptions. Therefore, the research focuses on limiting different features to design different distributed storage solutions to meet different usage scenarios. On the other hand, with the development of cloud computing technology, it brings new challenges to the design of distributed storage. In order to better use pay-as-you-go model in cloud computing, a dynamic resource deployment middleware (such as flexible controller) is proposed. Flexible control is a program that can help save application configuration costs without loss of application performance. It is very important to design an elastic control mechanism to deploy distributed storage system automatically. The challenge of this problem is to correctly judge the workload of the system and the data migration overhead (i.e., the time and resources of transferring data when the system adds or deletes an instance). The automatic deployment system designed in this paper mainly adopts the method of workload prediction. Mainly by monitoring the performance index data of the storage system (request delay CPU utilization, the number of instances, etc.), the workload data is detected to predict the workload of the system in the next period. Using the detected data, the system meets the minimum configuration of SLA to redeploy the system scale before the next prediction window begins. When an instance is added or removed, a data migration is made, which makes the service request more evenly access each storage instance. The prediction of workload is the core of the whole system. In this paper, a Wiener predictor based on the principle of Wiener filter is used to predict the workload of every workload prediction window, which is also the innovation of this subject. In this paper, a specific type of workload is used to predict the workload. According to its periodicity, the workload is divided into prediction windows of specific duration, and the workload of the next prediction window is predicted at the beginning of each prediction window. Finally, through a large number of experiments and tests on the proposed prediction algorithm and verify the performance of the distributed system, it is proved that the scheme based on Wiener prediction can guarantee SLA. Save platform resource at the same time also save configuration cost.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333;TP311.13

【参考文献】