基于信息分散的多云存储中间件的研究与实现

发布时间：2018-11-14 15:29

【摘要】：云存储作为云计算的重要应用模式,近年来被广泛应用于企业和个人,为个人和企业带来了前所未有的便捷性。云存储利用服务器集群技术、网格技术和分布式文件系统等技术,将不同的存储设备通过网络互联来提供数据存储方案。与传统的存储方案相比,云存储具有扩展性强、性价比高、接入性好等优点。然而,当企业或个人将数据完全托管给云存储服务提供商时也面临着服务可靠性、可用性、数据安全性、服务商锁定等问题。本文采用的基于信息分散(IDA)的混合云存储方案将多个公有云和私有云的混合,既能解决业界对于大容量存储的需求,又能保证数据存储和使用的可靠性和安全性。本文首先介绍了国内外流行的数据存取技术,并总结了已有方案在面临大数据存取以及数据安全性方面的优势和局限性,以此为背景对信息分散算法进行了研究,然后设计了基于信息分散算法的混合云存储中间件架构。此方案利用RS纠删码将用户终端上的数据进行分割、合并等处理,形成的文件元数据存储在用户的私有环境中,然后将处理后的数据存放在公有云存储服务提供商中,实现了用户文件的元数据与数据内容分离,这样便实现了文件存储服务的去中心化,任何公有平台均不存在用户的元数据。这样,既实现了文件存取的安全性、又发挥出云存储的优势。数据在公有云上的存取面临着云存储服务提供商选择的问题,涉及到网络传输速率、数据存取费用、服务可用性等问题,因此本文还研究了数据在多个公有云上的存取调度策略,保证数据存取的性能、费用等因素的协调,以实现混合云存储的经济性。在多云存储中间件架构设计的基础上,本文还描述了系统的架构、功能、模块设计,并利用多种开源工具完成系统的开发。最后我们将系统部署在HDFS、HBase组成的私有云上以及七牛云、金山云、阿里云等公有云存储服务提供商上。最后本文对此系统进行了功能和性能测试,经过测试本文所设计的混合云存储架构达到设计预期,可透明、稳定地满足用户数据在多个云存储服务提供商上的正常存取。
[Abstract]:Cloud storage, as an important application mode of cloud computing, has been widely used in enterprises and individuals in recent years, which has brought unprecedented convenience to individuals and enterprises. Cloud storage uses server cluster technology, grid technology and distributed file system technology to provide different storage devices through network interconnection to provide data storage schemes. Compared with the traditional storage scheme, cloud storage has the advantages of strong expansibility, high cost performance and good access. However, when enterprises or individuals completely host data to cloud storage service providers, they also face the problems of service reliability, availability, data security, service provider locking and so on. A hybrid cloud storage scheme based on distributed information (IDA) is proposed in this paper, which combines public cloud and private cloud. It can not only meet the needs of mass storage, but also ensure the reliability and security of data storage and usage. This paper first introduces the popular data access technology at home and abroad, and summarizes the advantages and limitations of the existing schemes in the aspects of big data access and data security. Then a hybrid cloud storage middleware architecture based on information dispersion algorithm is designed. This scheme uses RS erasure code to divide and merge the data on the user terminal, and the file metadata is stored in the private environment of the user, and then the processed data is stored in the public cloud storage service provider. The metadata of the user file is separated from the data content so that the file storage service is decentralized and there is no user metadata on any public platform. In this way, it not only realizes the security of file access, but also gives play to the advantages of cloud storage. Data access on the public cloud faces the problem of cloud storage service provider selection, which involves network transmission rate, data access cost, service availability and so on. Therefore, this paper also studies the data access scheduling strategy on multiple public clouds to ensure the coordination of data access performance, cost and other factors in order to achieve the economy of hybrid cloud storage. On the basis of the architecture design of multi-cloud storage middleware, this paper also describes the architecture, function and module design of the system, and uses a variety of open source tools to complete the development of the system. Finally, we will deploy the system to the private cloud composed of HDFS,HBase and seven Niuyun, Jinshan cloud, Ali cloud and other public cloud storage service providers. Finally, the function and performance of the system are tested. The hybrid cloud storage architecture designed in this paper can meet the design expectations, and can meet the normal access of user data on multiple cloud storage service providers in a transparent and stable manner.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】