云存储系统关键技术研究

发布时间：2018-06-14 19:35

本文选题：云存储 + HDFS　；参考：《郑州大学》2013年硕士论文

【摘要】：随着数据存储的高速增长,传统的数据库已经无法满足人们处理大规模数据的需求,有关海量数据的处理与存储成为当前的研究热点。尽管一些大型云服务提供商如亚马逊、谷歌已经在云存储服务方面取得了巨大成功,但由于难以克服的隐私和数据保护等问题,许多用户仍然不能将应用转移到云环境中。而私有云能提供对数据安全性的有效保证,因此对私有云存储的架构及相关技术的研究具有理论和应用价值。Hadoop文件系统(HDFS)作为Google File System的开源实现,成为业界研究云计算、云存储和实现云端应用的参考模型。但现有的HDFS架构对小文件的支持不足,单一的NameNode容易成为整个集群性能瓶颈等问题。论文针对通用云存储系统的架构进行研究,并给出基于Hadoop处理海量小文件方法。本文主要研究内容如下：在对HDFS深入分析研究的基础上,综合考虑了通用云存储系统的特征,以提高存储效率为目标,提出一个云存储系统的架构。借鉴了国内外关于HDFS中小文件处理方法,采用二维装箱算法思想和动态优化思想,设计了小文件处理算法,对HDFS中的小文件进行预处理,提高系统的响应速度。实验表明,所提出的小文件处理方法的能有效提高资源的利用率及系统的响应速度。设计实现了一个基于高校应用环境云存储原型系统,该系统将对象内容和对象元数据进行分离,采用多区域架构模型。原型系统验证了提出的云存储系统架构合理性和有效性。
[Abstract]:With the rapid growth of data storage, the traditional database has been unable to meet the needs of people to deal with large-scale data, the processing and storage of mass data has become the focus of current research. Although some large cloud service providers such as Amazon and Google have achieved great success in cloud storage, many users are still unable to move their applications to the cloud due to insurmountable privacy and data protection problems. Private cloud can provide an effective guarantee for data security, so the research of private cloud storage architecture and related technologies has theoretical and application value. Hadoop file system (HDFS) as an open source implementation of Google File system, has become the industry research cloud computing. Cloud storage and implementation of cloud applications reference model. However, the existing HDFS architecture has insufficient support for small files, and a single name Node can easily become the bottleneck of cluster performance. In this paper, the architecture of general cloud storage system is studied, and the method of dealing with large amount of small files based on Hadoop is given. The main contents of this paper are as follows: based on the in-depth analysis of HDFS, the characteristics of general cloud storage system are considered synthetically, and a cloud storage system architecture is proposed with the aim of improving storage efficiency. Referring to the domestic and foreign small and medium file processing methods, using the two-dimensional packing algorithm and the dynamic optimization idea, a small file processing algorithm is designed to preprocess the small files in HDFS to improve the response speed of the system. Experiments show that the proposed method can effectively improve the utilization of resources and the response speed of the system. A cloud storage prototype system based on university application environment is designed and implemented. The system separates object content from object metadata and adopts multi-area architecture model. The prototype system verifies the rationality and validity of the proposed cloud storage system architecture.
【学位授予单位】：郑州大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】