基于元数据与文件数据分离的多云存储模型的研究与实现

发布时间：2018-03-16 05:00

本文选题：元数据　切入点：碎片化　出处：《电子科技大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着互联网和智能设备的大力发展,数据量的增长加剧,促进了云存储技术的蓬勃发展。云存储技术中很重要的一部分就是数据的机密性,目前解决用户数据机密性主流的方案是通过数据加密,但加密和解密数据带来的性能消耗是无法避免的,特别是当数据量较大和客户端的处理能力不足时这样的性能消耗是无法接受的。本文提出的基于元数据与文件数据分离的多云存储模型是对现有存储方案的一种改进和补充,区别于传统的加密手段来保护用户数据隐私,该模型既能保证数据的机密性,又能够确保文件存取的效率。根据实际的存储系统需求以及本文提出的存储模型,本文进行了系统的设计与实现。本文以功能模块为单元进行分析、设计和实现,系统的功能模块主要包括用户管理、元数据管理、文件碎片化和上传、文件下载和还原、多云分发与授权。本文的主要工作内容如下:1.研究文件雕刻技术,分析当前碎片文件还原技术,设计并实现了适合于本系统的文件碎片化策略,该策略包括文件头部的处理和不均匀碎片化。2.根据碎片化功能的需求,设计了元数据的数据结构,该结构反应了用户在系统中已存储文件的层次结构和包含了原始文件与碎片之间的对应关系。在客户端实现了使用SQLite管理元数据的功能。3.在七牛、金山、百度云存储提供的云存储服务和自建云存储系统的基础上实现了基于多个节点的云存储功能,为用户提供文件存取功能。其中包括客户端实现文件的上传、下载、删除操作和逻辑服务器端的消息回调等功能。4.针对多云存储模型的文件上传目的地选择问题,实现了多云分发算法。该算法是在现有的负载均衡的算法基础上改进而来,为了实现考虑价格和速度因素的负载均衡。5.对于一个完整的存储系统,还实现了用户管理和数据统计等功能模块。这些功能模块主要用于进入系统的授权、文件传输和操作的权限管理等方面。6.通过文件雕刻技术测试了碎片化算法的可行性,以及采用实验和数据统计的方式分析了多云分发算法的合理性。同时,测试了各个功能模块的功能以满足需求,并通过性能测试来分析系统的实际运行效果。
[Abstract]:With the rapid development of the Internet and intelligent devices, the growth of data volume has accelerated the vigorous development of cloud storage technology. The confidentiality of data is an important part of cloud storage technology. At present, the mainstream solution to the confidentiality of user data is through data encryption, but the performance consumption caused by encrypting and decrypting data is unavoidable. Especially when the amount of data is large and the processing capacity of the client is insufficient, the performance consumption is unacceptable. The multi-cloud storage model based on the separation of metadata and file data is an improvement and supplement to the existing storage scheme. Different from the traditional encryption method to protect the privacy of user data, this model can not only guarantee the confidentiality of data, but also ensure the efficiency of file access. The function module of the system includes user management, metadata management, file fragmentation and upload, file download and restore. The main work of this paper is as follows: 1. Research the file carving technology, analyze the current fragment file restoration technology, design and implement the file fragmentation strategy suitable for the system. The strategy includes file header processing and uneven fragmentation. According to the requirements of fragmentation function, the metadata data structure is designed. The structure reflects the hierarchy of stored files in the system and contains the corresponding relationship between the original files and the fragments. The function of managing metadata using SQLite is implemented on the client side. On the basis of cloud storage service provided by Baidu cloud storage and self-built cloud storage system, the cloud storage function based on multiple nodes is realized, which provides file access function for users. Delete operations and logical server-side message callbacks. 4. To solve the problem of file upload destination selection in multi-cloud storage model, a cloud distribution algorithm is implemented. This algorithm is based on the existing load balancing algorithm. In order to realize load balancing considering price and speed. For a complete storage system, user management and data statistics modules are also implemented. These functional modules are mainly used for authorization to enter the system. The feasibility of fragmentation algorithm is tested by file carving technology, and the rationality of cloud distribution algorithm is analyzed by means of experiment and data statistics. The function of each function module is tested to meet the requirement, and the actual running effect of the system is analyzed by performance test.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】