大数据存储中虚拟RAID技术研究及应用

发布时间：2018-04-30 17:02

本文选题：大数据存储 + 虚拟盘组　；参考：《湖北工业大学》2017年硕士论文

【摘要】：随着大数据储规模的日益增大,其高能耗,安全性,数据管理已经成为一个无法忽视的问题,因此对数据存储的节能,安全,利用率等方面的研究是十分重要的。本文提出了虚拟盘组技术,设计出了一种大数据存储系统,通过并行处理任务,按照数据盘组进行数据的合并和存储,达到节能,安全,管理高效的目的。本文提出大数据存储系统架构,其中包括硬件设计和软件设计。在软件系统框架中提出了节点管理,实现多路数据并行存储,大大提高了软件的存储效率。在RAID技术的基础上,结合一种Virtual RAID技术,先生成种子文件,后形成数据盘组,最后完成数据的刻录存储。任务先经过预处理,添加到种子文件中,再调用删除算法以确保缓冲区大小,最后完成数据的存储形成盘组。当盘组终结后,节点服务器将数据盘组分发到各个空闲节点上,节点服务器完成数据刻录存储。当数据存储达到饱和后转为离线存储,这样既达到节能,安全,同时也提高了管理效率。为了提高服务端的兼容性,降低客户端与服务端的耦合度,大数据存储系统采用B/S架构,以数据库为应用层接口,重写ODBC数据库连接器,达到兼容多种数据库,支持多种客户端界面。本系统经过实际使用验证,在安全性、实用性、稳定性、管理效率等方面得到充分的肯定。大数据存储系统对于影视、银行等数据存储有着重要的意义。
[Abstract]:With the increasing size of big data storage, its high energy consumption, security, data management has become a problem that can not be ignored, so the research on energy saving, security and utilization ratio of data storage is very important. In this paper, a virtual disk group technology is proposed, and a big data storage system is designed. By parallel processing task, the data are merged and stored according to the data disk group to achieve the purpose of energy saving, security and efficient management. This paper proposes big data storage system architecture, including hardware design and software design. In the framework of software system, node management is proposed to realize multi-channel data parallel storage, which greatly improves the efficiency of software storage. On the basis of RAID technology, combined with a kind of Virtual RAID technology, a seed file is formed, then a data disk group is formed, and finally the data is recorded and stored. The task is preprocessed and added to the seed file, then the delete algorithm is called to ensure the buffer size, and finally the data is stored to form a disk group. When the disk group ends, the node server distributes the data disk group to each free node, and the node server completes the data burning storage. When the data storage reaches saturation, it is converted to offline storage, which not only achieves energy saving, security, but also improves management efficiency. In order to improve the compatibility of server and reduce the coupling degree between client and server, big data memory system adopts B / S structure, and rewrites ODBC database connector with database as application layer interface, so that it can be compatible with many kinds of databases. Supports multiple client interfaces. The system has been confirmed in safety, practicability, stability and management efficiency. Big data storage system for film and television, banks and other data storage has an important significance.
【学位授予单位】：湖北工业大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】