基于路由器的分布式存储技术的研究
发布时间：2018-11-10 11:22
【摘要】：随着科技的发展,信息技术已经深刻的影响着人类的方方面面,而数据海量化给存储技术带来了巨大挑战。相比传统的存储系统,分布式存储系统通过利用大量的小型服务器甚至个人PC来组成存储空间巨大的存储池,因此具有低成本和高扩展的优势。 分布式存储系统通过存储冗余数据在大量的存储节点来提供可靠的数据存储服务,保证文件的可靠性和可用性。副本和纠删码是两种常见的冗余机制。由于硬件故障、网络故障等原因会造成存储节点失效即节点离开系统,这种现象是很常见的。当存储节点失效时,造成该节点内文件不再被系统所使用。为了维持系统内文件的冗余度,需要数据修复机制来修复丢失的数据。另一方面网络中路由器性能越来越强大且能够对特定业务提供支持,本文创新的提出使用路由器对P2P存储提供支持。本文的主要内容包括如下两个方面： (1)数据修复协议 针对分布式存储的数据修复过程,本文创新的提出使用路由器的计算能力提供支持：对流入的多条数据流进行编码,发送出去一条数据流。根据该基本思路,本文提出了两种利用路由器的数据修复方案,并设计了两种数据修复协议：路由器支持的数据修复协议和树形路由器支持的数据修复协议。 在第一种数据修复协议中,由于路由器编码使数据修复过程的数据流数量减少,因此能减少整个过程传输的数据量。如果网络中有瓶颈链路,通过减少瓶颈链路的数据流,数据传输速度将变快因而能减少数据修复的时间。相比于第一种协议,树形修复协议主要是在修复之前先构造一棵再生树,选择带宽较大的边来进行数据传输,进一步提高数据修复的性能。通过实验仿真,使用两种数据修复协议能极大减少数据修复过程的修复时间和修复数据流量。 (2)分布式存储原型系统 除了利用路由器的计算能力,路由器的存储空间也可以被利用：在路由器中缓存存储系统的数据,使用户能够从最近的路由器上取的所需数据,减少下载时间。本文设计并实现了一套基于多核路由器平台的P2P存储系统。系统主要功能包括文件上传、文件下载、数据修复、节点检测等。系统测试表明,在有路由器支持的情况下系统主要性能要比无路由器支持的情况下获得较大提升。
[Abstract]:With the development of science and technology, information technology has deeply affected all aspects of human beings, and data sea quantization has brought great challenges to storage technology. Compared with traditional storage systems, distributed storage systems make use of a large number of small servers and even personal PC to form storage pools with huge storage space, so they have the advantages of low cost and high expansion. Distributed storage systems provide reliable data storage services by storing redundant data in a large number of storage nodes to ensure the reliability and availability of files. Copy and erasure code are two common redundancy mechanisms. Because of hardware failure and network failure, it is very common to cause storage node failure, that is, node leaving the system. When the storage node fails, the file in the node is no longer used by the system. In order to maintain the redundancy of files in the system, data repair mechanism is needed to repair the missing data. On the other hand, routers in networks are becoming more and more powerful and can support specific services. This paper proposes to use routers to support P2P storage. The main contents of this paper are as follows: (1) the data repair protocol is aimed at the data repair process of distributed storage. This paper proposes to use the computing power of routers to provide support: encoding incoming multiple data streams and sending out one data stream. According to this basic idea, this paper proposes two kinds of data repair schemes using routers, and designs two kinds of data repair protocols: data repair protocols supported by routers and data repair protocols supported by tree routers. In the first data repair protocol, the number of data streams in the data repair process can be reduced because of the router encoding, which can reduce the amount of data transmitted by the whole process. If there is a bottleneck link in the network, by reducing the data flow of the bottleneck link, the speed of data transmission will be faster and the time of data repair can be reduced. Compared with the first protocol, the tree repair protocol is mainly to construct a regenerated tree before the repair, and select the edge with large bandwidth for data transmission to further improve the performance of data repair. Through experimental simulation, using two kinds of data repair protocols can greatly reduce the repair time and repair data flow of the data repair process. (2) in addition to the computing power of the router, the storage space of the router can also be utilized in the distributed storage prototype system: the data of the storage system can be cached in the router, Enables users to retrieve the required data from the nearest router, reducing download time. This paper designs and implements a P2P storage system based on multi-core router platform. The main functions of the system include file upload, file download, data repair, node detection and so on. The system tests show that the main performance of the system with router support is better than that without router support.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333
本文编号：2322336
[Abstract]:With the development of science and technology, information technology has deeply affected all aspects of human beings, and data sea quantization has brought great challenges to storage technology. Compared with traditional storage systems, distributed storage systems make use of a large number of small servers and even personal PC to form storage pools with huge storage space, so they have the advantages of low cost and high expansion. Distributed storage systems provide reliable data storage services by storing redundant data in a large number of storage nodes to ensure the reliability and availability of files. Copy and erasure code are two common redundancy mechanisms. Because of hardware failure and network failure, it is very common to cause storage node failure, that is, node leaving the system. When the storage node fails, the file in the node is no longer used by the system. In order to maintain the redundancy of files in the system, data repair mechanism is needed to repair the missing data. On the other hand, routers in networks are becoming more and more powerful and can support specific services. This paper proposes to use routers to support P2P storage. The main contents of this paper are as follows: (1) the data repair protocol is aimed at the data repair process of distributed storage. This paper proposes to use the computing power of routers to provide support: encoding incoming multiple data streams and sending out one data stream. According to this basic idea, this paper proposes two kinds of data repair schemes using routers, and designs two kinds of data repair protocols: data repair protocols supported by routers and data repair protocols supported by tree routers. In the first data repair protocol, the number of data streams in the data repair process can be reduced because of the router encoding, which can reduce the amount of data transmitted by the whole process. If there is a bottleneck link in the network, by reducing the data flow of the bottleneck link, the speed of data transmission will be faster and the time of data repair can be reduced. Compared with the first protocol, the tree repair protocol is mainly to construct a regenerated tree before the repair, and select the edge with large bandwidth for data transmission to further improve the performance of data repair. Through experimental simulation, using two kinds of data repair protocols can greatly reduce the repair time and repair data flow of the data repair process. (2) in addition to the computing power of the router, the storage space of the router can also be utilized in the distributed storage prototype system: the data of the storage system can be cached in the router, Enables users to retrieve the required data from the nearest router, reducing download time. This paper designs and implements a P2P storage system based on multi-core router platform. The main functions of the system include file upload, file download, data repair, node detection and so on. The system tests show that the main performance of the system with router support is better than that without router support.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333
【参考文献】
相关期刊论文 前1条
1 田敬;代亚非;;P2P持久存储研究[J];软件学报;2007年06期
，本文编号：2322336
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2322336.html
