基于两级映射体系的分布式存储子系统的设计与实现

发布时间：2018-06-10 19:00

本文选题：分布式存储系统 + 数据一致性　；参考：《电子科技大学》2013年硕士论文

【摘要】：随着国家对下一代宽带无线移动网络建设的推进，能够满足各种电信业务数据要求的移动控制网络对后台的存储支持提出了越来越高的要求。传统的存储解决方案往往不能满足系统和用户对可扩展性和可用性的需求，在面对海量用户和海量文件数据的情况下更是捉襟见肘：系统规模无法适应用户与应用规模的增长，应对海量数据的处理与访问也显得力不从心。本文首先对国内外几种主流的分布式文件系统进行了调研与介绍，面对海量用户与海量文件的应用场景，设计并实现了分布式存储系统CSTORE中的资源定位规则和数据存储模块。 CSTORE采用纯分布式的架构，将元数据与数据的分别存储在不同的集群中，并实现了一种基于两级映射的资源定位规则，结合负载均衡与数据恢复策略，能够提供较好的可扩展能力。元数据和数据的分离使得客户端对两种数据的访问分开，便于它们各自的管理与性能的优化；而集群的利用进一步促进了数据的分布式管理，避免了单点失效带来的危险。资源定位规则建立了资源与存储数据的服务器之间的映射关系，它为客户端提供了一种可靠的寻址方法，系统中任何节点都可以使用它独立计算出资源的位置，既充分利用了客户端的资源，又减轻了服务器端的压力；它的伪随机特性和映射变换同时也为系统提供了较好的负载均衡和可扩展性。数据存储模块为用户提供数据存储与访问服务，存储到系统中的数据按资源定位规则被分为了多个不同的资源管理单元，资源管理单元的多个副本之间两两建立心跳连接以推送数据和交换时序，在一定程度上保证了数据的可用性的同时也保证了一致性；容错方面，冗余机制保证了系统可以容忍在一定范围内的失效，数据恢复机制可以使得失效的模块能够尽快的对客户端服务。在数据存储模块中所有文件片都使用全局唯一ID对所存储的数据进行了去重处理。数据去重可以节省存储和管理成本，提高物理存储空间利用率。本文最后对存储系统的功能和性能进行了测试，证明了它能正确执行所需功能并保证了其高效性。
[Abstract]:With the development of the next generation broadband wireless mobile network, the mobile control network, which can meet the data requirements of various telecommunication services, has put forward higher and higher requirements for the storage and support of the background. The traditional storage solutions often can not meet the scalability and usability requirements of the system and users, especially in the face of massive users and massive file data: the scale of the system can not adapt to the growth of users and applications. It is not enough to deal with the massive data processing and accessing. Firstly, this paper investigates and introduces several mainstream distributed file systems at home and abroad, facing the application scenarios of massive users and massive files. The resource location rules and data storage modules in CSTORE are designed and implemented. CSTORE uses a pure distributed architecture to store metadata and data in different clusters. A resource location rule based on two-level mapping is implemented, which combines load balancing and data recovery strategy to provide better scalability. The separation of metadata and data enables clients to access the two kinds of data separately to facilitate their respective management and performance optimization, while the utilization of clusters further promotes the distributed management of data. The resource location rules establish a mapping relationship between the resource and the server that stores data, which provides a reliable addressing method for the client. Any node in the system can use it to calculate the location of the resource independently, which not only makes full use of the resources of the client, but also reduces the pressure of the server. Its pseudorandom characteristics and mapping transformation also provide better load balancing and extensibility for the system. Data storage module provides users with data storage and access services. The data stored in the system is divided into a plurality of different resource management units according to the resource location rules. A heartbeat connection is established between a plurality of copies of the resource management unit to push the data and exchange time series. To a certain extent, the availability of the data is guaranteed, and the consistency is ensured. In the fault-tolerant aspect, the redundancy mechanism ensures that the system can tolerate the failure in a certain range. The data recovery mechanism can enable the invalid module to serve the client as soon as possible. In the data storage module, all the files are de-reprocessed using the global unique ID. Data removal can save storage and management costs and improve the utilization of physical storage space. Finally, the function and performance of the storage system are tested, which proves that it can correctly perform the required functions and ensure its high efficiency.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】