基于网络编码的云存储系统中负载均衡技术研究

发布时间：2018-09-10 16:43

【摘要】：当今的人们生活在IT技术日新月异的信息数字时代,Internet技术得到了前所未有的迅速发展与广泛应用。互联网的发展重心由硬件设备和软件应用转向海量数据存储,消费数据存储服务的主体渐渐由企业用户转向个人用户,个体所产生的绝大部分数据均为图片、文档、视频等非结构化数据。在可扩展性,高可用性,容错处理以及延时访问方面的限制,使得传统的IT存储技术不能适应海量存储的需求。云存储使得存储容量可以通过低成本节点单位扩展来增加,再加上其高可用性和可靠性,安全存储等特性,越来越多的大公司开始将目光转向以分布式存储为基础的云存储这一新兴市场的争夺,比较出名的商业云存储服务有Apache对GFS的开源实现Hadoop,微软的Live Skydrive和Dropbox, Amazon高可用和可伸缩的分布式数据存储系统Dynamo等。其中HDFS (Hadoop Distributed File System)由于其良好特性和开源实现,渐渐成为云存储基本支撑平台的主流应用文件系统。HDFS采用多副本的复制冗余技术,存储空间和带宽传输资源有一定的浪费,在数据海量增长的情况下,这种资源消耗会呈线性方式增加,这样一个文件操作频繁的系统长期运行后,后期的维护成本会比较高。针对该问题,我们以HDFS为基础,通过融入信息论数据容错编码相关技术,建立了一个基于网络编码的分布式云存储系统NC-HDFS,在有效降低冗余度的基础上提高了系统容错能力。本文以支持系统资源监控和负载均衡调度为目标,对NC-HDFS中的性能优化技术开展了研究,加入一定的负载均衡机制来优化读写,主要贡献有以下三点：(1)针对基于网络编码的分布式存储系统NC-HDFS的新特征和HDFS仅对磁盘容量相关参数进行监控这一缺陷,设计并实现了资源监控模块,支持对CPU、内存、磁盘I/O利用率等参数的实时收集与管理,为动态负载均衡优化设计提供了决策依据。(2)针对资源状态信息的实时更新问题,进一步优化了HDFS的心跳协议及名称节点端保存数据节点性能状态的相关数据结构,使新加入的节点参考数据能及时通过协议由名称节点接受并处理,在支持多维信息更新的同时,为名称节点了解数据节点的动态变化信息提供支持。(3)针对NC-HDFS的数据读写需求,建立了基于多属性约束的动态请求调度机制,实现写文件时n个文件块所在数据节点的选择和读文件时n个数据节点中k个的选择,有效提高了读写文件的效率,并达到了系统内资源使用的均衡。
[Abstract]:Nowadays, IT technology has been developed rapidly and widely used in the information digital age, which is characterized by the rapid development of IT technology. The focus of the development of the Internet has shifted from hardware equipment and software applications to massive data storage, and the main body of consumer data storage services has gradually shifted from enterprise users to individual users. Most of the data generated by individuals are pictures and documents. Unstructured data such as video. Due to the limitations of scalability, high availability, fault-tolerant processing and delayed access, the traditional IT storage technology can not meet the needs of mass storage. Cloud storage allows storage capacity to be increased through low-cost node unit expansion, plus its high availability and reliability, secure storage, etc. A growing number of big companies are turning to the emerging market of cloud storage based on distributed storage. The well-known commercial cloud storage services include Apache's open source implementation of GFS, Hadoop, Microsoft's Live Skydrive and Dropbox, Amazon's highly available and scalable distributed data storage system, Dynamo, and so on. Because of its good characteristics and open source implementation, HDFS (Hadoop Distributed File System) has gradually become the mainstream application file system. HDFS, which is the basic support platform of cloud storage, uses multi-copy duplication redundancy technology, and the storage space and bandwidth transfer resources are wasted to a certain extent. In the case of massive increase of data, the consumption of this kind of resources will increase linearly. After a system with frequent file operation runs for a long time, the later maintenance cost will be higher. To solve this problem, we build a distributed cloud storage system (NC-HDFS,) based on network coding, which is based on HDFS and integrates information theory data fault-tolerant coding technology, and improves the system's fault-tolerant ability on the basis of reducing redundancy effectively. Aiming at supporting system resource monitoring and load balancing scheduling, this paper studies the performance optimization technology in NC-HDFS, and adds a certain load balancing mechanism to optimize reading and writing. The main contributions are as follows: (1) aiming at the new features of distributed storage system (NC-HDFS) based on network coding and the defect that HDFS only monitors the relevant parameters of disk capacity, a resource monitoring module is designed and implemented to support the memory of CPU,. The real-time collection and management of disk I / O utilization ratio and other parameters provide a decision basis for dynamic load balancing optimization design. (2) aiming at the real-time update of resource status information, Furthermore, the heartbeat protocol of HDFS and the data structure related to the storage of the performance state of the data node at the name node end are optimized, so that the new node reference data can be received and processed by the name node in time through the protocol. At the same time, it provides support for the name node to know the dynamic change information of the data node. (3) aiming at the data reading and writing requirement of NC-HDFS, a dynamic request scheduling mechanism based on multi-attribute constraint is established. The selection of data nodes in n file blocks and k in reading files is realized, which effectively improves the efficiency of reading and writing files, and achieves the balance of resource utilization in the system.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】