重复数据删除关键技术优化研究及SRC路由协议设计

发布时间：2018-09-10 06:30

【摘要】：数据量的爆炸式增长以及海量数据的大规模集中使得数据重复所导致的空间浪费问题越来越严重,这促使了重复数据消除技术的出现和发展。重复数据删除概念很简单,如果做一分钟“电梯演讲”：一本浩瀚的大不列颠百科丛书有4400万个字符,共计3万多英文单词,所有的这一切都是由简单的26个英文字母构成。海量数据高达EB级别,重复数据删除技术是发现海量数据中的“字母”,并将数据以“字母”的形式构成,将重复的数据进行删除以获得更好的存储空间性价比。目前,重复数据删除方面的研究工作已经在消冗率提升、性能优化以及分布式路由算法等方面取得了一系列有价值的成果,有效地推动了该技术的应用。本文从研究热点与瓶颈出发,主要研究面向服务质量的分布式重复数据删除系统,面向服务质量就需要支持服务质量的可调节,研究目标是建立一种基于多重策略设计及优化的重复数据删除技术体系。首先,本文重点研究了重复数据删除的技术体系、关键技术和关键指标,以此建立重复数据删除原型系统,其中消冗引擎作为重复数据删除技术的关键核心,包括路由算法、数据块仓库、并行流水线控制、网络通讯规约。在此基础上对关键技术的模型进行分析,分析内容包括索引模型分析、数据模型分析、性能模型分析,并从理论角度进行验证。其次,路由算法是分布式重复数据删除存储系统中的关键技术,但现有的路由算法无法满足分布式系统在消冗效率、数据迁移及集群弹性上的要求,因此,本文设计了一种基于Chord的相似性路由检测算法SRC(Similarity Routing Based on Chord),并从理论角度完成算法的收敛性、一致性证明,进一步详述了SRC路由算法的三个阶段。最后,从三种关键技术的模型分析结果出发,提出具体技术策略改进、优化方案,其中包括索引的分级优化、基于数据价值的迁移优化、基于读请求重组的性能优化。在系统实验过程中,搭建了分布式集群环境,并选取测试数据集及实验工具,完成读写并发响应时间测试、集群读写能力测试、路由算法负载均衡测试、节点容错测试等。经实验验证,对于分布式重复数据删除系统中的关键技术策略的优化及SRC路由算法的设计,明显地克服了原有技术体系中的热点瓶颈及性能缺陷,多角度、深层次、宽领域的完成对重复数据删除关键技术的研究,进一步提高重复数据删除技术服务质量,推进以重复数据删除技术为中心的绿色存储理念在云存储概念中的进一步应用。
[Abstract]:The explosive growth of data volumes and massive concentration of data have led to a growing waste of space caused by data duplication, prompting the emergence and development of duplicate data elimination techniques. A million characters, more than 30,000 English words, all of which are made up of 26 simple English letters. Massive data is up to EB level, and duplicate data deletion technology is to discover the "letters" in massive data, and to delete duplicate data in the form of "letters" to achieve better storage space. Price ratio.
At present, the research work on duplicate data deletion has made a series of valuable achievements in the aspects of redundancy reduction, performance optimization and distributed routing algorithm, which effectively promote the application of this technology. Quality of Service Oriented (QoS-Oriented) needs to support the adjustability of QoS. The research goal is to establish a repetitive data deletion technology architecture based on multiple strategy design and optimization.
Firstly, this paper focuses on the technical architecture, key technologies and key indicators of duplicate data deletion, and establishes a duplicate data deletion prototype system. The redundancy engine is the key core of duplicate data deletion technology, including routing algorithm, data block warehouse, parallel pipeline control, network communication protocol. Secondly, routing algorithm is the key technology in distributed duplicate data deletion storage system, but the existing routing algorithm can not meet the efficiency of redundancy elimination, data migration and set. Therefore, this paper designs a Similarity Routing Based on Chord (SRC) routing algorithm based on Chord, and completes the convergence of the algorithm from the theoretical point of view. The consistency proves that the three stages of SRC routing algorithm are further elaborated. Finally, based on the analysis results of the three key technologies, the paper puts forward the three stages of SRC routing algorithm. Specific technical strategy improvements and optimization schemes are proposed, including index hierarchical optimization, data value based migration optimization and reader request reorganization based performance optimization.
In the process of system experiment, the distributed cluster environment is built, and the test data sets and experimental tools are selected to complete the test of read-write concurrent response time, cluster read-write ability, routing algorithm load balancing test, node fault tolerance test and so on. And the design of SRC routing algorithm has obviously overcome the hot bottleneck and performance defect in the original technology system. The research on the key technology of duplicate data deletion has been completed from multi-angle, deep-level and wide-field. The service quality of duplicate data deletion technology has been further improved, and the concept of green storage centered on duplicate data deletion technology has been promoted. Further application of cloud storage concept.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】