信息存储系统中重复数据删除技术的研究

发布时间：2018-11-07 11:31

【摘要】：重复数据删除技术是网络存储系统中一种数据无损压缩的解决方案，，可以有效地抑制数据存储开销过快的增长，缩减存储系统的构建以及运营管理的成本。在数据信息量迅猛增长的背景下，重复数据删除技术得到了学术界和产业界广泛的关注。但重复数据删除领域仍然存在诸多技术问题，如提高数据压缩率，减少处理时间，优化数据可靠性等方面。针对上述存在的问题，论文从重复数据删除处理方法，重复数据删除处理中的数据可靠性问题以及存储后台的数据分布策略三个方面展开了深入的研究。通过理论分析模型以及现实数据集的实测分析，对影响重复数据删除处理效果的因素展开了研究。目标数据的重复特征对重复数据删除处理的效果具有较大影响，因此，提出了一种基于重复特征的重复数据删除策略，对数据压缩率以及处理时间开销进行优化。该策略主要包括基于语义的数据分组策略和渐进式数据分割粒度判定法。基于语义的数据分组策略根据语义信息对数据的重复特征以及相似性进行判别并完成对目标数据的分组操作。渐进式数据分割粒度判定法是以数据分组为操作单位，根据重复特征对数据分割策略进行合适地设置。实验测试表明基于重复特征的重复数据删除策略相对于其它重复数据删除解决方案，在数据压缩率以及处理时间开销上获得了更加优异的综合性能。针对重复数据删除处理中数据可靠性的问题提出了一种最优冗余度计算模型，根据数据的引用热度提高目标数据的可靠性。为了将该理论模型应用到现实存储系统中，采用抽取数据单元样本空间计算经验数值的方法对理论模型进行了可行性优化，并提出一种基于引用热度的数据冗余策略。该数据冗余策略根据数据单元的相关属性（数据单元的大小以及引用热度）配置最优的冗余度，确保目标数据集使用最小的存储开销获得最优的数据可靠性。仿真实验验证了基于引用热度的数据冗余策略的可行性和有效性。针对当前数据分布策略中灵活性不足的问题，提出了一种基于容量感知的数据分布策略，以改善在物理节点间存储资源不相等的情况下存储负载的均衡程度。该策略提供了两种情况下的数据分布策略解决方案。在不考虑数据冗余度情况下，提出了一种基于容量感知的数据分布式策略，该策略基于一致性哈希数据分布算法，引入了虚拟化的设计思路，采用虚拟节点分配法进行存储资源的分配；并采用基于节点容量感知的负载均衡方法对物理存储节点之间的数据负载分布进行优化调整。在考虑数据冗余度情况下，提出了一种支持多冗余度的数据分布策略，为数据冗余策略提供灵活的平台支持，并对存储负载均衡程度进行优化。仿真测试结果表明两种数据分布策略在各自应用背景下均有助于改善存储数据负载的均衡水平。
[Abstract]:Repeated data deletion is a kind of data lossless compression solution in network storage system. It can effectively restrain the fast growth of data storage overhead and reduce the cost of building storage system and operation management. Under the background of rapid growth of data information, repeated data deletion technology has been widely concerned by academia and industry. However, there are still many technical problems in the field of repeated data deletion, such as increasing data compression ratio, reducing processing time, optimizing data reliability and so on. In view of the above problems, this paper has carried out in-depth research from three aspects: repetitive data delete processing, data reliability in repetitive data deletion processing and data distribution strategy in storage background. Based on the theoretical analysis model and the real data set, the factors that affect the processing effect of repeated data deletion are studied. The repetition feature of target data has great influence on the effect of repeated data deletion. Therefore, a repetitive data deletion strategy based on repetition feature is proposed to optimize the data compression ratio and processing time cost. The strategy mainly includes semantic data grouping strategy and progressive data segmentation granularity decision method. According to the semantic information, the data grouping strategy based on semantics is used to distinguish the repeated features and similarity of the data and to complete the grouping operation of the target data. Progressive data segmentation granularity determination method is based on the data grouping as the unit of operation, according to the repeated characteristics of the data segmentation strategy is properly set. The experimental results show that the repetitive data deletion strategy based on repetition features has better comprehensive performance in data compression ratio and processing time than other repetitive data deletion solutions. In order to solve the problem of data reliability in repeated data deletion, an optimal redundancy calculation model is proposed to improve the reliability of target data according to the heat of reference. In order to apply the theoretical model to the real storage system, this paper optimizes the feasibility of the theoretical model by taking the sample space of the data unit to calculate the empirical value, and proposes a data redundancy strategy based on the heat of reference. The optimal redundancy is configured according to the relative attributes of the data unit (the size of the data unit and the heat of reference) to ensure the optimal data reliability of the target data set using the minimum storage cost. Simulation results demonstrate the feasibility and effectiveness of the data redundancy strategy based on citation heat. Aiming at the lack of flexibility in the current data distribution strategy, a capacity-aware data distribution strategy is proposed to improve the balance of storage load in the case of unequal storage resources between physical nodes. This strategy provides a data distribution policy solution in two cases. Without considering data redundancy, a capacity-aware distributed data strategy is proposed, which is based on the consistent hash data distribution algorithm and introduces the design idea of virtualization. Virtual node allocation method is used to allocate storage resources. The load balancing method based on node capacity awareness is used to optimize the data load distribution between physical storage nodes. Considering the data redundancy, a data distribution strategy supporting multiple redundancy is proposed, which provides flexible platform support for the data redundancy policy, and optimizes the storage load balancing degree. The simulation results show that the two data distribution strategies are helpful to improve the balance level of storage data load in their respective application background.
【学位授予单位】：华中科技大学
【学位级别】：博士
【学位授予年份】：2012
【分类号】：TP333

【共引文献】