基于访问热度分类的元数据副本技术研究

发布时间：2018-04-14 23:36

本文选题：分布式文件系统 + 元数据副本　；参考：《华中科技大学》2016年硕士论文

【摘要】：在分布式文件系统中,元数据服务器是整个系统的核心,副本技术是提高系统可用性和性能的主要方法之一。而现有的元数据副本技术大多只是简单的对元数据进行复制保存,忽视了元数据访问具有的局部性和相关性,已经无法满足大数据时代对低延迟、高性能的存储访问需求。因此,非常有必要研究新的元数据副本技术,来满足这些需求。通过对分布式文件系统的元数据访问特征的分析和总结,提出了一种基于访问热度分类的元数据副本生成方案,首先利用k-means聚类算法对元数据的访问热度进行分类,将具有相似访问热度的元数据分类到一起,然后生成元数据副本,并将元数据副本发送到其他元数据服务器,当客户端再次发送元数据请求时,元数据服务器可以直接将包含该元数据的元数据副本发送给客户端,客户端可以不需要与元数据服务器交互,直接从副本中查找所需要的元数据,从而大大减少元数据服务器的响应时间,提高系统性能。同时元数据服务器还能根据副本的命中率动态地调整和更新元数据副本,保证元数据副本的高利用率。通过对所提出的副本生成方案的对比测试,得出以下结论:基于访问热度分类的副本技术能显著提高元数据读操作的IOPS;随着文件目录深度的增加,改进后系统响应时间要比未改进前的系统响应时间短,并且拥有两个元数据副本服务器的系统响应时间要比拥有单个元数据副本服务器的系统响应时间更短。
[Abstract]:In distributed file system, metadata server is the core of the whole system, and replica technology is one of the main methods to improve the availability and performance of the system.However, most of the existing metadata replica technologies simply copy and preserve metadata, ignoring the locality and relevance of metadata access, and can not meet the low latency and high performance storage access requirements of big data era.Therefore, it is necessary to study new metadata replica technology to meet these requirements.Based on the analysis and summary of metadata access features of distributed file systems, a metadata replica generation scheme based on access heat classification is proposed. Firstly, k-means clustering algorithm is used to classify metadata access heat.Categorizes metadata with similar access heat, generates a metadata copy, and sends a metadata copy to another metadata server, when the client sends the metadata request again,The metadata server can directly send a copy of the metadata containing the metadata to the client, which can find the required metadata directly from the copy without having to interact with the metadata server.Therefore, the response time of metadata server is greatly reduced and the system performance is improved.At the same time the metadata server can dynamically adjust and update the metadata replica according to the hit ratio of the replica to ensure the high utilization of the metadata replica.By comparing and testing the proposed replica generation scheme, the following conclusions are drawn: the replica technology based on access heat classification can significantly improve the IOPSs of metadata reading operation, and with the increase of file directory depth,The response time of the improved system is shorter than that of the unimproved system, and the response time of the system with two metadata replica servers is shorter than that with a single metadata replica server.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【参考文献】