基于HDFS的多租户小文件存储系统的研究与设计

发布时间：2018-06-24 21:01

本文选题：HDFS + 多租户　；参考：《浙江大学》2016年硕士论文

【摘要】：计算机技术尤其是互联网技术的高速发展,使得人们产生与分享信息变得愈发便利与多样。如当前普遍使用的微信,人们随时随地可以产生大量短小的图片、音频及视频内容。这对存储和处理这些巨量的信息提出了很大的挑战,尤其是小文件存储与处理系统,它是目前互联网应用服务必须具备的底层服务。目前Hadoop系统已经成为目前用于大数据存储和处理的公认标准。但Hadoop包含的底层分布式存储系统HDFS(Hadoop Distributed File System)的设计着重于高延时、高吞吐大文件场景的数据处理,其元数据访问模式对小文件场景十分不利。HDFS运行时必须将所有的文件元数据信息载入到服务器内存中,因而一个集群的文件存储数量受限于元数据服务器内存的大小。大量小文件会和大文件一样产生对应的元数据条目,但其实际占用的存储空间却比后者小很多,这使得元数据和集群空间利用率被很大程度地降低。在文件访问上,通常需要多达四次的网络请求客户端才能正真获取到需要的文件内容。对小文件而言,网络请求耗时比大文件占比要高且直接影响到访问效率。另外当前HDFS还不具备多租户机制,只能对用户级别的容量和文件数做配额约束,而不能动态、细粒度地限定用户的资源使用。针对上述问题,本文通过修改HDFS的元数据存储模式,首次将多级分离缓存引入到HDFS元数据存储管理中。该设计从根本上来解决了大量小文件对元数据内存的消耗问题,且可以通过不同的缓存策略降低由此引入的缓存失效延迟问题。同时本文还提出了一种新的文件直接访问模式,使得客户端可以不通过元数据节点而直接访问数据节点并获取数据,提升访问效率。最后本文首次将支持细粒度租户资源隔离与控制的多租户机制加入到HDFS中,使得各个租户在元数据使用、I/O访问上不出现恶性竞争,由此提升HDFS在多租户、大量负载应用场景下的稳定性。
[Abstract]:With the rapid development of computer technology, especially Internet technology, it becomes more and more convenient and diverse for people to produce and share information. As the current popular WeChat, people can produce a large number of short pictures, audio and video content anytime and anywhere. This poses a great challenge to the storage and processing of these huge amounts of information, especially the small file storage and processing system, which is the underlying service that Internet application services must have at present. At present, Hadoop system has become the accepted standard for big data storage and processing. However, the design of Hadoop distributed File system (HDFS), which includes the underlying distributed storage system, focuses on the data processing of high latency, high throughput and large file scenarios. The metadata access mode is very disadvantageous to the small file scenario. When HDFS runs, all file metadata information must be loaded into server memory, so the number of file storage in a cluster is limited by the size of metadata server memory. A large number of small files will produce corresponding metadata entries as large files, but their actual storage space is much smaller than the latter, which greatly reduces the utilization of metadata and cluster space. In file access, it usually takes up to four network requests to get the required file content. For small files, network requests take more time than large files and directly affect access efficiency. In addition, HDFS does not have multi-tenant mechanism at present, it can only restrict the capacity and the number of files at the user level, but it can not dynamically and finely define the resource usage of the user. In order to solve the above problems, the multi-level separation cache is introduced into the management of HDFS metadata storage for the first time by modifying the metadata storage mode of HDFS. This design fundamentally solves the problem that a large number of small files consume metadata memory, and can reduce the cache invalidation delay problem by different cache strategies. At the same time, this paper also proposes a new file direct access mode, which enables clients to directly access data nodes and obtain data without metadata nodes, so as to improve access efficiency. Finally, the multi-tenant mechanism which supports fine-grained tenant resource isolation and control is added to HDFS for the first time, so that there is no vicious competition in accessing metadata using I / O, so that HDFS can be promoted in multi-tenant. Stability in heavy load application scenarios.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP333

【相似文献】