分布式文件系统元数据存取技术研究

发布时间：2018-06-27 20:05

本文选题：小文件存储 + 高并发访问　；参考：《重庆交通大学》2016年硕士论文

【摘要】：伴随着互联网应用的飞速发展及人们消费观念的不断转变,互联网应用已经渗透到人们生活的方方面面,这带给互联网企业的不仅仅是用户量和利益的暴增,还带来了文件高并发访问和大量小文件需要存储等问题,分布式文件系统为解决这些问题提供了很好的平台。当前主流的分布式文件系统大多采用集中式元数据存储结构并且主要针对大文件存储进行设计。但是目前很多大型网站都需要存储大量图片文件,而这些图片文件大多小于1MB,主流分布式文件系统在存储小文件时存储和检索效率并不高。并且大型网站的资源文件并发访问量很高,集中式结构的中心服务器已很难支撑高并发的资源请求。主流分布式文件系统在大型网站应用中存在两方面不足:(1)存储小文件时存储与检索效率不高;(2)集中式结构很难支撑高并发访问。而针对小文件存储优化的研究主要集中在两个方向:对元数据存储优化和对数据文件存储优化。由于高并发问题与元数据存储结构相关,因此本文主要从元数据存储优化方向进行研究。针对以上问题本文的主要研究内容如下:(1)提出了将有序哈希表用于存储子目录。由于分布式文件系统中子目录数动辄上万,检索速度较慢,因此文中提出构建复合数据结构有序哈希表来解决该问题。通过实验证明有序哈希表比动态数组检索速度提高99.93%。(2)提出了分布式目录树结构。通过分布式目录树构建的目录系统能够将目录分散存储在不同的服务器中,并且所有数据在逻辑上保持树形结构。考虑到小文件存储占用太多元数据服务器内存,并且存储数量受到元数据服务器的内存限制,而通过将元数据分布保存到不同服务器中,则使得元数据存储容量可以通过添加元数据服务器的方式线性增加,以便能够存储更多小文件。通过实验比较分布式目录树存储空间比HDFS多使用了16.4%,而检索速度比HDFS提升了73.21%。(3)提出了将基于REST风格的HTTP协议作为分布式文件系统的服务提供方式,使用HTTP协议后浏览器能够直接与所有服务器进行交互。并且通过实验证明使用HTTP协议比RPC协议能更好地利用浏览器端缓存,有效减少文件请求次数。文中通过对分布式目录树的元数据创建多个副本并保存到不同服务器中的方式,不仅提高了系统的可靠性也使得浏览器通过访问副本服务器,减少了对原数据服务器的请求次数,降低了服务器的负载。通过实验证明使用HTTP协议后部分命令的请求次数下降了35.5%。
[Abstract]:With the rapid development of Internet application and the constant change of people's consumption concept, Internet application has penetrated into every aspect of people's life, which brings not only the explosion of users and benefits to Internet enterprises. It also brings problems such as high concurrent access to files and large number of small files to be stored. Distributed file system provides a good platform to solve these problems. Current distributed file systems mostly use centralized metadata storage structure and are mainly designed for large file storage. However, many large web sites need to store a large number of image files, and most of these image files are less than 1MB. the mainstream distributed file system is not efficient in storing and retrieving small files. In addition, the resource files of large websites are visited simultaneously, so it is difficult to support the high concurrent resource request by the centralized central server. The main distributed file system has two disadvantages in the application of large websites: (1) the efficiency of storing and retrieving small files is not high; (2) the centralized structure is difficult to support high concurrent access. The research on the optimization of small file storage is mainly focused on two directions: the optimization of metadata storage and the optimization of data file storage. Because the high concurrency problem is related to the metadata storage structure, this paper mainly studies the optimization direction of metadata storage. The main contents of this paper are as follows: (1) an ordered hash table is proposed to store subdirectories. Because the number of subdirectories in distributed file system is more than ten thousand and the retrieval speed is slow, this paper proposes to construct an ordered hash table with compound data structure to solve this problem. The experimental results show that the ordered hash table is 99.93 faster than the dynamic array search speed. (2) A distributed directory tree structure is proposed. The directory system constructed by the distributed directory tree can distribute the directory into different servers, and all the data can be logically maintained in the tree structure. Considering that small file storage takes up too much metadata server memory, and that the amount of storage is limited by the memory of metadata servers, by distributing metadata to different servers, The metadata storage capacity can be increased linearly by adding metadata servers, so that more small files can be stored. Compared with HDFS, the storage space of distributed directory tree is 16.4 more than that of HDFS, and the retrieval speed is 73.21. (3) the REST-based HTTP protocol is proposed as the service provider of distributed file system. Using the HTTP protocol, the browser can interact directly with all servers. Experiments show that HTTP protocol can make better use of browser cache than RPC protocol, and reduce the number of file requests effectively. This paper not only improves the reliability of the system but also enables the browser to access the replica server by creating multiple copies of the metadata of the distributed directory tree and saving them to different servers. Reduce the number of requests to the original data server and reduce the load on the server. It is proved by experiment that the number of requests for some commands after using HTTP protocol has dropped by 35.5.
【学位授予单位】：重庆交通大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP333

【相似文献】