面向代理服务的分布式文件系统设计与实现

发布时间：2019-05-27 00:43

【摘要】：随着互联网的快速普及、技术的不断成熟，越来越多的互联网服务正在快速地涌现。随之而来的存储需求问题，，成了众多问题中亟待解决的首要问题。网络代理服务是典型的公共服务之一，具有不断向集中式服务演变的趋势。其新的业务数据每天以TB速度增长，本地存储模式已经不再适应，需要一种新的存储模式，来提供高效的存储与检索性能以及良好的可扩展性。目前，代理服务尚缺少适用于这种环境的系统架构。据此，本文就目前的形势开发了一种新型的面向代理服务的分布式文件系统，以解决当前代理数据存储方面的问题。首先，本文给出分布式系统的整体架构，并针对代理服务的特点，设计与实现了分布式文件系统的通信协议。该协议支持扩容时节点的自动加入，因故障发生而导致节点异常退出时的集群节点恢复等功能。其次，对整个分布式文件系统进行了整体功能的设计与实现，其中重点对全局命名空间管理模块、数据管理模块以及负载均衡模块进行了详细的阐述。重点突破了如下关键技术：多NameServer节点的全局命名空间管理机制、快速的文件定位与检索技术、热点实时负载均衡机制以及高效的失效副本替换策略。其中，提出的定位文件的机制达到O(1)时间复杂度。最后，通过功能测试和性能测试验证了面向代理服务的分布式文件系统的有效性。综上，本文提出的面向代理服务的分布式文件系统，有效地解决了代理数据分布式存储问题。
[Abstract]:With the rapid popularization of the Internet and the continuous maturity of technology, more and more Internet services are emerging rapidly. The following problem of storage requirements has become the primary problem to be solved urgently. Network agent service is one of the typical public services, which has the trend of continuous evolution to centralized services. The new business data grows at the speed of TB every day, and the local storage mode is no longer suitable. A new storage mode is needed to provide efficient storage and retrieval performance and good scalability. At present, the agent service still lacks the system architecture which is suitable for this kind of environment. Based on this, a new distributed file system oriented to agent service is developed to solve the problem of proxy data storage. Firstly, the overall architecture of distributed system is given, and the communication protocol of distributed file system is designed and implemented according to the characteristics of proxy service. The protocol supports the automatic join of nodes when capacity expansion occurs, and the recovery of cluster nodes when nodes exit abnormally due to faults. Secondly, the overall function of the whole distributed file system is designed and implemented, in which the global naming space management module, the data management module and the load balancing module are described in detail. The key technologies are broken through: the global naming space management mechanism of multi-NameServer nodes, the fast file location and retrieval technology, the hot spot real-time load balancing mechanism and the efficient invalid replica replacement strategy. Among them, the proposed location file mechanism reaches O (1) time complexity. Finally, the effectiveness of agent-oriented distributed file system is verified by functional test and performance test. In summary, the distributed file system oriented to agent service proposed in this paper effectively solves the problem of distributed storage of proxy data.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【共引文献】