MongoDB中的海量数据动态平衡

发布时间：2018-07-04 11:54

本文选题：云计算 + MongoDB　；参考：《上海交通大学》2014年硕士论文

【摘要】：在云计算的技术架构中，数据存储层是基础。当前，云平台中存储了大量的结构化和非结构化数据，这些数据具有高并发性、数据量大、数据之间联系弱等特点。相比关系型数据库，非关系型数据库由于其强大的适应能力得到了广泛地应用。然而由于非关系型数据库发展时间尚短，在负载均衡技术方面的发展还不够成熟，因此，针对提高资源利用率的需求设计有效的非关系型数据库的负载均衡策略具有重要意义。目前大部分数据库的负载均衡技术都是基于数据量实现的，这是因为数据库的职责一直被界定为数据存储，一般认为访问负载的监控应该由应用层实现。然而，随着web2.0的发展，数据访问量的差异被凸显，热点数据分布的均衡特性难以保证，而访问负载过重会导致系统性能下降，访问负载过轻会造成存储系统资源浪费。因此，传统的基于数据量的负载均衡策略难以满足现状。本文旨在研究云平台下非关系型数据库的负载均衡问题，并有针对性地优化MongoDB的负载均衡架构，以提高存储资源的使用率和存储系统的性能。本文主要完成了以下内容的研究：首先，本文分析了MongoDB现有的负载均衡架构的局限性。通过分析目前MongoDB的负载均衡框架，发现其主要存在三个方面需要优化。第一，其是基于数据量的负载均衡策略，这与现实中多数情况下相同数据量的数据负载不相当的现状不符。第二，其没有充分利用副本集和分布式文件系统的特点，事实上副本集除可用来灾后修复外也可用来分担负载，从而以较小的代价平衡系统中的负载。第三，其只有通过自动分片的扩容操作，并没有相应的减少容量的相关操作，另外，在进行平衡负载操作时，其主要策略是进行数据迁移，但是对于迁移数据的选择不具有针对性且迁移数据时资源消耗较大。接着本文提出了优化的MongoDB负载均衡架构和算法。根据目前MongoDB在负载均衡方面的不足，结合分布式文件系统的特点，提出了优化的负载均衡策略和算法。该策略在原有的架构之上引入负载监控器，并改造原有的负载均衡器平衡负载的工作流程和主副本切换机制，以“预防为主，最低消耗处理”的思想实现了基于访问负载的动态负载平衡，并大大减少了资源的消耗。最后本文设计并进行试验以证明此优化框架的可行性。在大规模分布式系统中，快速的响应和高可靠性是终端用户和应用程序追求的两个重要的性能指标，这些性能很大程度上由资源管理策略以及系统中各节点的负载状况来决定。因为负载量的均匀分配能够帮助优化资源利用、最大化吞吐量和消除潜在负载过重的威胁。负载均衡策略优劣的判断有两个重要的标准，，分别为迁移代价和对存储系统性能的影响。本文提出的优化的MongoDB的负载均衡策略能够以更小的代价和更低的影响性实现系统的再次平衡，从而提高存储资源的利用率。另外，本文中所设计的负载均衡策略可以广泛用于非关系型数据库，对于云平台下非关系型数据的大规模存储策略的优化具有现实意义。
[Abstract]:In the technical architecture of cloud computing, the data storage layer is the foundation. At present, a large number of structured and unstructured data are stored in the cloud platform. These data have the characteristics of high concurrency, large amount of data, weak connection between data and so on. Compared with relational database, the non relational data base has been widely used because of its strong adaptability. However, due to the short development time of non relational database, the development of load balancing technology is not mature enough. Therefore, it is of great significance to design an effective non relational database load balancing strategy for improving the demand for resource utilization.
At present, most database load balancing technologies are based on data volume implementation, which is because the responsibility of the database has been defined as data storage. It is generally believed that the monitoring of the access load should be implemented by the application layer. However, with the development of Web2.0, the difference of data access is highlighted, and the equilibrium characteristics of hot data distribution are difficult. It is guaranteed that overload of access will cause the performance degradation of the system and the overload of access load will cause the waste of storage system resources. Therefore, the traditional load balancing strategy based on the data amount is difficult to meet the status quo.
The aim of this paper is to study the load balancing of non relational databases under the cloud platform, and to optimize the load balancing architecture of MongoDB to improve the utilization of storage resources and the performance of the storage system.
This article mainly completed the following research:
First, this paper analyzes the limitations of the existing load balancing architecture of MongoDB. By analyzing the current load balancing framework of MongoDB, it is found that there are three main aspects to be optimized. First, it is a load balancing strategy based on the data quantity, which is not consistent with the current situation that the data load of the same data is not equal in most cases. Second, it does not make full use of the features of the replica set and the distributed file system. In fact, the replica set can also be used to share the load in addition to the post disaster recovery. In order to balance the load in the system at a smaller cost. Third, it has no corresponding operation to reduce the capacity. When balancing load operation, its main strategy is data migration, but the choice of migrating data is not targeted and the consumption of resources is large when migrating data.
Then this paper proposes an optimized MongoDB load balancing architecture and algorithm. Based on the shortage of load balancing in the current MongoDB and the characteristics of the distributed file system, the optimized load balancing strategy and algorithm are proposed. This strategy introduces the load monitor on the original architecture, and reforms the original load balancer balance load. The workflow and master copy switching mechanism, with the idea of "prevention mainly, minimum consumption processing", realized dynamic load balancing based on access load, and greatly reduced the consumption of resources.
Finally, this paper designs and tests to prove the feasibility of the optimization framework.
In large-scale distributed systems, fast response and high reliability are two important performance indicators pursued by end users and applications. These performance is largely determined by the resource management strategy and the load status of each node in the system. Because the uniform distribution of the load can help optimize the utilization of resources and maximize the swallowing. There are two important criteria for the judgment of the load balance strategy, which are the migration costs and the performance of the storage system. The optimized MongoDB load balancing strategy can achieve the rebalance of the system at a smaller cost and lower impact, thus improving the storage capacity. In addition, the load balancing strategy designed in this paper can be widely used in non relational databases, which is of practical significance for the optimization of large-scale storage strategy for non relational data under the cloud platform.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.09;TP311.13

【共引文献】