面向风电系统的分布式云存储关键技术研究

发布时间：2018-11-25 20:13

【摘要】：在当今全球经济飞速发展的时代背景下,各国对能源的开采量也不断增加,同时环境污染也日渐严峻。为保证国民经济长久有效地发展,对新能源的开发和使用是最为有效的方法之一。风能是目前大规模开发利用潜能最大的可再生能源,随着装机容量的上升,用户越来越多,对存储空间的需求也越来越大,数据增长呈现出难以想象的趋势。然而在面对海量数据时,传统数据库往往会出现数据库溢满、可靠性低及可扩展性低等问题。分布式云存储技术采取了Map/Reduce的并行思路,主要用于海量数据的存取和管理,它具有可扩展、低成本、高性能等优点,能有效解决传统数据库引发的问题。对于风电系统的每台服务器,由于用户访问量增大,如果每次都向原始数据库发送访问请求,同样会加重计算机节点的负载,增加用户请求的响应时间,影响系统性能。分布式缓存技术是在经济开销成本最低的情况下提高数据查询速度的方法。本文结合四川省某风电系统风功率预测项目的实际需求,针对项目中数据库服务器端的访问负载量和用户请求的延迟等待时间等问题,展开了分布式缓存系统的研究。着重对缓存系统的数据分布和替换算法进行了改进,并完成了整个系统的设计。本文所做的工作主要包括:(1)分析了分布式云储存的相关理论知识,包括分布式文件系统、分布式搜索查询技术、并行处理、数据缓存及负载平衡技术。(2)对分布式缓存系统的整体框架和各个模块进行深入研究。重点分析了拓扑结构、数据分布、缓存同步、缓存替换算法以及通信协议,结合风电系统的特征,为风电场数据的有效管理和存储设计出一套较为合理的方案。(3)完成了分布式缓存系统的实现,首先在QT环境下搭建系统的整体构架平台,然后实现各模块的功能。着重针对数据分布模式和缓存替换算法进行了改进,为解决一致性哈希算法数据分布不均匀的问题,引入“虚拟节点”对其进行改进,在现有的替换算法基础上,设计了基于综合因素的替换算法FST(Frequency,Object Size,Access Time),解决了传统算法考虑因素单一、命中率低下等问题。
[Abstract]:Under the background of the rapid development of global economy, the exploitation of energy is increasing, and the environmental pollution is becoming more and more serious. In order to ensure the long-term and effective development of national economy, the development and use of new energy is one of the most effective methods. Wind energy is the renewable energy with the greatest potential in large-scale development and utilization. With the increase of installed capacity, more and more users, more and more demand for storage space, data growth presents an unimaginable trend. However, in the face of large amounts of data, traditional databases often appear the problems of overflowing, low reliability and low scalability. Distributed cloud storage technology adopts the parallel idea of Map/Reduce, which is mainly used in the access and management of massive data. It has the advantages of scalability, low cost and high performance, and can effectively solve the problems caused by traditional databases. For each server of wind power system, if the access request is sent to the original database every time, it will also increase the load of the computer node, increase the response time of the user request, and affect the performance of the system. Distributed cache technology is a method to improve the speed of data query with the lowest cost. According to the actual demand of wind power prediction project in a wind power system in Sichuan province, the distributed cache system is studied in this paper, aiming at the problems of the access load of the database server and the delay waiting time of the user request in the project. The data distribution and replacement algorithm of cache system is improved, and the design of the whole system is completed. The main work of this paper is as follows: (1) the related theoretical knowledge of distributed cloud storage is analyzed, including distributed file system, distributed search and query technology, parallel processing. Data caching and load balancing technology. (2) the whole framework and each module of distributed cache system are deeply studied. The topology, data distribution, cache synchronization, cache replacement algorithm and communication protocol are analyzed. A more reasonable scheme is designed for the effective management and storage of wind farm data. (3) the implementation of distributed cache system is completed. Firstly, the whole architecture platform of the system is built in QT environment, and then the function of each module is realized. This paper focuses on improving the data distribution pattern and cache replacement algorithm. In order to solve the problem of uneven data distribution in the consistent hash algorithm, "virtual node" is introduced to improve it, based on the existing replacement algorithm. A synthetic factor based replacement algorithm FST (Frequency,Object Size,Access Time),) is designed to solve the problems of single factor and low hit rate in the traditional algorithm.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【相似文献】