面向时间序列的流式对象存储文件系统的设计与实现

发布时间：2018-11-13 06:34

【摘要】：在生活中,视频监控网、雷达基站、股票市场等都会实时产生大量独立、连续的数据流,这种随时间延续而无限增长的动态数据集合就是流数据。而且随着经济的发展,各行业的流数据的并发量和流量越来越大,尤其是视频安防监控领域,传统的存储系统由于受到其物理组成以及功能上的局限,已不能满足海量数据存储的需要。高效能的存储流数据具有重要的现实意义。为了存储规模庞大的流数据,基于现有的软硬件技术,适合采用分布式存储系统。分布式存储系统,是将数据分散存储在多台独立设备上的存储系统。分布式存储系统具有高可扩展、低成本、高性能和易用等优势。专用于保存流数据的分布式流存储系统不同于其他分布式存储系统,流数据时源源不断的快速连续到达,对网络和磁盘形成巨大的负载压力,写任务多于读,系统中存储的旧数据几乎永不变更但是会在一段时间后过期。由于流数据的时间相关性,流数据存储中通常用时间戳来标定对应的流数据,因此具备单调递增的时间序列特征,可用于后期的基于时间戳或访问的数据定位及读取。此外,流数据会存在存储生命周期的需求,例如针对监控存储的原始数据,会存在一个固定的过期时间。针对这种实际的情况,本文讨论了一种用在分布式视频流存储系统中的数据存储服务器上使用的面向时间序列的流式对象存储文件系统的设计与实现,以满足高写入压力的存储需求。该文件系统,主要包括如下3个重要的功能点:1、实现了仿文件系统语义供存储系统开发者做二次开发;2、高并发高吞吐读写流数据,同时保持良好的可靠性;3、通过多磁盘的优化调度,在满足系统读写任务的需求前提下,降低系统能耗,延长磁盘使用寿命,从而达到节能环保。
[Abstract]:In daily life, video surveillance network, radar base station, stock market and so on will produce a large number of independent and continuous data streams in real time. The dynamic data set that grows infinitely with time is stream data. And with the development of economy, the concurrent amount and flow of stream data in various industries are increasing, especially in the field of video security monitoring, the traditional storage system is limited by its physical composition and function. Can no longer meet the need for massive data storage. High performance storage stream data has important practical significance. In order to store large-scale stream data, distributed storage system is suitable based on the existing software and hardware technology. Distributed storage system is a storage system that distributes data on multiple independent devices. Distributed storage systems have the advantages of high scalability, low cost, high performance and easy to use. Unlike other distributed storage systems, distributed stream storage systems that are dedicated to storing streaming data arrive at a constant and rapid rate, creating a huge load on the network and disk, and writing more tasks than reading. The old data stored in the system almost never changes but will expire after a period of time. Due to the temporal correlation of stream data, time stamp is usually used to calibrate the corresponding stream data in stream data storage, so it has monotone increasing feature of time series and can be used to locate and read data based on time stamp or access. In addition, stream data will have a storage lifecycle requirement, such as a fixed expiration time for monitoring stored raw data. In view of this practical situation, this paper discusses the design and implementation of a time series oriented streaming object storage file system, which is used on a data storage server in a distributed video stream storage system. To meet high write pressure storage requirements. The file system mainly includes the following three important function points: 1, the semantic of the file system is realized for the storage system developer to do secondary development, 2, high concurrency, high throughput, read and write stream data, while maintaining good reliability; 3. By optimizing the scheduling of multi-disks, the energy consumption of the system is reduced, and the service life of the disk is prolonged under the premise of satisfying the task of reading and writing of the system, so as to achieve energy saving and environmental protection.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】