分布式视频流存储元数据服务器的设计与实现

发布时间：2019-01-26 19:34

【摘要】：近年来公共安全威胁愈发严峻,将安防监控技术应用到公共安全中刻不容缓。现有安防监控领域对数据的存储普遍采用普通分布式存储系统架构。然而,这样的存储架构下,系统使用多副本策略保证数据的可靠性,存储空间利用率低;此外,多数系统使用轮询调度算法调度存储节点,忽略了存储节点的负载均衡性问题,导致节点性能抖动较大;而且,在数据丢失恢复任务中,没有考虑视频流存储的有效时长等因素,数据恢复价值较低。本系统将分布式存储架构技术与安防监控行业相结合,设计与实现了元数据服务器,旨在提高监控数据的可靠性和系统可扩展性,在接入互联网的情况下,支持用户在合法权限内,获取任意摄像头的任意时间的数据。元数据服务器结合时间戳策略,设计了基于纠删码的分布式存储系统和卷式元数据管理结构,实现了对存储任务和恢复任务的均衡调度。具体研究内容如下:1.设计和实现了时间戳策略来维护存储系统中不同服务器之间卷信息元数据的一致性。存储系统中的每个服务器同时记录系统中所有卷数据的时间戳信息,以元数据服务器上的卷时间戳信息为系统唯一参考标准,判断其余模块中卷信息元数据的有效性。2.设计和实现了以节点和卷为索引的两个维度的元数据管理结构。元数据服务器维护每个存储节点的系统健康状态信息,为任务的均衡调度提供参考;同时,记录每个存储节点上数据分布情况和卷式数据的纠删码策略相关信息,为数据丢失后的恢复任务提供依据。同时,支持系统节点的动态扩展。3.设计和实现了基于多目标优化算法的任务调度策略。针对存储任务的分发,目标函数不仅保证节点的负载均衡,而且让不同节点之间的执行任务数量差值最小;针对恢复任务优先级的制定,目标函数优先选择数据冗余率低、丢失率高、视频剩余有效时间长的卷,提高系统的可靠性。通过对分布式视频流存储系统进行功能和性能测试,元数据服务器达到了对视频流数据存储、读取和恢复等功能的控制管理需求;另外,存储任务和恢复任务的调度策略,不仅降低了存储节点的负载失衡机率,而且延迟了低效恢复任务的执行,提高了系统恢复数据的后期使用价值。
[Abstract]:In recent years, the threat of public security is becoming more and more serious, so it is urgent to apply security monitoring technology to public safety. General distributed storage system architecture is widely used in data storage in the field of security monitoring and control. However, in this storage architecture, the system uses multi-replica strategy to ensure the reliability of data, and the utilization of storage space is low. In addition, most systems use polling scheduling algorithm to schedule storage nodes, ignoring the load balancing problem of storage nodes, resulting in greater performance jitter. Moreover, in the task of data loss recovery, the effective time of video stream storage is not taken into account, so the value of data recovery is low. This system combines distributed storage architecture technology with security monitoring industry, and designs and implements metadata server, which aims to improve the reliability of monitoring data and system scalability, in the case of access to the Internet, Allows users to access arbitrary camera data at any time within legitimate permissions. The distributed storage system based on erasure code and the volume metadata management structure are designed in the metadata server combined with the timestamp strategy. The balanced scheduling of storage and recovery tasks is realized. The specific research contents are as follows: 1. A timestamp strategy is designed and implemented to maintain the consistency of volume information metadata between different servers in the storage system. Each server in the storage system simultaneously records the timestamp information of all volume data in the system. The volume timestamp information on the metadata server is regarded as the unique reference standard of the system to judge the validity of the volume information metadata in the other modules. 2. A metadata management structure based on nodes and volumes is designed and implemented. The metadata server maintains the system health status information of each storage node and provides a reference for the balanced scheduling of tasks. At the same time, the data distribution on each storage node and the erasure policy of volume data are recorded to provide the basis for the recovery task after data loss. At the same time, support the dynamic expansion of system nodes. 3. The task scheduling strategy based on multi-objective optimization algorithm is designed and implemented. For the distribution of storage tasks, the objective function not only ensures the load balance of nodes, but also minimizes the difference in the number of execution tasks between different nodes. In order to determine the priority of the recovery task, the objective function first selects the volume with low data redundancy rate, high loss rate and long video residual effective time to improve the reliability of the system. By testing the function and performance of the distributed video stream storage system, the metadata server achieves the control and management requirements for the video stream data storage, reading and recovery functions. In addition, the scheduling strategy of storage task and recovery task not only reduces the load imbalance probability of storage node, but also delays the execution of inefficient recovery task, and improves the later application value of system recovery data.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TN948.6;TP333

【参考文献】