基于Hadoop的广电监测数据管理云平台设计

发布时间：2018-05-19 08:03

本文选题：Hadoop云平台 + 分布式存储管理　；参考：《电子科技大学》2015年硕士论文

【摘要】：随着计算机技术的高速发展,互联网行业进入了大数据的时代,每天都会产生许许多多的各类数据。广电系统的监测设备同样也会产生越来越多各个类型的数据,然而传统的广电数据存储与管理系统采用集中处理的方式,在巨大数据的冲击下出现了瓶颈。海量数据的存储与管理在近些年一直在被研究。在许多开源社区的推动下,分布式数据的存储管理技术得到巨大的发展,多种多样的计算框架以及平台被提出来。Hadoop作为分布式计算平台中的翘楚在通信、医疗和金融等各大领域都有了一定的研究和应用。广电监测系统产生的数据主要是结构化的文本数据和视频音频信号等非结构化数据。由于需要解决大量数据的处理,符合大数据平台处理标准。所以本文将Hadoop这种分布式数据存储与管理的技术运用到广电系统中以解决数据计算的瓶颈问题。本文主要研究的内容如下:1、对HDFS文件系统和MapReduce并行计算框架进行分析,以及对文本数据搜索和视频数据的转码进行研究。根据广电系统的数据处理需求,提出一个系统设计的解决方案。2、为解决平台对广电监测系统监测数据的传输、处理和节点信息监控等问题,本文主要研究了海量数据存储和管理平台的数据节点信息管理及优化、数据上传下载的转存和传输模块设计、文本数据的搜索和监控视频数据的分布式转码和存储。最后对本文设计的海量数据管理系统进行自动化数据测试,通过对数据存储和管理平台的各类数据存储、数据上传和下载速度、文本数据的搜索和视频数据的分布式转码测试和分析。可以得到实验结果:Hadoop云平台应用到广电系统是可行的,同时通过分析得到的实验数据得出结论,云平台对大数据的存储和管理性能较原有数据管理平台有较大的提升。
[Abstract]:With the rapid development of computer technology, the Internet industry has entered the era of big data, producing many kinds of data every day. The monitoring equipment of the radio and television system will also produce more and more kinds of data. However, the traditional radio and television data storage and management system adopts the centralized processing method, which appears the bottleneck under the impact of the huge data. The storage and management of massive data has been studied in recent years. Driven by many open source communities, distributed data storage and management technology has been greatly developed. A variety of computing frameworks and platforms have been proposed. Hadoop as the leader of the distributed computing platform in communication, Medical and financial fields and other major areas have a certain degree of research and application. The data generated by the radio and television monitoring system are mainly structured text data and unstructured data such as video and audio signals. Due to the need to solve a large number of data processing, in line with the big data platform processing standards. In this paper, Hadoop, a distributed data storage and management technology, is applied to radio and television systems to solve the bottleneck of data computing. The main contents of this paper are as follows: 1. The HDFS file system and MapReduce parallel computing framework are analyzed, as well as text data search and video data transcoding. According to the data processing requirement of the radio and television system, this paper puts forward a system design solution. In order to solve the problems such as the transmission, processing and node information monitoring of the monitoring data of the radio and television monitoring system based on the platform, This paper mainly studies the data node information management and optimization of the massive data storage and management platform, the design of data upload and download transfer and transmission module, the search of text data and the distributed transcoding and storage of monitoring video data. Finally, the paper carries on the automatic data test to the massive data management system designed in this paper, through the data storage and the management platform each kind of data storage, the data uploads and downloads the speed, Text data search and distributed transcoding test and analysis of video data. It is feasible to apply the cloud platform to radio and television system. At the same time, by analyzing the experimental data, it is concluded that the storage and management performance of the cloud platform to big data is better than that of the original data management platform.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TN949.6

【相似文献】