基于Hadoop的云转码系统研究及性能优化

发布时间：2018-01-16 04:24

本文关键词：基于Hadoop的云转码系统研究及性能优化　出处：《北京交通大学》2014年硕士论文　论文类型：学位论文

更多相关文章： 云计算 Hadoop 云转码 HDFS 负载均衡

【摘要】：摘要：目前,视频流量已经成为互联网的主要流量,各种视频应用层出不穷,从数字高清电视到IPTV。互联网用户使用视频应用的终端也日益多样化,从PC到手机。然而,不同的网络视频平台和终端支持的视频内容和格式,如编码格式、分辨率、帧率等参数不尽相同。为了满足不同平台和用户的视频服务需求,往往需要对视频进行转码,即进行相应的编码格式、分辨率和帧率等格式转换。视频转码是一项非常耗时耗资源的工作,随着视频数量的急剧增长,传统的单机或者集中式转码已经不能满足人们对效率和质量的要求。而云计算通过集中、分配资源可以提供强大的计算能力,并且有良好的扩展性和较高的容错能力。所以可以将视频转码工作转移到云计算平台上。采用云平台进行视频转码,不仅可以承受海量视频数据的存储、转码需求,同时由于云计算本身具有的资源聚集特性,取用方便,费用低廉。在众多的云计算平台中,Hadoop由于其开源特性,是目前应用最为广泛的云计算平台。本论文首先设计和实现了基于Hadoop的云转码系统。该系统利用MapReduce分布式机制进行视频转码。系统包括代理服务器,视频转码模块,Cache模块三大组件。代理服务器负责处理用户的视频服务请求,视频转码模块负责视频处理工作,Cache模块负责管理原视频和转码后的视频文件。接着,论文对所实现的转码系统的性能进行了测试和分析。比较该系统与单机的视频转码性能,测试分析了分段数量和分段大小对系统转码性能的影响,分析了各个阶段在系统执行过程中所占的时间比例。在系统的执行过程中,视频文件需要进行多次对HDFS进行读写,当前HDFS读数据时副本选择策略是选择离客户端网络拓扑距离最近的节点,当热门副本集中在同一节点或者一个机架内时,用户就会对有限的资源进行激烈的竞争,造成该节点或者该机架的负载大大增加,从而影响整个集群的性能。为了克服该不足,论文提出了基于负载均衡的副本选择策略,使用线性加权法定量描述节点的负载量,选择负载量最轻的节点作为读取节点。仿真实验表明,改进的算法有效减少了副本传输时间,增加了HDFS集群的吞吐率。
[Abstract]:Absrtact: at present, video traffic has become the main flow of the Internet, a variety of video applications emerge in endlessly, from digital HDTV to IPTV.Internet users using video applications are increasingly diverse. However, different network video platforms and terminals support video content and formats, such as encoding formats, resolution. Frame rate and other parameters are different. In order to meet the needs of different platforms and users, it is often necessary to transcode the video, that is, the corresponding coding format. Video transcoding is a very time-consuming and resource-intensive task, with the rapid growth of the number of videos. Traditional single-machine or centralized transcoding can not meet the requirements of efficiency and quality. Cloud computing can provide powerful computing power through centralized allocation of resources. And has good expansibility and high fault-tolerant ability, so we can transfer the work of video transcoding to cloud computing platform. Using cloud platform for video transcoding, not only can withstand the massive storage of video data. Transcoding requirements, at the same time due to cloud computing itself has the characteristics of resource aggregation, easy to use, low cost. Hadoop in many cloud computing platforms due to its open source features. Is the most widely used cloud computing platform. This paper first designs and implements a cloud transcoding system based on Hadoop. The system uses MapReduce distributed mechanism to transcode video. The system includes proxy server and video transcoding module. The proxy server is responsible for handling the user's video service request and the video transcoding module is responsible for the video processing. The Cache module is responsible for managing the original video and the video files after transcoding. Then, the performance of the transcoding system is tested and analyzed. The video transcoding performance of the system is compared with that of a single machine. The effects of the number of segments and the size of segments on the transcoding performance of the system are tested and analyzed. The time ratio of each stage in the process of system execution is analyzed. During the execution of the system, the video file needs to read and write the HDFS several times. The current replica selection strategy when HDFS reads the data is to select the node nearest to the client network topology. When the hot copy is concentrated in the same node or a rack, the user will compete for the limited resources, resulting in the load of the node or rack increased greatly. Therefore, the performance of the whole cluster is affected. In order to overcome this deficiency, a replica selection strategy based on load balancing is proposed, which uses linear weighted quantification to describe the load of nodes. The simulation results show that the improved algorithm can effectively reduce the copy transfer time and increase the throughput of the HDFS cluster.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TN919.81

【参考文献】