视频云存储服务场景下的HDFS负载均衡工具

发布时间：2018-05-02 20:48

本文选题：在线视频服务 + 云存储　；参考：《小型微型计算机系统》2017年02期

【摘要】：在线视频服务是互联网服务的重要内容,存储是在线视频服务提供的基础.HDFS作为面向通用文件的云存储系统,被很多视频服务网站采用,但其负载均衡工具没有考虑利用视频文件在线播放时的带宽消耗特性使集群的带宽资源得到更充分的利用.为解决这一问题,提出视频存储场景下的负载均衡方法 VOBM,它对视频文件在线播放时的带宽消耗与视频文件的码率、数据块大小和访问热度的关系进行了分析并建立了新的负载评估模型,在此基础上它在负载方案生成和负载调度两个环节中加入了对带宽消耗因素的考虑.在HDFS原有负载均衡工具的基础上实现了该方法,实验证明该方法能够有效避免高带宽消耗数据块的聚集,在高带宽消耗视频文件作为服务访问热点的实验场景中,该方法在90%的场景中优于原有负载均衡方法,最高能使数据节点集群中瓶颈节点的带宽峰值降低20%.
[Abstract]:Online video service is an important part of Internet service. Storage is the basis of online video service. HDFS, as a cloud storage system oriented to common files, is adopted by many video service websites. However, its load balancing tool does not take into account the bandwidth consumption when video files are played online so that the bandwidth resources of the cluster can be utilized more fully. In order to solve this problem, a load balancing method called VOBM in video storage scenario is proposed, which consumes the bandwidth and the bit rate of the video file when the video file is played online. The relationship between data block size and access heat is analyzed and a new load evaluation model is established. On this basis, the bandwidth consumption factor is considered in the load scheme generation and load scheduling. The method is implemented on the basis of the original load balancing tool of HDFS. Experiments show that the method can effectively avoid the aggregation of high bandwidth consuming data blocks, and in the experimental scenario where high bandwidth consuming video files are used as service access hotspots. This method is superior to the original load balancing method in 90% scenarios, and can reduce the peak bandwidth of bottleneck nodes in the data node cluster by 20%.
【作者单位】：浙江大学计算机科学与技术学院;浙江省现代服务业电子服务工程技术研究中心;
【基金】：国家自然科学基金项目(61272129)资助国家“八六三”高技术研究发展计划项目(2013AA01A213)资助教育部优秀人才计划项目(NCET-12-0491)资助浙江省杰出青年科学基金项目(LR13F020002)资助
【分类号】：TP333

【相似文献】