一种适用于P2P-VoD业务的分布式存储系统

发布时间：2018-07-23 11:57

【摘要】：近年来,在线网络视频业务已逐步发展为网络音乐、网络新闻和即时通信以外的第四大类互联网应用,成为人们获取电影、电视、视频等数字内容的重要媒介。P2P视频点播业务作为目前主流的网络视频服务之一,拥有大规模的用户群体。随着互联网基础设施的大范围普及和影视媒体琳琅满目的发展,越来越多的用户选择使用视频点播服务来满足自己获取信息和娱乐的需求。然而,用户数量的不断增多和用户对热门节目需求的迅速增长同时也为在线网络视频服务提出了更加严格的要求。媒体数据量的急剧增长为视频点播系统带来了新的问题。首先,海量的媒体数据存储为视频点播系统带来了巨大的硬件开销；其次,为了维持系统的正常运转,系统的可用性及数据的可靠性需要得到保障；最后,为了满足大量用户同时使用系统服务,同时使用户获得更流畅的播放体验,需要尽可能的降低系统负载,提高系统性能。 P2P视频点播系统中的媒体源服务器负责存储系统中所有节目的完整数据。短时间内大量并发的数据请求对媒体源服务器造成巨大的压力,导致系统负载过高,资源占用率急剧升高,处理能力下降。另外,一些突发性的系统故障可能导致媒体源服务器宕机、系统崩溃甚至数据丢失,影响整个系统的正常运行,最终导致视频播放不流畅,用户体验下降。随着信息服务的日益深化,越来越多的数据要求进行长期的低成本存储,海量数据的存储和处理已成为当前数据中心面临的首要问题,数据的分布式存储及云计算技术迅速成为IT领域的研究热点。Hadoop分布式文件系统(HDFS)是Apache Hadoop的基础文件系统,它能够使用廉价的PC机集群构建的系统为海量数据提供高可靠性、高容错性的保障,同时能够为数据密集型应用提供高吞吐量的数据访问,非常适合于大规模数据集长期的低成本存储。本文针对P2P视频点播系统中媒体源服务器的种种问题,提出一项新的解决方案—基于HDFS的适用于P2P-VoD业务的分布式存储系统。该分布式存储系统利用HDFS的可扩展性及可靠性特点,同时在HDFS的数据节点上新增一项本地缓存的功能以改进系统负载性能。通过搭建真实实验环境,使用多种测试方法从功能和性能上验证了该系统的可用性、可靠性,同时验证了改进后的媒体源服务器在系统负载性能上得到了很大提升。
[Abstract]:In recent years, the online network video business has gradually developed into the fourth largest category of Internet applications other than network music, network news and instant messaging. As one of the mainstream network video services, P2P video-on-demand (VOD), an important medium of digital content such as video, has a large number of users. With the wide spread of Internet infrastructure and the development of video media, more and more users choose to use VOD services to meet their needs for information and entertainment. However, the increasing number of users and the rapid growth of users' demand for popular programs also put forward more stringent requirements for online video services. The rapid growth of media data has brought new problems to VOD system. First, mass media data storage brings huge hardware overhead for VOD system. Secondly, in order to maintain the normal operation of the system, the availability of the system and the reliability of the data need to be guaranteed. In order to satisfy a large number of users using system services at the same time, while enabling users to get a smoother playback experience, it is necessary to minimize the system load as much as possible. The media source server in P2P VOD system is responsible for storing the complete data of all programs in the system. A large number of concurrent data requests in a short period of time cause tremendous pressure on the media source server, resulting in excessive system load, a sharp increase in resource occupancy, and a decline in processing power. In addition, some sudden system failures may lead to media source server downtime, system crash or even data loss, affecting the normal operation of the entire system, resulting in the video playback is not smooth, user experience decline. With the deepening of information services, more and more data require long-term low-cost storage, mass data storage and processing has become the current data center facing the primary problem. Hadoop distributed file system (HDFS) is the basic file system of Apache Hadoop. It can provide high reliability for mass data by using cheap PC cluster system. The guarantee of high fault tolerance and the ability to provide high throughput data access for data-intensive applications are very suitable for the long-term low-cost storage of large data sets. Aiming at the problems of media source server in P2P VOD system, this paper presents a new solution, a distributed storage system based on HDFS for P2P-VoD services. The distributed storage system takes advantage of the extensibility and reliability of HDFS and adds a local cache to the data node of HDFS to improve the load performance of the system. The availability and reliability of the system are verified by using a variety of test methods in a real experimental environment. At the same time, it is verified that the improved media source server has greatly improved the load performance of the system.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【共引文献】