基于云计算的影视大数据处理关键问题研究与实现
发布时间：2019-05-24 17:42
【摘要】：随着微博、社交网络、移动APP、基于位置的服务等技术的兴起,全球数据出现了爆炸式的增长。大数据为数据的挖掘和分析提供了丰富的来源,从中挖掘出来的结果将变得更有意义甚至充满惊喜。然而面对大数据传统的处理方法和处理能力都渐渐变得力不从心。如何高效的进行大数据的处理已经成为各行各业争相研究的热点问题。本文从影视大数据智能分析系统建设过程中遇到的大数据处理的问题出发,分析研究了现有的数据处理方案,结合了云计算和大数据处理的相关处理技术,提出了影视大数据处理策略及方法。主要研究主要内容包括以下几个方面,首先对云计算的概念和相关技术进行研究。选定了 Hadoop开源云平台作为项目研究的基础平台,深入研究了 MapReduce分布式编程模型以及HDFS分布式文件存储系统。其次深入分析了影视数据各部分的业务需求,为应对不同的数据处理需求,充分利用关系型数据库和Hadoop分布式平台的优点,提出了两种数据处理方案。一种基于关系型数据库的单节点数据处理,一种基于Hadoop的分布式数据处理。最后根据影视大数据智能分析系统不同的数据特点和不同的数据处理需求,选择合适的数据处理方案进行数据处理流程的设计和实现,并成功的应用于影视大数据的处理中,为影视大数据分析系统提供了可靠的数据支持。
[Abstract]:With the rise of technologies such as Weibo, social networks and mobile APP, location-based services, there has been explosive growth in global data. Big data provides a rich source for data mining and analysis, from which the results will become more meaningful and even full of surprises. However, in the face of big data's traditional treatment methods and processing capacity have gradually become unable to do so. How to deal with big data efficiently has become a hot issue in various industries. Starting from the problems encountered by big data in the construction of big data intelligent analysis system for film and television, this paper analyzes and studies the existing data processing schemes, and combines the related processing technologies of cloud computing and big data processing. The treatment strategy and method of big data in film and television are put forward. The main research contents include the following aspects, first of all, the concept of cloud computing and related technologies are studied. The Hadoop open source cloud platform is selected as the basic platform of project research, and the MapReduce distributed programming model and HDFS distributed file storage system are deeply studied. Secondly, the business requirements of each part of film and television data are deeply analyzed. In order to meet the different data processing requirements and make full use of the advantages of relational database and Hadoop distributed platform, two data processing schemes are proposed. A single node data processing based on relational database and a distributed data processing based on Hadoop. Finally, according to the different data characteristics and different data processing requirements of the film and television big data intelligent analysis system, the appropriate data processing scheme is selected to design and implement the data processing flow, and it is successfully applied to the processing of film and television big data. It provides reliable data support for the film and television big data analysis system.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13
本文编号：2485057
[Abstract]:With the rise of technologies such as Weibo, social networks and mobile APP, location-based services, there has been explosive growth in global data. Big data provides a rich source for data mining and analysis, from which the results will become more meaningful and even full of surprises. However, in the face of big data's traditional treatment methods and processing capacity have gradually become unable to do so. How to deal with big data efficiently has become a hot issue in various industries. Starting from the problems encountered by big data in the construction of big data intelligent analysis system for film and television, this paper analyzes and studies the existing data processing schemes, and combines the related processing technologies of cloud computing and big data processing. The treatment strategy and method of big data in film and television are put forward. The main research contents include the following aspects, first of all, the concept of cloud computing and related technologies are studied. The Hadoop open source cloud platform is selected as the basic platform of project research, and the MapReduce distributed programming model and HDFS distributed file storage system are deeply studied. Secondly, the business requirements of each part of film and television data are deeply analyzed. In order to meet the different data processing requirements and make full use of the advantages of relational database and Hadoop distributed platform, two data processing schemes are proposed. A single node data processing based on relational database and a distributed data processing based on Hadoop. Finally, according to the different data characteristics and different data processing requirements of the film and television big data intelligent analysis system, the appropriate data processing scheme is selected to design and implement the data processing flow, and it is successfully applied to the processing of film and television big data. It provides reliable data support for the film and television big data analysis system.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13
【参考文献】
相关期刊论文 前10条
1 刘圆;王峰;杨明川;;面向大数据的分布式存储技术研究[J];电信技术;2015年06期
2 程学旗;靳小龙;王元卓;郭嘉丰;张铁赢;李国杰;;大数据系统和分析技术综述[J];软件学报;2014年09期
3 唐世庆;李云龙;田凤明;胡海荣;;基于Hadoop的云计算与存储平台研究与实现[J];四川兵工学报;2014年08期
4 冯登国;张敏;李昊;;大数据安全与隐私保护[J];计算机学报;2014年01期
5 陈忻;;畅游数据海洋[J];中国建设信息;2013年06期
6 张松林;;云计算的核心技术与应用实例[J];电子世界;2013年05期
7 张耀祥;;云计算和虚拟化技术[J];计算机安全;2011年05期
8 詹洪文;;云计算核心技术及其产业化浅析[J];硅谷;2011年06期
9 李成华;张新访;金海;向文;;MapReduce:新型的分布式并行计算编程模型[J];计算机工程与科学;2011年03期
10 胡云;;对云计算技术及应用的研究[J];电脑开发与应用;2011年03期
相关硕士学位论文 前1条
1 丛中昌;基于云计算平台的电信经营分析系统中海量数据处理研究[D];南京邮电大学;2011年
，本文编号：2485057
本文链接：https://www.wllwen.com/kejilunwen/ruanjiangongchenglunwen/2485057.html
