基于Hadoop的大数据平台设计与实现及在推荐系统中的应用

发布时间：2018-04-13 18:28

本文选题：数据爆炸 + 大数据平台　；参考：《北京邮电大学》2016年硕士论文

【摘要】：近年来,随着移动通信的发展,尤其是3G/4G移动通信网络的迅速扩张以及云计算技术的快速发展,各种互联网技术不仅仅局限于PC端,智能设备终端以及各种云服务的迅速膨胀。随之而来就是数据的井喷式大爆炸,在大数据时代,我们必须想方设法存储和分析这些数据。但是目前对于任何单独的硬件都无法存储和分析这些超大数据,因此,基于该问题以分布式系统为基础,本论文旨在构建一个能够采集、预处理、存储、分析计算的并且可以进行资源监控分析运行状态的大数据平台,并基于此平台做一些推荐应用的简单研究测试。本论文针对超大规模数据的存储、分析、应用以及可监控统计分析工作流的平台需求,提出一种基于Hadoop生态系统及其组件的集群平台构建模型。该模型是针对超大数据量的存储分析而提出的,由数据采集预处理系统、数据存储系统、数据计算分析系统以及平台资源统计分析系统构成。数据采集预处理系统是由基于分布式消息系统的Kafka集群搭建而成,该系统既可以将数据发往基于Hadoop的离线处理系统又可以发往基于Spark的准实时处理系统。数据存储系统由基于Hadoop的HDFS分布式文件系统和HBase分布式数据库组成。数据计算分析系统是由MapReduce和以MapReduce为底层计算模型的Hive组成。平台资源统计分析系统则对Hadoop大数据平台集群的计算资源和存储资源的统计分析进行展示,即可以实时监控集群运行状态,也可以根据系统的分析模块得出最近几天或者当天的平台集群的运行状况,指导用户使用集群各种服务、调试和排查作业问题以及优化作业执行过程。最后,结合Mahout库对数据平台在推荐系统上的应用进行了研究测试。
[Abstract]:In recent years, with the development of mobile communication, especially the rapid expansion of 3G/4G mobile communication network and the rapid development of cloud computing technology, all kinds of Internet technologies are not limited to PC, smart device terminals and various cloud services.Then came the blowout explosion of data, which must be stored and analyzed in big data's time.However, for any single hardware, it is impossible to store and analyze these super-large data. Therefore, based on the distributed system, this paper aims to build a system that can collect, preprocess and store these data.Big data platform, which can be used to monitor and analyze the running state of resources, is analyzed and calculated, and based on this platform, some simple research and tests of recommended applications are made.In this paper, a cluster platform model based on Hadoop ecosystem and its components is proposed to meet the platform requirements of large-scale data storage, analysis, application and monitoring statistical analysis workflow.The model is proposed for the storage and analysis of large amount of data. It is composed of data acquisition and preprocessing system, data storage system, data calculation and analysis system and platform resource statistics and analysis system.The data acquisition and preprocessing system is built by Kafka cluster based on distributed message system. The system can send data to both off-line processing system based on Hadoop and quasi-real-time processing system based on Spark.The data storage system is composed of HDFS distributed file system based on Hadoop and HBase distributed database.The data calculation and analysis system is composed of MapReduce and Hive with MapReduce as the underlying computing model.The platform resource statistics and analysis system shows the statistical analysis of the computing and storage resources of the Hadoop big data platform cluster, that is, it can monitor the running state of the cluster in real time.According to the analysis module of the system, we can get the running status of the platform cluster in the last few days or that day, and guide the users to use various services of the cluster, debug and check the job problems and optimize the process of job execution.Finally, the application of data platform in recommendation system is studied and tested with Mahout library.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13;TP391.3

【相似文献】