基于Hadoop的移动互联网数据导入系统的设计与实现

发布时间：2018-03-09 19:21

本文选题：移动互联网　切入点：网络数据　出处：《北京邮电大学》2014年硕士论文　论文类型：学位论文

【摘要】：随着移动互联网络基础建设步伐的加快,智能手机终端的飞速发展和网络应用的日益广泛,移动互联网络用户的数量正在飞速增长。移动互联网络正在成为获取信息的主要渠道。随之而来的是移动互联网络数据流量的爆炸式增长。这些均对移动网络运营商的网络平台规划和管理能力提出了新的要求。移动互联网络的用户行为也呈现出新的规律。因此,对于移动互联网络网络资源的使用状况,业务成分分析和用户行为特征的把握就有了极大的必要性。近年来,海量数据处理的需求量正在逐渐加大,分布式作为一种对计算机的存储和计算能力的有效整合方式也随之发展起来。而Hadoop作为现行的一种开源的、有效的分布式编程框架正在各项研究和项目中逐渐流行本文将移动互联网络数据的数据特征以及监测需求综合考虑,提出了基于Hadoop的移动互联网络数据的两种管理方式,分别是离线数据导入系统(DataLoader)和实时数据导入系统(LogUploader)。对应分别解决了实验室集群数据上传中大数据清理的问题和针对运营商的话单查询系统中海量话单数据的导入问题。两个系统均完成原始数据和Hadoop的对接,离线数据导入系统主要负责对已经形成文件的数据进行一些清洗和其他处理,上传至HDFS,为对这些数据的分析做必要的准备；它实现了由传统话单数据到HDFS的快速上传和处理,并且,它为实验室集群的数据导入提供了一个指导性的编程框架,为以后的数据导入需求提供了一种快速实现的方式。实时数据导入系统主要分布在网络监测设备中,实时得将网络产生的原始数据进行处理后上传至HDFS,并形成文件分片和BloomFilter索引结构,为后续的分析和查询需求提供支持；它利用了Hadoop系统的稳定性和一定的控制机制有效的保证了从数据采集到数据上传过程中数据的完整性。最后,本文还对这两个系统的测试进行了详细的介绍。
[Abstract]:With the rapid development of mobile Internet infrastructure, the rapid development of smart phone terminals and network applications are becoming increasingly widespread. The number of mobile internet users is growing rapidly. Mobile internet is becoming the main way to get information. Then comes the explosive growth of mobile internet traffic. The planning and management ability of business network platform has put forward new requirements. The user behavior of mobile Internet also presents new rules. For the use of mobile Internet network resources, business component analysis and user behavior characteristics of the grasp of great necessity. In recent years, the demand for massive data processing is gradually increasing. Distributed as an effective integration of computer storage and computing capabilities has also developed. Hadoop as an existing open source, effective distributed programming framework is gradually becoming popular in various research and projects. Considering the data characteristics and monitoring requirements of mobile Internet data, this paper proposes two management methods of mobile Internet data based on Hadoop. It is the offline data import system (DataLoader) and the real time data import system (LogUploader. respectively). The problem of big data cleaning in the data upload of laboratory cluster and the import of the massive volume of single data in the telephone list query system of the operator are solved respectively. Problem. Both systems complete the docking of raw data and Hadoop, The off-line data import system is mainly responsible for some cleaning and other processing of the data that has been formed and uploaded to the HDFS to make necessary preparations for the analysis of these data, and it realizes the rapid uploading and processing of the data from the traditional telephone bill to the HDFS. Moreover, it provides a guiding programming framework for the data import of the laboratory cluster, and provides a fast way to realize the data import requirements in the future. The real-time data import system is mainly distributed in the network monitoring equipment. The raw data generated by the network can be processed and uploaded to the HDFS in real time, and the file slicing and BloomFilter index structure can be formed to support the subsequent analysis and query requirements. It makes use of the stability of Hadoop system and a certain control mechanism to ensure the integrity of data from data acquisition to data upload. Finally, the test of the two systems is introduced in detail.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.01;TN929.5

【相似文献】