融合通信系统中分布式存储引擎的设计与实现

发布时间：2018-03-03 09:53

本文选题：融合通信　切入点：分布式　出处：《中国科学院大学(中国科学院沈阳计算技术研究所)》2017年硕士论文　论文类型：学位论文

【摘要】：互联网OTT业务和移动互联网应用得到了快速的普及,使人们对信息获取的方式开始变得多元化,对通信服务的质量要求也越来越高,传统意义上基于语音通话或者短信技术层面上的通信业务已经无法满足人们的日常需求,通信业务也逐渐向包含语音、视频等的多媒体通信方向发展,因此而形成的将传统通信技术与互联网信息技术相融合的融合通信技术,成为当下计算机应用领域的一个研究热点。在融合通信系统数据存取方面,需要对即时消息、文档、组织架构等关键数据提供完善的存储引擎机制,而传统的基于关系数据库或者基于传统文件系统的存储方式,在存储数据安全性、获取数据效率以及后续数据挖掘与分析等方面都存在不满足的情况,因此亟需一种能够更好的满足需求的存储服务模式。随着Hadoop技术和Hadoop相关子系统的发展成熟,分布式存储的优势日益明显,本文在分析HDFS、MapReduce并行计算框架以及HBase/Hive体系结构和各自特点的基础之上,提出一种基于HBase-Hive集成设计的存储引擎设计方案,以此来满足融合通信系统对数据安全性、数据获取实时性和可靠性等方面的要求,同时充分研究数据挖掘的基础理论以及K-means、PAM聚类算法,结合MapReduce并行计算模型设计并实现了改进型K-means聚类算法,以此作为融合通信对数据挖掘需求的解决方案。在论文结构上,本文首先详细分析了课题研究的背景、现状、意义和相关基础技术,结合PAM算法改进K-means算法,然后设计并实现了分布式存储引擎的各个功能模块,最后通过对比试验进行功能和性能测试以及针对算法的仿真实验,验证了分布式存储引擎在融合通信系统中的可行性和合理性。
[Abstract]:Internet OTT services and mobile Internet applications have been rapidly popularized, making people become more and more diverse in the way of obtaining information, and the quality of communication services is becoming more and more demanding. The traditional communication service based on voice call or short message technology has been unable to meet the daily needs of people, and the communication service has gradually developed towards multimedia communication including voice, video and so on. Therefore, the fusion communication technology that combines traditional communication technology with Internet information technology has become a research hotspot in the field of computer application. In the aspect of data access of fusion communication system, instant messaging and documents are needed. Organization structure and other key data provide perfect storage engine mechanism, while traditional storage methods based on relational database or traditional file system are used to store data security. Data acquisition efficiency and subsequent data mining and analysis are not satisfied, so it is urgent to have a storage service mode that can better meet the requirements. With the development of Hadoop technology and Hadoop related subsystems, The advantages of distributed storage are becoming more and more obvious. Based on the analysis of HDFS MapReduce parallel computing framework, HBase/Hive architecture and their respective characteristics, a storage engine design scheme based on HBase-Hive integrated design is proposed. In order to meet the requirements of data security, real-time and reliability of data acquisition, the basic theory of data mining and K-means-PAM clustering algorithm are studied. Combined with MapReduce parallel computing model, the improved K-means clustering algorithm is designed and implemented as a solution to the data mining requirements of fusion communication. In the structure of the thesis, the background and present situation of the research are analyzed in detail. Significance and related basic technology, combined with PAM algorithm to improve K-means algorithm, and then designed and implemented each functional module of the distributed storage engine. Finally, the function and performance of the distributed storage engine were tested by contrast experiment and the simulation experiment for the algorithm was carried out. The feasibility and rationality of distributed storage engine in fusion communication system are verified.
【学位授予单位】：中国科学院大学(中国科学院沈阳计算技术研究所)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】