基于HADOOP的海量录音文件存储系统的设计与实现

发布时间：2018-04-18 10:23

本文选题：录音文件存储 + HDFS　；参考：《电子科技大学》2014年硕士论文

【摘要】：随着通信服务行业的快速发展,各企业的业务种类与服务类别也变得多样化,用户与企业客服人员的互动数量也快速增长。许多公司的客服压力也越来越大,公司为保证服务人员正常工作,以及防止恶意骚扰电话,同时为了保证客服人员按照客户的要求办理业务,通常会对客服通话进行录音保存。这些录音文件数量巨大,而且还处于快速动态增长中。但是它们对于公司后期数据分析,用于提升员工服务品质管理等非常重要,通常会长期保存。传统的存储方式处理这些数据效率低,工作量大,而且成本高。如何安全有效且低成本的存储这些数据是当前迫切需要解决的一个问题。本文主要研究了如何利用多种存储设备,构建分布式存储系统,用以解决海量录音文件存储问题。本文分析了国内云储技术发展的现状,通过调研分布式文件系统技术,再结合海量录音文件的存储需求,设计并实现了基于HADOOP的海量录音文件存储系统。该系统是以HDFS为存储基础开发的一个海量录音文件存储系统,系统由客户端、消息队列以及服务存储端组成。客户端主要提供友好的用户交互界面,消息队列是客户端与服务端的纽带,服务端实现了分布式存储并响应客户端的各种请求。该系统是典型的生产消费者模式,客户端发送请求到消息队列,服务端处理从消息队列获取的请求并响应客户端。该系统服务端是可动态扩展的,新增服务端程序时,服务程序只要监听到指定的消息队列,就可以接收并处理客户端的请求。以下介绍从需求到系统实现的步骤:第一,通过分析企业的海量录音文件存储的现状,针对传统录音文件存储方式的不足,本文进行分布式存储技术调研,提出了基于HADOOP的海量录音文件系统的需求;第二,结合海量录音文件存储的现状及需求,对海量录音文件存储系统进行了详细设计,主要包含用户权限鉴定设计、录音文件的上传下载设计、录音文件管理功能设计和录音文件分享设计等;第三,本文对分布式文件系统的环境搭建做了详细介绍,并实现了海量录音文件存储系统,包括用户权限鉴定功能实现、录音文件的上传下载功能实现、录音文件管理功能实现和录音文件的分享功能实现等。考虑到可移植性及安全性,系统用Java语言开发。本系统分为客户端、服务端、数据库和分布式存储系统。本系统界面简单整洁,方便用户的使用;第四,对系统的安全性、文件传输效率以及系统的功能做了测试,在录音文件存储方式上,本系统和传统系统相比,具有更好的安全性能、数据传输效率也有很大的提升、数据的存储成本也更低。
[Abstract]:With the rapid development of the communication service industry, the business types and service categories of the enterprises have become diversified, and the number of interaction between users and customer service personnel has also increased rapidly.Many companies are also under increasing customer service pressure, to ensure that service personnel work properly, and to prevent malicious harassment of phone calls, and to ensure that customer service personnel do business in accordance with customer requirements, customer service calls are usually recorded and preserved.These audio files are huge in number and still in rapid dynamic growth.But they are important for later data analysis, for improving employee quality management, and are often preserved for long periods of time.The traditional storage method is low efficiency, heavy workload and high cost.How to store these data safely and efficiently and at low cost is an urgent problem.This paper mainly studies how to use a variety of storage devices to build a distributed storage system to solve the problem of massive audio file storage.This paper analyzes the present situation of cloud storage technology in China, designs and implements the storage system of massive audio files based on HADOOP by investigating the distributed file system technology and combining the storage requirements of massive audio files.This system is a massive recording file storage system based on HDFS. The system consists of client, message queue and service storage.The client mainly provides a friendly user interface. Message queue is the link between the client and the server. The server realizes distributed storage and responds to various requests from the client.The system is a typical production consumer mode, the client sends the request to the message queue, the server processes the request from the message queue and responds to the client.The system server is dynamically extensible. When the server program is added, the service program can receive and process the request of the client as long as it listens to the specified message queue.The following are the steps from requirements to system implementation: first, by analyzing the current situation of mass audio file storage in enterprises, aiming at the shortcomings of traditional recording file storage methods, this paper investigates distributed storage technology.This paper puts forward the requirement of mass audio file system based on HADOOP. Secondly, combining with the current situation and demand of mass audio file storage, the paper designs the storage system of mass sound recording file in detail, which mainly includes the design of user authority authentication.Audio file upload and download design, recording file management function design and audio file sharing design. Thirdly, this paper introduces the environment of distributed file system in detail, and realizes the massive audio file storage system.Including the realization of user authority authentication function, audio file upload and download function realization, recording file management function realization and audio file sharing function realization and so on.Considering portability and security, the system is developed with Java language.The system is divided into client, server, database and distributed storage system.The interface of the system is simple and clean, and it is convenient for users to use. Fourthly, the security of the system, the efficiency of file transfer and the functions of the system are tested. In the storage mode of recording files, the system is compared with the traditional system.With better security performance, data transmission efficiency is also greatly improved, data storage costs are lower.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP333

【相似文献】