面向移动终端的云存储性能优化的研究与实现

发布时间：2018-04-04 06:47

本文选题：移动终端　切入点：云存储　出处：《南京邮电大学》2017年硕士论文

【摘要】：随着移动终端的发展进步,终端数据飞速增长,各类照片、视频、音乐等文件将终端填满,所以终端用户对于机器存储性能的需求越来越大。因为用户数据的本地存储难以满足广大用户的要求,所以提出了本文研究方向,即将用户的本地数据放到云端存储。在进行了全面的学习后发现,在面向终端的云存储中有着NameNode内存消耗严重和缓存的选择策略不理想的问题,这些问题影响着到用户的体验。为了解决存受限问题,使用云存储技术作为终端用户数据的一个存储媒介,运用Eclipse、JDK等开发工具,调用HDFS API,使得终端上的数据能够直接上传到HDFS集群中进行存储,减轻终端的负担,并且能够随时下载到本地使用,简单的达到终端和HDFS集群之间的交互操作。HDFS集群的存储对于移动数据内容支持并不很理想,所以本文在分析了现有的几种小文件优化机制的基础上提出一种小文件处理方案,在移动终端将数据存放在HDFS的过程中添加一个处理模块,来对移动终端上传的文件进行判断处理,将满足小文件判断条件的文件进行合并操作,优化存储结构。在进行对比实验后发现改进方案能够显著减少存储大量小文件时NameNode内存的消耗。针对HDFS集群中的的缓存选择策略不适合移动终端数据存储问题。本文在分析了终端数据结构的基础上,发现使用缓存能够显著的降低系统本身的读写操作,降低操作时间。本文在分析了各种缓存机制后,提出一种缓存选择算法,将此算法应用于HDFS系统中。最后得出改进策略对于文件的读取效率提高显著,达到了优化的目的。
[Abstract]:With the development and progress of mobile terminal, the terminal data grows rapidly, and all kinds of photos, video, music and other files will fill the terminal, so the demand of the end user for the storage performance of the machine is increasing.Because the local storage of user data is difficult to meet the requirements of the majority of users, the research direction of this paper is put forward, that is, the local data of the user is stored in the cloud.After a comprehensive study it is found that there are serious problems of NameNode memory consumption and poor cache selection strategy in the terminal oriented cloud storage which affect the user's experience.In order to solve the problem of storage limitation, cloud storage technology is used as a storage medium for end-user data, and HDFS API is called by using Eclipse JDK and other development tools, so that the data on the terminal can be directly uploaded to the HDFS cluster for storage.Reduce the burden on the terminal, and can be downloaded to local use at any time, simply to achieve the terminal and HDFS cluster of interactive storage. HDFS cluster for mobile data content support is not ideal,Therefore, based on the analysis of several existing optimization mechanisms of small files, this paper proposes a small file processing scheme, which adds a processing module in the process of the mobile terminal storing data in HDFS.To judge and process the files uploaded by the mobile terminal, the files that meet the judgment condition of small files are merged and the storage structure is optimized.The comparison experiments show that the improved scheme can significantly reduce the NameNode memory consumption while storing a large number of small files.The cache selection strategy in HDFS cluster is not suitable for mobile terminal data storage.Based on the analysis of the terminal data structure, it is found that using cache can significantly reduce the read and write operations and the operation time of the system itself.After analyzing various cache mechanisms, this paper proposes a cache selection algorithm, which is applied to HDFS system.Finally, it is concluded that the improved strategy improves the efficiency of file reading significantly and achieves the purpose of optimization.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】