云文件同步系统关键技术研究与实现

发布时间：2019-01-04 18:20

【摘要】：随着金山快盘、Dropbox、EverNote等云存储应用快速兴起，云计算开始真正融入人们的日常生活。实现高可用的云存储应用的关键技术之一，是如何解决用户数据在多个设备之间的同步问题。随着用户规模的逐渐扩大，并发访问量递增，存储容量需求爆炸式增长，云存储系统势必面临极其严峻的考验。开发与合作是当今互联网的主题之一，云存储系统对外开放亦是大势所趋。本文设计并实现了一个安全可靠、可扩展的云存储子系统，称为云文件同步系统。并为该系统开发了云网盘客户端程序和设计了REST架构风格的开放API。此外，还对云环境下的文件同步算法这一关键技术做了深入的研究，提出了一种基于文件元数据集合的多设备文件同步算法。本文主要工作包含以下四部分。云文件同步系统的整体设计。通过引入负载均衡技术和MongoDB分布式集群技术，提高了该系统并发处理能力和带故障正常运行能力。整体设计能够满足安全可靠、可扩展的基本要求。多设备文件同步算法实现。在云环境下，文件同步参与者之间呈现星形的拓扑结构，中央节点是提供文件存储服务的云端，连接中央节点的是各种客户端。假设仅有一个客户端，那么通过比对该客户端或云端上次同步完成和本次同步开始这两个时间点的文件元数据集合，可以得到云端和客户端的文件变化，进而合并云端和客户端文件变化得到最终的文件变化，最后应用文件变化即可实现该客户端与云端的文件同步。此处的获取文件变化的过程描述起来很简单，实际情况处理起来却是异常棘手。不过，经过一些巧妙的抽象和反抽象后，整个获取文件变化的过程变得非常简便。进一步将情景抽象为同步时间独占、客户端与客户端关系确定为松耦合，使得每个客户端和云端的同步操作看起来都是独立的，从而可以将多个客户端与云端的文件同步问题都看成是之前假设的单个客户端与云端的文件同步问题。云文件同步系统的工程实现。主要工作包括基于Jersey框架的REST架构风格Web服务程序的开发和供用户使用的的云网盘Windows客户端程序的开发。云文件同步系统的部署与测试。部署指的是将该系统部署到实际生产环境曙光6000超级计算机上。该系统的所有数据最终都保存在ParaStor上，测试则指的是用基准测试工具YCSB对MongoDB和ParaStor协同工作时的性能测试。经过工程应用的实际检验，，该云文件同步系统基本上可以正常工作。
[Abstract]:With the rapid rise of cloud storage applications such as Jinshan disk, Dropbox,EverNote and so on, cloud computing begins to truly integrate into people's daily life. One of the key technologies to implement high availability cloud storage applications is how to solve the problem of synchronization of user data between multiple devices. With the expansion of the scale of users, the increasing number of concurrent visits and the explosive growth of storage capacity, cloud storage system is bound to face a severe test. Development and cooperation is one of the themes of the Internet today, cloud storage system opening to the outside world is also the trend. This paper designs and implements a secure and extensible cloud storage subsystem called cloud file synchronization system. The client program of cloud network disk and the open API. with REST architecture style are developed for the system. In addition, the key technology of file synchronization algorithm in cloud environment is deeply studied, and a multi-device file synchronization algorithm based on file metadata set is proposed. The main work of this paper consists of the following four parts. The whole design of cloud file synchronization system. By introducing load balancing technology and MongoDB distributed cluster technology, the ability of concurrent processing and normal operation with faults is improved. The whole design can meet the basic requirements of safety, reliability and extensibility. Implementation of multi-device file synchronization algorithm. In the cloud environment, the file synchronization participants present star topology. The central node is the cloud that provides file storage service, and the central node is connected by various clients. Assuming that there is only one client, you can get the file changes between the cloud and the client by comparing the collection of file metadata between the last synchronization completion of that client or cloud and the start of this synchronization Then the final file changes can be obtained by merging the changes of the cloud and client files. Finally, the file synchronization between the client and the cloud can be realized by applying the file changes. The process of obtaining a file change here is simple to describe, but difficult to handle in practice. However, after some clever abstractions and anti-abstractions, the whole process of getting file changes becomes very easy. The scenario is further abstracted as synchronization time exclusivity, and the client-client relationship is determined to be loosely coupled, making synchronization operations on each client and cloud look independent. Therefore, the problem of file synchronization between multiple clients and the cloud can be regarded as the previously assumed problem of file synchronization between a single client and a cloud. The project implementation of cloud file synchronization system. The main work includes the development of REST architecture style Web service program based on Jersey framework and the development of cloud disk Windows client program for users. Deploy and test the cloud file synchronization system. Deployment refers to the deployment of the system to the actual production environment dawning 6000 supercomputer. All the data of the system are stored on ParaStor, and the test refers to the performance test of MongoDB and ParaStor when the benchmark tool YCSB is used to work together. After the practical test of engineering application, the cloud file synchronization system can basically work normally.
【学位授予单位】：重庆大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【相似文献】