个人电子数据云存储系统的设计与实现

发布时间：2018-05-03 18:04

本文选题：云存储 + Hadoop分布式文件系统　；参考：《大连理工大学》2012年硕士论文

【摘要】：随着社交网络的发展,个人电子数据呈爆炸性增长趋势,传统的集中式数据存储模式已经无法满足其存储和使用需求。本文针对个人电子数据的特点,提出了基于云环境的通用存储模式,通过设计索引,可以提供多种快速查询功能。本系统能够有效的管理日益复杂的个人电子数据,满足用户对于存储能力、数据可用性和资源共享的需求。云计算目前已经成为学术界的研究热点。而基于云概念的存储系统,也已经初见端倪。云存储强调透明性,是以集群的方式对外提供海量的数据存储能力,通过增加存储节点,可以扩展存储容量,而数据的冗余备份,则可以保证容错性能。另外,分布式的数据存储,可以支持并行化数据管理,从而提高了数据存取性能。本文针对日益增长的海量个人电子数据,结合分布式云存储技术,设计并实现了个人电子数据的云存储系统。针对个人电子数据文件大小异构的特点,本文设计了通用的个人电子数据存储模型,即利用Hadoop分布式文件系统(HDFS)直接存放视频等大数据文件,而相对较小的文件,利用HBase直接存放。该模型能根据数据的不同类别,如图片、文档、视频进行分区存储。针对视频文件,本文设计了名称索引和主题索引,能够支持两种情况下的快速检索。针对个人电子数据中文档数据内容杂乱而用户又需要按内容检索的特点,本文利用MapReduce框架编程,建立Lucene倒排索引,实现了分布式索引的构建与维护。在此基础上,本文利用分布式检索工具Katta对分布式索引进行检索,实现了对海量数据信息的高效获取。此外,本文利用HDFS和HBase提供给用户的操作接口,对分布式系统上的数据实现上传、下载、删除等功能。
[Abstract]:With the development of social network, personal electronic data has an explosive growth trend. The traditional centralized data storage mode has not been able to meet its storage and use requirements. In this paper, a general storage mode based on the cloud environment is proposed for the characteristics of personal electronic data. By setting the index, a variety of fast query functions can be provided. The system can effectively manage the increasingly complex personal electronic data to meet users' needs for storage capacity, data availability and resource sharing.
Cloud computing has become a hot topic in the academic field, and the cloud based storage system has also been seen. Cloud storage emphasizes transparency. It provides massive data storage capacity in a cluster and can expand storage capacity by increasing storage nodes, while redundant backup of data can guarantee fault tolerance. In addition, distributed data storage can support parallel data management, thus improving data access performance.
This paper designs and implements a cloud storage system for personal electronic data in view of the growing mass of personal electronic data and distributed cloud storage technology. In view of the characteristics of the size of individual electronic data files, this paper designs a general personal electronic data storage model, which is stored directly by the Hadoop distributed file system (HDFS). Large data files such as video, while relatively small files are stored directly by HBase. The model can be partitioned according to different types of data, such as pictures, documents, and video. In this paper, the name index and subject index are designed to support the fast retrieval of two situations. According to the characteristics of the disorderly content and the user need to retrieve the content according to the content, this paper uses the MapReduce framework programming to establish the Lucene inverted index, and realizes the construction and maintenance of the distributed index. On this basis, this paper uses the distributed retrieval tool Katta to retrieve the distributed index, and realizes the efficient acquisition of the mass data information. In this paper, we use HDFS and HBase to provide user interface, upload, download and delete data on distributed system.

【学位授予单位】：大连理工大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】