数字博物馆海量数据的分布式存储关键技术研究与实现

发布时间：2018-04-14 13:35

本文选题：数字博物馆 + HDFS　；参考：《浙江大学》2013年硕士论文

【摘要】：数据存储系统是数字博物馆体系架构的核心部分。在当前数字博物馆资源快速增长的环境下,数字博物馆海量资源对数据存储和管理系统提出了新的挑战。本文针对当前数字博物馆海量数据存储和管理方法存在的若干问题,主要研究了数字博物馆海量多媒体小文件的存储和多尺度空间数据的管理技术,分别提出了基于HDFS的小文件合并策略和基于HBase的海量文物空间数据的管理技术。取得的主要成果如下：一、根据数字博物馆包含大量多媒体小文件资源的特点,提出了一种基于HDFS的海量多媒体小文件合并存储策略,其基本思想是将大量的小文件合并成大的聚簇文件后,一起保存到HDFS中,该策略相比于原始的HDFS,有效地减少了大量小文件对主节点的内存消耗,提高了文件写入性能,同时为聚簇文件中的小文件建立文件索引,降低了文件的读取时间。二、研究了文物的多尺度空间数据的特点,提出了一种适于海量多尺度空间数据的POZ空间填充曲线及基于HBase的文物多尺度空间数据存储模型,实现对海量多尺度空间数据的管理和快速访问,与传统关系数据存储方法对比,该方法在管理海量文物的多尺度空间数据方面优秀性能。三、最后我们基于本文的研究成果,设计并部署了数字博物馆海量数据的分布式存储系统,系统的构建采用三层体系结构,即存储层、组织层和数据访问层；存储层采用HDFS分布式文件系统,提供海量数据存储能力；组织层基于HBase提供结构化的数据组织,同时实现了多媒体小文件合并和文物的多尺度空问数据的管理；数据访问层是系统与上层应用的交互接口,主要提供数据操作和数据管理功能。
[Abstract]:Data storage system is the core of digital museum architecture.With the rapid growth of digital museum resources, massive digital museum resources pose a new challenge to the data storage and management system.Aiming at some problems existing in the methods of storing and managing mass data in digital museum, this paper mainly studies the storage of mass multimedia small files and the management technology of multi-scale spatial data in digital museum.The small file merging strategy based on HDFS and the management technology of massive cultural relic spatial data based on HBase are put forward respectively.The main results achieved are as follows:First, according to the characteristic that the digital museum contains a large number of multimedia small file resources, this paper proposes a large amount of multimedia small file merge and storage strategy based on HDFS, its basic idea is to merge a large number of small files into large cluster files.Compared with the original HDFS, this strategy can effectively reduce the memory consumption of a large number of small files to the master node, improve the performance of file writing, and index the small files in cluster files.Reduced the read time of the file.Secondly, the characteristics of multi-scale spatial data of cultural relics are studied, and a POZ spatial filling curve suitable for massive multi-scale spatial data and a storage model of cultural relic multi-scale spatial data based on HBase are proposed.Compared with the traditional relational data storage method, this method has excellent performance in managing the multi-scale spatial data of massive cultural relics.Finally, based on the research results of this paper, we design and deploy the distributed storage system of mass data in digital museum. The system is constructed with three layers: storage layer, organization layer and data access layer.The storage layer uses HDFS distributed file system to provide massive data storage, organization layer provides structured data organization based on HBase, and realizes multimedia small file merging and multi-scale spatial data management.The data access layer is the interface between the system and the upper application, which mainly provides data operation and data management functions.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】