全球剖分编码海量瓦片文件的有序存储和预取技术研究
发布时间：2018-08-28 14:24
【摘要】：地理信息服务具有数据量大、文件数多、大量用户并发访问等特点,传统文件系统和以Hadoop分布式文件系统HDFS(Hadoop Distributed File System)为代表的分布式文件系统无法满足海量地理空间数据的存储与访问要求。为了满足千亿规模海量小文件的存储和访问需求,本人所在项目组基于HDFS实现了海量小文件支持分布文件系统(SMDFS)。测绘信息系统的地图瓦片数据通常以金字塔组织,数据访问具有空间局部性的特点,因此如果能够实现文件预取将有效改善文件访问性能。然而SMDFS文件系统将成千上万小文件聚合成一个聚合文件进行存储,很难将地理相邻的若干文件通过一次I/O就反馈给用户。针对测绘数据访问的空间局部性特点和单个瓦片文件访问效率低的问题,本文提出基于地理位置的周边图片预取技术,目的是减少SMDFS的I/O访问次数,提高文件系统的访问性能。实现预取的一个前提条件是金字塔内瓦片文件存储是有序的。本文提出递归四分排序方法的全球剖分编码海量瓦片文件顺序存储技术,使得二维瓦片聚合文件依据地理位置信息进行排序,使地理相邻的图片存储时相邻。在顺序存储技术基础上,本文提出并实现了基于有序金字塔的预取技术,成功解决海量瓦片读取效率低、并发访问支持能力不足等问题。
[Abstract]:Geographical information service is characterized by large amount of data, large number of files, concurrent access by a large number of users, etc. Traditional file system and distributed file system represented by Hadoop distributed file system HDFS (Hadoop Distributed File System) can not meet the storage and access requirements of massive geospatial data. In order to meet the storage and access requirements of hundreds of billions of large and massive small files, my project team implemented a large number of small files supporting distributed file system (SMDFS).) based on HDFS. Map tile data of surveying and mapping information system is usually organized by pyramid, and data access has the characteristic of space locality. Therefore, if file prefetching can be realized, the performance of file access will be improved effectively. However, the SMDFS file system aggregates thousands of small files into one aggregate file for storage, so it is difficult to feed back several files to users through one I / O. In view of the spatial locality characteristics of surveying and mapping data access and the low efficiency of single tile file access, this paper proposes a prefetching technique based on geographical location for peripheral images, which aims to reduce the number of I / O visits of SMDFS and improve the access performance of file system. A prerequisite for pre-fetching is that the storage of tile files in the pyramid is orderly. In this paper, a recursive quadrature sorting method is proposed, in which the sequential storage technology of massive tile files in global partition coding is proposed, which makes the two-dimensional tile aggregation files be sorted according to the geographic location information, so that the adjacent images can be stored adjacent to each other. Based on the sequential storage technology, this paper proposes and implements the prefetching technology based on the ordered pyramid, which successfully solves the problems of low reading efficiency and insufficient concurrent access support capacity of massive tiles.
【学位授予单位】：国防科学技术大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333
，
本文编号：2209651
[Abstract]:Geographical information service is characterized by large amount of data, large number of files, concurrent access by a large number of users, etc. Traditional file system and distributed file system represented by Hadoop distributed file system HDFS (Hadoop Distributed File System) can not meet the storage and access requirements of massive geospatial data. In order to meet the storage and access requirements of hundreds of billions of large and massive small files, my project team implemented a large number of small files supporting distributed file system (SMDFS).) based on HDFS. Map tile data of surveying and mapping information system is usually organized by pyramid, and data access has the characteristic of space locality. Therefore, if file prefetching can be realized, the performance of file access will be improved effectively. However, the SMDFS file system aggregates thousands of small files into one aggregate file for storage, so it is difficult to feed back several files to users through one I / O. In view of the spatial locality characteristics of surveying and mapping data access and the low efficiency of single tile file access, this paper proposes a prefetching technique based on geographical location for peripheral images, which aims to reduce the number of I / O visits of SMDFS and improve the access performance of file system. A prerequisite for pre-fetching is that the storage of tile files in the pyramid is orderly. In this paper, a recursive quadrature sorting method is proposed, in which the sequential storage technology of massive tile files in global partition coding is proposed, which makes the two-dimensional tile aggregation files be sorted according to the geographic location information, so that the adjacent images can be stored adjacent to each other. Based on the sequential storage technology, this paper proposes and implements the prefetching technology based on the ordered pyramid, which successfully solves the problems of low reading efficiency and insufficient concurrent access support capacity of massive tiles.
【学位授予单位】：国防科学技术大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333
，
本文编号：2209651
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2209651.html
