云存储系统中基于溯源关系的文件查询与管理方法研究

发布时间：2018-01-03 15:18

本文关键词：云存储系统中基于溯源关系的文件查询与管理方法研究　出处：《华中科技大学》2016年博士论文　论文类型：学位论文

【摘要】：随着备份、归档、视频等网络服务的广泛应用,云存储系统的规模越来越大,给文件访问的性能带来挑战。用户在访问文件之前需依靠文件查询来确定待访问的文件及其地址,因此需要提高文件查询的性能。而现有的元数据查询方法无法有效地支持文件的高性能查询,因此需要挖掘更多的文件相关性来提高元数据查询的性能。云存储系统中,大部分的数据都处于冷数据状态,被存储在价格便宜、性能较差的设备上。有些应用需要保证高效的冷数据访问性能,因此需要通过分析冷数据文件在文件访问以及文件属性的相关性来提高文件管理的性能,建立一个高效的冷数据的分布及其元数据的索引结构。现在,大数据存储及其网络传输中视频已经占了65%,这些视频中大量的近似重复视频带来了巨大的传输和存储开销。针对这种特殊文件的应用和管理,需要通过挖掘这些视频之间的关系来提高视频的存储空间有效性以及访问速度。因此,挖掘文件之间的相关性对于提高云存储系统中文件访问性能具有重要作用。在云存储系统中,具有溯源关系的文件之间具有数据相关性,这种数据相关性导致这些文件具有相同或相似的内容,因此从溯源数据中可以挖掘到具有溯源关系的文件相关性,包括很强的内容相似性、属性相似性和读写特征的相似性以及较弱的内容差异性等。同时,文件的溯源数据记录了影响这个文件最终状态的所有进程和文件,通过对溯源数据的分析不仅在空间维度上可挖掘到更多文件的相关性,而且在时间维度上可以得到文件之间的相关性的变化,从而可提高文件相关性度量的精确度。因此,围绕加快文件访问的速度这一目标,本文利用文件相关特性提出了三种优化方法。(1)针对云存储系统规模的扩大会降低文件访问所必需的元数据查询操作的性能的问题,提出了一种挖掘具有溯源关系的文件之间的内容相似性的元数据查询方法PROMES,通过在查询流程中增加关系图查询减少元数据索引树中的查询范围而加快了查询速度,通过引入关系的时效性和文件的权重两个参数提高了关系图中文件相关性度量的准确性而增加了查询准确率。PROMES中的元数据查询分为三步：通过一个相关性感知的元数据索引树来定位到一些文件的元数据作为种子。绝大部分的种子满足查询条件,然后使用种子在根据溯源关系所生成的关系图中查询剩余的查询结果,最后进行排序并精简以得到最终的查询结果。因为由溯源关系生成的关系图具有文件相关性紧密、轻量索引的特点,所以PROMES具有显著的高查询准确率和低延迟的优点。实验测试表明：PROMES比现在最新的元数据查询方法减小了1到2个数量级的查询时间,并具有更高的查询准确率。(2)针对云存储服务供应商通常在低功耗的设备上存储冷数据文件及其元数据而导致文件访问速度受限的问题,提出了一种基于溯源关系中文件的属性和访问特征相似性的冷数据分布及其元数据索引机制,通过挖掘具有溯源关系的文件之间的访问特征相似性调整文件的分布状态减少访问冷数据文件的等待时间并节省能源,通过挖掘具有溯源关系的文件之间的元数据相似性对文件元数据进行逻辑分组减少冷数据文件的元数据查询的延时。这个机制包含两个方法：基于溯源关系中文件的访问特征相似性的冷数据重分布方法Prodi和基于溯源关系中文件的属性特征相似性的冷数据元数据索引方法P-index。通过测试表明Prodi节省了25%能源,而P-index的性能比现有的元数据索引结构减少了1到2个数量级的查询时间开销。(3)针对云存储系统中大量的近似重复视频降低了用户的体验质量,并消耗了服务供应商的很多资源的问题,提出了一种基于溯源关系中文件的内容差异性的视频压缩与传输机制Provis,利用视频的溯源数据能够支持视频重建和记录了具有溯源关系的视频之间内容差异的溯源数据的大小远小于视频文件的特性,通过保存视频的溯源数据来替代保存视频的方法压缩近似视频而提高存储系统的空间有效性,通过上传视频的溯源数据后再在云端重建视频的方法来替代直接上传视频而加快了视频上传的速度并减少上传的网络开销。通过两个视频数据集的测试比较了Provis与其它现存的视频压缩技术,结果显示Provis能够显著节省空间并降低视频上传的网络开销,溯源图的存储开销和视频重建的时延在用户可接受范围内。综上所述,本文主要解决云存储系统规模的扩大给文件的查询与管理性能带来新的挑战的问题,通过挖掘具有溯源关系的文件之间的各种相关特性来优化云存储系统的文件查询和管理,提出了一系列的方法,这些方法不仅可以用来提高存储系统的文件访问的性能,而且为溯源关系的广泛应用提供了相关理论和技术支撑。
[Abstract]:With the backup, archiving, extensive application of video network services, cloud storage system is larger and larger, to the performance of file access challenges. Users need to rely on the query to determine the file to access files and access address in the file before it is necessary to improve the performance of file query and metadata query. The existing methods can not effectively support the high performance file query, therefore need to dig more correlation of files to improve the performance of metadata query. Cloud storage system, most of the data in the data is stored in a cold state, low price, poor performance of the device. Some applications need to ensure efficient cold data access performance, therefore need to analysis cold data files in the file access and file attribute correlation to improve the performance of file management, the establishment of an efficient distribution of data and the number of cold According to the index structure. Now, big data storage and network transmission of video has accounted for 65% of the large number of near duplicate video video brings huge transmission and storage overhead. The application and management of this special file, need to improve the effectiveness of video storage space and access speed by mining these relationships between video. Therefore, the correlation between mining documents plays an important role in improving the cloud storage system file access performance. In the cloud storage system, with data due to the correlation between the relationship between this kind of data files, these files associated with the same or similar content, so it can be from the original data mining to file associated with traceability the relationship, including strong content similarity and attribute similarity and read-write character similarity and weak content differences. When the source data file records all the process and effect of the final state of the file. This file, through the analysis of the origin of data mining to correlation can not only more files in the space dimension, but also can get the correlation between files in the time dimension, which can improve the accuracy of document relevance measure. Therefore, around the goal to accelerate the file access speed, this paper uses the file related characteristics of three methods are put forward. (1) to expand the scale of the cloud storage system will reduce the performance of metadata required for file access query operation problems, put forward a mining due to the relationship between the content of the document similarity search method PROMES the metadata, by increasing the relationship between map query and reduce metadata index tree queries in the query range in the process and speed up the query speed, through the introduction of The relationship between weight of the timeliness and files of the two parameters to improve the accuracy of measurement in the correlation diagram file and increase the accuracy of query metadata query in.PROMES is divided into three steps: the metadata index tree of a correlation aware to locate the file to some metadate as seeds. Most of the seeds that satisfy the query the conditions, and then use the remaining seed query results in graph generated according to the traceability relationship, finally sorted and trimmed to obtain the final results. Because the diagram generated by tracing relationship has close correlation characteristics of lightweight file, index, so PROMES has high query accuracy and significant advantages of low delay. Experiment results show that the PROMES now than the latest metadata query method is reduced by 1 to 2 orders of magnitude of the query time, and has higher query quasi Accuracy. (2) cloud storage service providers usually in low power devices on the cold storage data files and metadata according to the resulting file access speed limitation, puts forward a kind of similarity and attribute access characteristics based on the relationship between the source file data and metadata distribution mechanism of cold index, due to the relationship between access characteristics through mining document similar to the distribution of the adjustment file to reduce the waiting time to access the cold data file and save energy, to reduce the metadata file metadata for cold data file query delay logical grouping similar metadata has traceability relations through mining files. This mechanism includes two methods: access profile traceability relations based on the similarity of the cold data redistribution method Prodi and similarity attribute relationship based on the source file P-index. metadata index data through the method of cold test shows that Prodi saves 25% energy, while P-index performance than the existing structure of the metadata index decreased by 1 to 2 orders of magnitude of the query time. (3) according to the cloud storage system in a large number of near duplicate video reduces the quality of the user experience, a lot of resources and consumption of services the supplier of the problem, puts forward a content file in the relationship between the difference of source video compression and transmission mechanism based on Provis, using video traceability data can support characteristics of video reconstruction and record the difference between traceability data due to the relationship of the video is much smaller than the size of video files, video data storage method by tracing the alternative to save video compression and improve the effectiveness of the spatial approximation of video storage system, data upload video by tracing back in heavy cloud The construction method to replace the direct video upload video and accelerate the video upload speed and reduce the network overhead. Upload video data through two sets of test and comparison of Provis and other existing video compression technology, the result shows that Provis can significantly save space and reduce the video upload network overhead, storage overhead and traceability chart the delay in the reconstruction of video users within the acceptable range. To sum up, this paper mainly solves the expansion of cloud storage system scale brings new challenges to the query and file management performance problems related with traceability relations through mining characteristics between files to optimize cloud storage system file query and management, put forward a series of methods these methods, not only can be used to improve the performance of storage system file access, and provides the theory for wide application of traceability relations And technical support.

【学位授予单位】：华中科技大学
【学位级别】：博士
【学位授予年份】：2016
【分类号】：TP333

【相似文献】