云环境中的移动文件存储和时空数据分析关键技术研究

发布时间：2018-09-11 07:08

【摘要】：云计算是近年来蓬勃发展的一个重要领域。云计算环境下的海量数据存储、管理与分析存在诸多挑战，其中时空数据存储访问和应用分析已成为云环境下数据研究的重要方向。一方面，随着个人数字终端的普及，移动设备势必成为数字化社会中数据的重要来源。移动设备产生的数据往往具有时空特性，如何为移动设备提供便捷的海量时空数据存储访问机制是云计算环境下数据管理的一大关键问题。另一方面，复杂的时空数据特性势必成为影响数据分析的关键因素，如何为时空数据提供准确而高效的应用分析能力是云计算环境下数据管理的又一关键问题。本文针对时空数据管理和数据分析展开深入研究，主要解决移动设备访问异构云存储资源效率不高、使用不方便等问题，克服时空数据使用过程中数据不精确、数据量庞大等因素带来的挑战。本文的研究工作和创新点如下： (1)提出了一种基于文件系统的异构云存储服务无缝统一访问方法Wukong。目前不同的云存储服务访问接口各异，由于缺乏无缝访问多种云存储服务的统一接口，，时空数据应用不能够不加修改地直接访问和使用后端云存储数据。文件系统是被应用程序广泛支持的一种数据使用接口，以文件系统接口封装各种云存储服务可以保证应用程序对异构云存储服务的直接访问。本文首次提出了一种基于文件系统的异构云存储服务无缝统一访问方法Wukong文件系统。Wukong通过异构服务接口抽象封装技术，实现了对多种云存储服务的一致化管理，保证时空数据应用对异构云存储服务的统一访问。Wukong通过基于桥接模式的接口封装技术，实现了云存储访问接口到文件系统接口的转换，保证时空数据应用程序对云存储服务的透明访问。针对移动环境，本文通过自适应压缩及离线服务伪装等优化技术，降低了系统对网络环境的要求，保证移动设备在不同环境下对云存储服务的有效访问。 (2)提出了一种支持多数据视图的跨设备文件同步机制EaSync。由于个人数字设备日益普及，人们往往拥有多个设备。在时空数据应用使用、收集数据过程中，设备之间的数据可以相互辅助应用，例如设备可以使用采集的GPS数据来修正地图数据，也可以使用地图数据来进一步提高采集得到的GPS数据的精度等。因此，多个设备之间数据的透明同步服务有利于为时空数据应用提供更好的支撑与保障。但是，由于设备用途及处理能力不同，各种设备所需要且能处理的文件集合（即数据视图）存在差异，如何为多个设备提供异构的数据视图是同步服务的一大挑战。针对数据同步服务面临的设备种类众多、数据视图变化快、视图差异大等问题，本文提出了一种支持多数据视图的跨设备文件同步机制EaSync。EaSync通过基于时间戳的文件同步机制，保证数据跨设备访问的一致性，实现了数据在服务端与多个设备端之间的同步存储。EaSync通过基于树型结构的视图组织方式，实现了差异视图的层次化管理，解决了数据视图更新快、差异大的问题。 (3)提出了一种基于传统地图匹配算法的交互式真实路径发现方法。随着各类交通信息服务需求的不断增长，智能交通系统在社会生活中日趋重要。构建有效的智能交通系统依赖于大量的GPS时空数据，而原始GPS数据存在精度不高的问题。地图匹配算法是用来提高数据精度的常用方法，但是传统的地图匹配算法存在真实路径缺失的问题。本文提出了一种交互式的真实路径发现方法TruthFinder，通过人工评估调整与传统地图匹配算法相结合的路径发现技术，从历史轨迹数据中准确地发掘出真实路径信息，解决传统地图匹配算法真实路径缺失的问题。本文通过基于用户操作次数统计的开销模型分析，提出了地图匹配算法自主选择与人为误差有效避免等的优化策略，实现了大规模真实路径的快速生成。实验分析表明，TruthFinder在保证生成的真实路径的精度的同时，降低了人工操作的评估与调整开销，提高了真实路径的发现效率。 (4)提出了面向时空数据的并行化索引创建和并行化数据聚类分析方法。随着时空数据的不断增长，现有的索引创建方法与聚类分析算法已无法处理当前庞大的数据集合，针对大规模时空数据的索引创建与聚类分析是海量时空数据挖掘的重大难题。本文提出了面向时空数据的并行化索引创建和并行化数据聚类分析方法。通过基于MapReduce模型的并行化处理框架，本文实现了时空数据索引的自底向上打包创建，解决了传统索引创建方法面对海量时空数据所带来的主存容量有限、处理周期过长甚至无法处理的问题；本文提出了并行的DBScan聚类分析算法，解决了大规模时空数据聚类分析中数据分区困难、局部分簇合并复杂的问题，为时空大数据的处理提供了技术基础。通过实验分析表明，我们的并行化方法在保证数据处理性能的同时具良好的可扩展性，解决了数据量庞大、计算复杂所带来的主存容量受限、处理周期过长等问题。通过以上四方面研究，我们为云计算环境下的海量数据存储、管理与分析提供了有力的技术支撑。
[Abstract]:Cloud computing is an important area of vigorous development in recent years. There are many challenges in mass data storage, management and analysis in cloud computing environment. Access to spatio-temporal data storage and application analysis have become an important direction of data research in cloud environment. On the one hand, with the popularity of personal digital terminals, mobile devices are bound to become digital. Data generated by mobile devices are often spatio-temporal. How to provide a convenient storage and access mechanism for massive spatio-temporal data for mobile devices is a key issue for data management in cloud computing environment. How to provide accurate and efficient application analysis capability for spatio-temporal data is another key problem of data management in cloud computing environment. In this paper, spatio-temporal data management and data analysis are deeply studied, mainly to solve the problems of inefficient access to heterogeneous cloud storage resources by mobile devices, inconvenient use, and to overcome the over-use of spatio-temporal data. The research work and innovation of this paper are as follows:1.
(1) a seamless unified access method for heterogeneous cloud storage service based on file system is proposed. Wukong.
At present, different cloud storage services have different access interfaces. Due to the lack of a unified interface for seamless access to various cloud storage services, spatio-temporal data applications can not directly access and use back-end cloud storage data without modification. File system is a widely supported data usage interface by applications, encapsulating various kinds of data with file system interfaces. Cloud storage services can guarantee direct access to heterogeneous cloud storage services by applications. In this paper, a seamless and unified access method for heterogeneous cloud storage services based on file system, Wukong file system, is proposed for the first time. Wukong realizes the transformation from cloud storage access interface to file system interface by using the interface encapsulation technology based on bridge mode to ensure transparent access to cloud storage services by spatio-temporal data applications. The installation optimization technology reduces the requirement of the system to the network environment and guarantees the effective access to cloud storage services by mobile devices in different environments.
(2) a cross file synchronization mechanism for multi data views is proposed, EaSync..
Because of the popularity of personal digital devices, people often have multiple devices. In the process of collecting data, the data between devices can be used to assist each other. For example, the devices can use the collected GPS data to correct the map data, and can also use the map data to further improve the collected GPS data. Therefore, transparent data synchronization services between multiple devices can provide better support and protection for spatio-temporal data applications. However, due to different device uses and processing capabilities, various devices need and can process different sets of files (i.e. data views), how to provide heterogeneous data views for multiple devices This paper proposes a cross-device file synchronization mechanism, EaSync.EaSync, which supports multi-data views to ensure consistency of data access across devices through a timestamp-based file synchronization mechanism. It realizes the synchronous storage of data between the server and multiple devices. EaSync realizes the hierarchical management of different views by organizing views based on tree structure, and solves the problem of fast updating of data views and large differences.
(3) an interactive real path discovery method based on traditional map matching algorithm is proposed.
Intelligent Transportation System (ITS) is becoming more and more important in social life with the increasing demand for various kinds of traffic information services. Constructing an effective ITS relies on a large number of GPS spatio-temporal data, while the original GPS data has the problem of low accuracy. Map matching algorithm is a common method to improve the accuracy of data, but the traditional map map. TruthFinder, an interactive real path discovery method, is proposed in this paper, which combines manual evaluation and adjustment with traditional map matching algorithm to accurately extract real path information from historical track data and solve the real path of traditional map matching algorithm. By analyzing the cost model based on the statistics of user's operation times, this paper proposes the optimization strategy of map matching algorithm, such as autonomous selection and effective avoidance of human error, to realize the fast generation of large-scale real paths. The evaluation and adjustment cost of manual operation improves the discovery efficiency of real path.
(4) put forward parallel index creation and parallel data clustering analysis method for spatiotemporal data.
With the continuous growth of spatio-temporal data, the existing index creation methods and clustering analysis algorithms can not deal with the current huge data set. For large-scale spatio-temporal data, index creation and clustering analysis is a major problem in massive spatio-temporal data mining. According to the clustering analysis method, through the parallel processing framework based on MapReduce model, this paper realizes the bottom-up packing of spatio-temporal data index creation, and solves the problem that traditional index creation methods face the massive spatio-temporal data, such as limited main memory capacity, long processing cycle and even unable to process. An clustering analysis algorithm solves the problems of data partitioning and local clustering and merging in large-scale spatio-temporal data clustering analysis, and provides a technical basis for large-scale spatio-temporal data processing. The experimental results show that our parallelization method has good scalability while guaranteeing the performance of data processing, and solves the problem of large-scale data. The complexity of the main memory and the long processing cycle caused by the large computation complexity.
Through the above four aspects of research, we provide a strong technical support for mass data storage, management and analysis in the cloud computing environment.
【学位授予单位】：国防科学技术大学
【学位级别】：博士
【学位授予年份】：2013
【分类号】：TP333;TP311.13

【共引文献】