面向交通物流大数据处理的元数据管理系统

发布时间：2018-04-06 01:16

本文选题：元数据管理　切入点：数据整合　出处：《上海交通大学》2015年硕士论文

【摘要】：在交通物流领域中,建立统一、高效的数据管理方式是降低运营成本、提高数据质量、规范物流管理的迫切需要。本文以某省交通物流云平台项目为背景,该项目包含一系列交通物流大数据处理需求,如对物流企业、应用开发商及平台数据中心的数据进行数据整合并提供统一管理。目前,在数据整合及管理的过程中,主要存在以下问题:1)业务元数据和技术元数据没有建立映射关系,无法对企业异构数据源进行统一管理;2)没有将业务元数据与技术元数据结合,指导数据整合过程;3)由于涉及到安全及隐私问题,数据整合时对企业私有数据访问困难。针对上述问题,本论文在对元数据标准SDMX信息模型、RDF数据立方词汇表以及数据整合等相关技术进行深入研究的基础上,提出了一个面向交通物流领域的业务元数据模型,建立了技术元数据与业务元数据的映射,并与ETL工具相结合,设计并实现一个面向交通物流大数据处理的元数据管理系统。测试及应用表明,该系统是可行及有效的。与其他类似系统相比,本文工作具有以下特点:1)结合SDMX信息模型及RDF数据立方词汇表,提出了一个面向交通物流领域的业务元数据模型,对交通物流领域的术语、业务实体等进行建模;同时建立并维护业务元数据与技术元数据的映射关系。2)提出了基于元数据的交通物流数据整合方法,将业务元数据应用到交通物流领域的数据整合。在用户开发数据整合任务时,利用业务元数据与技术元数据的映射,自动建立整合字段源和目标的匹配,简化了开发过程。3)设计并实现了元数据管理系统,该系统包括元数据管理工具与数据整合工具两部分。元数据管理工具提供技术元数据获取、元数据管理、元数据映射、前置机管理及系统管理功能;数据整合工具提供数据整合任务开发与数据整合任务执行功能。应用情况表明,该系统增强了数据管理能力并提高了数据整合效率。
[Abstract]:In the field of transportation logistics, the establishment of unified and efficient data management is an urgent need to reduce operation cost, improve data quality and standardize logistics management.Based on a provincial transportation logistics cloud platform project, this project includes a series of traffic logistics big data to deal with the needs, such as logistics enterprises, application developers and platform data center data integration and provide unified management.At present, in the process of data integration and management, there are mainly the following problems: 1) Business metadata and technical metadata are not mapped.It is difficult to access private data of enterprise when data integration is involved because of the security and privacy problems which can not be unified management of heterogeneous data sources in enterprise (2) it does not combine business metadata with technical metadata, and guide the process of data integration.In order to solve the above problems, based on the in-depth study of metadata standard SDMX information model and related technologies such as data cube vocabulary and data integration, this paper proposes a business metadata model for traffic logistics domain.The mapping between technical metadata and business metadata is established, and a metadata management system for traffic logistics big data is designed and implemented by combining with ETL tools.The test and application show that the system is feasible and effective.Compared with other similar systems, this paper has the following characteristics: 1) combined with SDMX information model and RDF data cube vocabulary, this paper proposes a business metadata model for traffic logistics.At the same time, the mapping relationship between business metadata and technical metadata is established and maintained. (2) A method of integrating traffic logistics data based on metadata is proposed, and the business metadata is applied to the data integration in the field of transportation logistics.When the user develops the task of data integration, using the mapping of the business metadata and the technical metadata, the matching of the integrated field source and the target is established automatically, which simplifies the development process. 3) the metadata management system is designed and implemented.The system includes two parts: metadata management tool and data integration tool.Metadata management tools provide technical metadata acquisition, metadata management, metadata mapping, preprocessor management and system management functions, and data integration tools provide data integration task development and data integration task execution functions.The application shows that the system enhances the ability of data management and improves the efficiency of data integration.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP311.13

【相似文献】