数据中心的数据质量管理工具设计与实现

发布时间：2018-07-12 17:49

本文选题：数据中心 + 数据质量　；参考：《华中科技大学》2013年硕士论文

【摘要】：随着信息技术在各行业的不断发展，各行业也逐渐了累积了大量的业务数据，为了能够有效地利用这些业务数据便构建了数据中心。而为了保证进入数据中心的数据在数据质量上符合要求，便出现了各种以处理数据质量问题的数据清洗工具，但是即便如此，由于逻辑问题或者清洗处理过程中关注点不同等各种原因而导致进入数据中心后的数据仍可能有数据质量问题，因此，需要对进入数据中心后的数据在数据质量进行检测处理。为了分析处理进入数据中心后数据的数据质量，设计了数据中心的数据质量管理工具，包括数据质量模型的研究分析以及对数据质量管理工具的体系结构的分析。在具体实现上，有数据源管理模块、规范化管理模块、数据检测管理模块、数据质量属性分析和可视化模块。数据源管理模块用于处理数据中心的异构数据源的信息；规范化管理模块包括对规范化元规则的分析和实现的管理以及将数据源和相应的规范化规则进行关联并能够根据关联信息对数据源进行规范化处理等功能；数据检测管理模块包括由数据质量属性而提出四类数据检测规则的实现，以及将数据源中的数据集或者规范化后的数据集通过使用相应的检测规则进行处理的检测流程管理；数据质量属性分析和可视化模块主要是对数据质量属性进行定量分析以及根据数据检测模块处理后的数据分析出对应检测数据集在数据质量属性的整体情况，，并根据分析结果给予相关建议。通过对数据质量管理工具进行测试，然后对相应的结果进行分析，说明了该工具在功能方面的可用性，能够对数据中心的数据进行有效地分析处理。
[Abstract]:With the continuous development of information technology in various industries, each industry has gradually accumulated a large number of business data, in order to effectively use these business data to build a data center. In order to ensure that the data entering the data center meets the requirements in terms of data quality, a variety of data cleaning tools have emerged to deal with data quality problems, but even so, The data after entering the data center may still have data quality problems due to logic problems or different concerns in the cleaning process. Therefore, it is necessary to check the data quality after entering the data center. In order to analyze the data quality after entering the data center, the data quality management tools of the data center are designed, including the research and analysis of the data quality model and the analysis of the architecture of the data quality management tool. In the implementation, there are data source management module, standardized management module, data detection management module, data quality attribute analysis and visualization module. The data source management module is used to deal with the information of heterogeneous data sources in the data center. The standardized management module includes the management of the analysis and implementation of the normalized meta-rules, the association of the data source and the corresponding normalized rules, and the ability to normalize the data sources according to the association information. The data detection management module includes the implementation of four kinds of data detection rules proposed by the data quality attribute, and the management of the data set or the standardized data set in the data source by using the corresponding detection rules. The data quality attribute analysis and visualization module is mainly for the quantitative analysis of the data quality attributes, and according to the data processing of the data detection module to analyze the whole situation of the corresponding detection data set in the data quality attributes. And according to the results of the analysis to give relevant recommendations. By testing the data quality management tool and analyzing the corresponding results, the availability of the tool in function is illustrated, and the data in the data center can be effectively analyzed and processed.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP308

【参考文献】