数据质量校验规则提取技术的研究

发布时间：2018-06-11 17:16

本文选题：数据质量 + 规则提取　；参考：《东北石油大学》2017年硕士论文

【摘要】：随着信息行业的发展,数据已经渗透到各行各业生产经营环节中,数据量随之越来越庞大。但同时“数据丰富,信息匮乏”现象也变得越来越突出,造成这种现象的主要原因:一方面是现在并没有强有力的数据集成及数据分析技术,另一方面是随着脏数据的出现,严重的影响了数据质量,导致各行业不能有效的利用现有的数据。数据质量是数据分析、挖掘、决策的前提和基础。数据质量的提高,不但可以准确的反映现实世界的状况,同时也可以高效地支持企业的运作和决策。因此数据质量问题成为数据管理领域研究的一个热点问题。数据质量管理的方式主要是采用数据质量校验规则来判断数据合法性以及评估数据质量等级。数据质量校验规则与业务领域紧密关联,目前数据质量校验规则通常依靠领域专家和数据管理专家采用手工方式制定。手工制定规则工作量大,效率低、耗时长,且规则完整性难以保证。因此本文采用软件工程中“逆向工程”思想,借助机器学习相关技术,研究数据质量校验规则自动生成技术,可以为领域专家提供更多的规则备选方案,提高数据质量校验规则制定效率。为了全方面的检查出数据库中的所有质量问题,本文研究了数据质量维度评估标准,并以规则约束为研究点,针对Oracle和Excel数据源的文本数据格式、值域以及函数依赖关系展开研究,设计三种数据质量校验规则提取的学习算法的,研发具有较高通用性,且不受领域限制的数据质量校验规则提取系统。
[Abstract]:With the development of information industry, the data has penetrated into all walks of life production and management, and the amount of data has become more and more huge. But at the same time, the phenomenon of "data rich, lack of information" has become more and more prominent. The main reason for this phenomenon is that there is no strong data integration and data analysis technology, and on the other hand, with the emergence of dirty data, Seriously affect the quality of data, resulting in the industry can not effectively use the existing data. Data quality is the premise and foundation of data analysis, mining and decision-making. The improvement of data quality can not only accurately reflect the real world situation, but also effectively support the operation and decision-making of enterprises. Therefore, data quality has become a hot issue in the field of data management. The main way of data quality management is to use the data quality check rule to judge the validity of data and evaluate the grade of data quality. The data quality verification rules are closely related to the business domain. At present, the data quality verification rules are usually formulated manually by domain experts and data management experts. Manual rule-making work is heavy, inefficient, time-consuming, and the integrity of the rules is difficult to guarantee. Therefore, this paper adopts the idea of "reverse engineering" in software engineering, with the help of machine learning related technology, to study the automatic generation technology of data quality verification rules, which can provide more alternative schemes for domain experts. Improve the efficiency of data quality calibration rules. In order to check out all the quality problems in the database in all aspects, this paper studies the evaluation standard of data quality dimension, and takes the rule constraint as the research point, aiming at the text data format of Oracle and Excel data sources. In this paper, three learning algorithms of data quality check rule extraction are designed and studied, and a data quality check rule extraction system with high generality and no limitation of domain is developed.
【学位授予单位】：东北石油大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：F49;TP311.13

【参考文献】