基于业务规则的数据中心数据质量研究

发布时间：2019-01-10 13:21

【摘要】：为了提高数据质量，国内外对影响数据质量的因素及改善数据质量的方法进行了大量研究。这些研究主要集中在数据仓库中的数据质量问题，提出了数据质量度量指标及指标的计算方法。目前对数据质量的研究主要存在以下问题：首先，没有形成系统化的数据质量评估指标，导致不能形成完整的数据质量体系；其次，还没有形成一个权威性的数据质量参考模型，当前的研究都是针对单一问题进行的；最后，数据质量内容的定义是变化的，这就需要数据质量模型具有相应的扩展性，以满足这种变化需求。针对这些问题重点进行了以下研究。首先，提出并构建了完整的数据质量评估体系。定义了准确性、一致性等七类数据质量元素和非空约束、值域约束等十五个维度的规则，其中数据质量元素用于描述数据质量，数据质量约束规则反映了具体业务规则和领域知识；对数据质量评估指标给出了定义和具体算法；提出了数据质量分析评估体系结构及流程，整个体系结构分为数据层和应用层。数据层包括实例层、模式层、数据质量层和数据质量扩展层，数据质量层即数据质量元模型，数据质量扩展层提供了对数据质量元模型的扩展；应用层包括数据质量分析评估层、展示层。再次，针对数据中心中存在的相似重复记录问题，，采用了传统的“排序合并”的方法，本文提出了一种改进的基于内码序值聚类的检测方法，在字符串匹配算法中本文借鉴了生物信息学中的序列比对算法。改进后的方法提高了检测效率，并在实际应用中取得了良好效果。最后，以大庆油田井下作业分公司数据中心数据质量检测与评估为背景，对所提出的数据质量检测与评估体系进行了设计与实现，该系统实现了对各种业务规则的管理与维护，对各种数据质量指标的评估。该系统已在井下作业数据中心运行，对数据中心的据质量的改善起到了重要作用。
[Abstract]:In order to improve data quality, the factors affecting data quality and the methods to improve data quality have been studied extensively at home and abroad. These researches mainly focus on the data quality problems in data warehouse, and put forward the data quality measurement index and the calculation method of data quality index. At present, there are the following problems in the research of data quality: firstly, there is no systematic evaluation index of data quality, which leads to the failure to form a complete data quality system; Secondly, there is not yet an authoritative data quality reference model, the current research is aimed at a single problem; Finally, the definition of data quality content is variable, which requires that the data quality model has the corresponding expansibility to meet the changing requirements. In view of these problems, the following research focus has been carried out. Firstly, a complete data quality evaluation system is proposed and constructed. Seven kinds of data quality elements, such as accuracy, consistency, and non-empty constraints, and range constraints are defined. The data quality elements are used to describe the data quality. Data quality constraint rules reflect specific business rules and domain knowledge. The definition and algorithm of data quality evaluation index are given, and the architecture and process of data quality analysis and evaluation are presented. The whole architecture is divided into data layer and application layer. The data layer includes instance layer, mode layer, data quality layer and data quality extension layer. The data quality layer is the data quality metadata model, and the data quality extension layer provides the extension of the data quality metadata model. The application layer includes data quality analysis and evaluation layer and presentation layer. Thirdly, aiming at the problem of similar duplicate records in the data center, the traditional method of "sorting and merging" is adopted, and an improved detection method based on inner code order value clustering is proposed in this paper. In the string matching algorithm, we draw lessons from the sequence alignment algorithm in bioinformatics. The improved method improves the detection efficiency and achieves good results in practical application. Finally, under the background of data quality detection and evaluation of data center of Daqing oil field downhole operation branch, the proposed data quality detection and evaluation system is designed and implemented. The system realizes the management and maintenance of various business rules. Evaluate various data quality indicators. The system has been running in the underground operation data center, which plays an important role in improving the quality of the data center.
【学位授予单位】：东北石油大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP308

【参考文献】