基于关联数据的一致性和时效性清洗方法

发布时间：2018-03-14 11:33

本文选题：数据一致性　切入点：数据时效性　出处：《计算机学报》2017年01期 　论文类型：期刊论文

【摘要】：数据一致性和数据时效性是大数据质量管理所关注的两个重要内容.条件函数依赖(CFDs)和时效约束(CCs)分别是用于分析数据一致性和数据时效性的有效技术手段.现实生活中的数据会夹杂一些关于一致性和时效性的潜在错误,这些错误又无法为CFDs和CCs检测和修复,最终影响数据的整体质量.值得一提的是,这些数据通常是相互关联的,这种关联关系可以用来发现数据中的潜在错误.文中使用了一种条件合并的函数依赖(CCFDs)将关联数据放在一起进行处理.基于此,该文提出了一种基于关联数据的一致性和时效性清洗方法.在数据清洗过程中,数据的检测和修复是两个相互影响的过程.所以,该文设计了一种新的自动清洗框架,迭代地进行数据检测和数据修复.其次,该文对关联数据的一致性和时效性清洗的相关问题进行了分析,并且证明了关于CCFDs和CCs的最小代价修复问题是一个Σ~p_2完全(NP~(NP))问题.进而,该文采用一种启发式的修复方法对错误进行修复.为了提高修复的准确性,该文还提出了一种修复序列图的概念.最后,通过在两组真实数据上进行实验,验证了方法的实用性和高效性.
[Abstract]:Data consistency and data timeliness are two important contents of big data's quality management. Conditional function dependence (CFDs) and time-dependent constraint (CCS) are effective technical means to analyze data consistency and data timeliness respectively. The data in real life can be mixed with potential errors about consistency and timeliness, These errors, in turn, cannot be detected and fixed for CFDs and CCs, and ultimately affect the overall quality of the data. This association can be used to detect potential errors in data. A conditional merge function dependency CCFDs is used to process the associated data together. In this paper, a method of consistency and timeliness cleaning based on associated data is proposed. In the process of data cleaning, the detection and repair of data are two interactive processes. Therefore, a new automatic cleaning framework is designed in this paper. Data detection and data repair are carried out iteratively. Secondly, the paper analyzes the problems related to the consistency and timeliness of the associated data cleaning. It is proved that the least cost repair problem for CCFDs and CCs is a 危 PST2 complete CCFDs problem. Furthermore, a heuristic method is used to fix the errors. This paper also proposes a concept of repairing sequence diagram. Finally, the practicability and efficiency of the method are verified by experiments on two groups of real data.
【作者单位】：东北大学计算机科学与工程学院;
【基金】：国家“九七三”重点基础研究发展规划项目基金(2012CB316200,2012CB316201) 国家自然科学基金(61033007,61472070,61672142) 中央高校基本科研业务费专项资金(N150408001-3,N150404013)资助~~
【分类号】：TP311.13

【相似文献】