基于动态可配置规则的数据清洗方法

发布时间：2018-06-12 02:48

本文选题：大数据 + 数据质量　；参考：《计算机应用》2017年04期

【摘要】：针对传统数据清洗方法通过硬编码方法来实现业务逻辑而导致系统的可重用性、可扩展性与灵活性较差等问题,提出了一种基于动态可配置规则的数据清洗方法——DRDCM。该方法支持多种类型规则间的复杂逻辑运算,并支持多种脏数据修复行为,集数据检测、数据修复与数据转换于一体,具有跨领域、可重用、可配置、可扩展等特点。首先,对DRDCM方法中的数据检测和数据修复的概念、实现步骤以及实现算法进行描述;其次,阐述了DRDCM方法中支持的多种规则类型以及规则配置;最后,对DRDCM方法进行实现,并通过实际项目数据集验证了该实现系统在脏数据修复中,丢弃修复行为具有很高的准确率,尤其是对需遵守法定编码规则的属性(例如身份证号码)处理时其准确率可达100%。实验结果表明,DRDCM实现系统可以将动态可配置规则无缝集成于多个数据源和多种不同应用领域且该系统的性能并不会随着规则条数增加而极速降低,这也进一步验证了DRDCM方法在真实环境中的切实可行性。
[Abstract]:In view of the problems of reusability, scalability and flexibility, a data cleaning method based on dynamic configuration rules is proposed in the traditional data cleaning method, which leads to the reusability, scalability and flexibility of the system by hard coding to achieve business logic. The DRDCM. method supports complex logic operations among various types of rules and supports a variety of dirty numbers. According to the repair behavior, set data detection, data repair and data conversion in one, it has the characteristics of cross domain, reusable, configurable, and extensible. First, it describes the concept of data detection and data repair in the DRDCM method, the implementation steps and the implementation of the algorithm. Secondly, it expounds the various rules type and rules supported by the DRDCM method. In the end, the DRDCM method is implemented, and the actual project data set verifies that the implementation system has a high accuracy rate of discarding the repair behavior in the dirty data repair, especially for the properties (such as identity card numbers) that need to comply with the legal code rules (such as the ID number). The accuracy of the system can reach the result of the 100%. experiment, and the DRDCM implementation system is shown. The system can integrate dynamically configurable rules seamlessly into multiple data sources and many different applications and the performance of the system will not decrease with the increase in the number of rules. This further validates the feasibility of the DRDCM method in the real environment.
【作者单位】：中国科学院新疆理化技术研究所多语种信息技术研究室;中国科学院大学计算机与控制学院;新疆民族语音语言信息处理重点实验室;
【基金】：新疆维吾尔自治区高技术研究发展计划项目(201512103) 中国科学院西部之光人才培养计划项目(XBBS201313) 新疆维吾尔自治区青年科技创新人才培养工程计划项目(2014721033)~~
【分类号】：TP311.13

【相似文献】