基于Hadoop的ETL部件在分布式数据挖掘引擎中的应用

发布时间：2018-11-22 09:59

【摘要】：分布式数据挖掘引擎引擎系统随着互联网数据量不断的增加,把海量的数据逐渐成为数据沉淀数据挖掘出价值需求日益强烈的情况下产生。分布式数据挖掘引擎系统对多种源数据进行处理、数据分析挖掘为使用者提供决策的信息支撑。从当前的情况来看在网联网领域得数据者得天下,阿里巴巴通过海量的用户购买数据不断优化推荐与匹配规则;百度则通过海量的行为数据进行推荐相关以及广告。分布式数据挖掘引擎系统已经应用各种场景并在持续发展中。本文从分布式数据挖掘引擎系统的整体架构和核心技术(数据仓库、数据挖掘、实体管理器)入手。主要介绍了数据仓库和实体管理器以及系统搜索系统,重点是基本Hadoop平台的ETL部件的设计与实现。ETL主要讲述了数据预处理、数据文件的上传、数据抽取、数据变换、数据加载到数据仓库、编码、数据接口以及ETL问题与解决等。
[Abstract]:With the increasing of the Internet data, the distributed data mining engine system will gradually become the data precipitation data mining value of the increasingly strong demand to produce. Distributed data mining engine system processes multiple sources of data, and data analysis and mining provide users with information support for decision-making. From the current situation in the field of Internet access to data, Alibaba through massive users to buy data to constantly optimize the recommendation and matching rules; Baidu through a massive amount of behavioral data to recommend related and advertising. Distributed data mining engine system has applied various scenarios and is developing continuously. This paper starts with the whole architecture and core technology (data warehouse, data mining, entity manager) of the distributed data mining engine system. This paper mainly introduces the data warehouse, entity manager and system search system, with emphasis on the design and implementation of the ETL part of the basic Hadoop platform. ETL mainly describes the data preprocessing, data file upload, data extraction, data transformation, etc. Data loading into the data warehouse, coding, data interface and ETL problems and solutions.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】