BDAP——一个基于Spark的数据挖掘工具平台

发布时间：2018-08-09 07:56

【摘要】：大数据处理系统是大数据领域的一个热点,为此首先研究大数据分析平台的架构与功能,将大数据分析平台分为数据源、数据吸收层、数据存储层、平台层、安全与监控层、设备层和应用层.平台包含多个数据预处理和算法模块,平台架构为大数据分析了奠定基础.在功能上,该平台功能全面,可以自由组合各种操作,模块之间耦合度低,便于维护和拓展.在用户体验上,调参、建立流程、监控、数据挖掘过程都是可视的,融合工作流和调度流技术.在性能上,该平台相应算法的性能优于Hive和MLlib.最后,举例说明大数据挖掘平台的应用场景.可以对电网线路故障和气象数据进行预处理,从而对故障进行预测和分类,可以通过视频挖掘组件,对数据分类.
[Abstract]:Big data processing system is a hot spot in the field of big data. Firstly, the architecture and function of big data analysis platform are studied. The big data analysis platform is divided into data source, data absorption layer, data storage layer, platform layer, security and monitoring layer. Device layer and application layer. The platform consists of several data preprocessing and algorithm modules. The platform architecture lays the foundation for big data analysis. In terms of function, the platform has comprehensive function, can combine various operations freely, and has low coupling degree between modules, so it is easy to maintain and expand. In the user experience, the processes of adjusting parameters, establishing flow, monitoring and data mining are all visible, and the workflow and scheduling technologies are integrated. The performance of the algorithm is better than that of Hive and MLlib. Finally, an example is given to illustrate the application scenario of big data mining platform. It can preprocess the line fault and meteorological data of power network, so as to predict and classify the fault, and can classify the data by video mining component.
【作者单位】：北京邮电大学智能通信软件与多媒体北京重点实验室;北京邮电大学计算机学院;国网山东省电力公司电力科学研究院;
【基金】：国家高技术研究发展(863)计划(2015AA050204) 国网科技项目(520626150032) 北京市教育委员会共建项目建设计划资助
【分类号】：TP311.13

【相似文献】