面向大数据处理的分布式机器学习算法编排系统的研究与实现

发布时间：2018-05-28 03:00

本文选题：机器学习 + 分布式　；参考：《北京邮电大学》2017年硕士论文

【摘要】：计算机技术与互联网产业发展迅猛,数据量呈爆炸式的增长,我们进入了大数据时代。海量的数据中蕴含着大量的知识价值,机器学习作为一种可以从海量数据中提取有用信息的关键技术,它的学习成本和使用门槛都比较高。数据分析的业务人员需要具备统计学知识、数据建模能力、算法设计能力和编程能力,为了降低业务人员的使用门槛,我们需要一个通用性强、简单易用、性能优异的针对大数据的分布式处理工具,实现对数据的分析。本论文设计和实现了一个分布式机器学习算法编排系统,该系统在分布式环境上给用户提供简单易用的机器学习服务,降低了机器学习算法的应用门槛,让用户不用通过编写程序的方式进行数据分析和预测,让用户通过Web界面的简单操作流畅直观的完成数据导入、数据解析、模型训练、预测等操作,通过良好的Web界面交互为开发人员、业务分析师提供了交互式的机器学习服务。本系统以OpenStack云平台作为底层环境,提供了弹性可扩展的计算资源和存储资源,通过Hadoop在云平台上搭建分布式集群,其中 YARN (Yet Another Resource Negotiator)框架提供了并行计算的能力,HDFS (Hadoop Distributed File System)保证了海量数据的存储,上层的Spark为需要迭代的机器学习算法提供了更高效的内存计算能力。在机器学习算法方面,本系统在数据处理层实现了核心算法模块,提供了机器学习领域分类、回归、聚类等经典算法。在算法编排方面,本系统在业务逻辑层实现了工作流管理模块,提供了工作流编排的业务逻辑。在表现层通过交互组件模块实现了用户与网页交互的全部接口。系统测试结果显示本系统能保证所有功能稳定运行,实现简单友好的算法编排操作。同时本系统的算法性能也达到了预期性能要求。
[Abstract]:With the rapid development of computer technology and Internet industry and the explosive growth of data volume, we have entered the era of big data. As a key technology to extract useful information from massive data, machine learning, which contains a lot of knowledge value, has a high learning cost and a high threshold of use. The business personnel of data analysis need to have the knowledge of statistics, the ability of data modeling, the ability of algorithm design and the ability of programming. In order to reduce the threshold of use of business people, we need a universal, simple and easy to use. Excellent performance for big data distributed processing tools to achieve data analysis. In this paper, a distributed machine learning algorithm arrangement system is designed and implemented. The system provides users with a simple and easy-to-use machine learning service in distributed environment, and reduces the threshold of application of machine learning algorithm. So that users do not have to write a program for data analysis and prediction, let users through the simple operation of the Web interface smooth and intuitive completion of data import, data analysis, model training, prediction and other operations, Through good Web interface interaction for developers, business analysts provide interactive machine learning services. The system takes the OpenStack cloud platform as the underlying environment, provides flexible scalable computing resources and storage resources, and builds a distributed cluster on the cloud platform through Hadoop. The YARN / Yet Another Resource Negotiator) framework provides the ability of parallel computing. HDFS / Hadoop Distributed File System) guarantees the storage of massive data, and the upper Spark provides a more efficient memory computing capability for the machine learning algorithm that needs iteration. In the aspect of machine learning algorithm, the system implements the core algorithm module in the data processing layer, and provides classical algorithms such as machine learning domain classification, regression, clustering and so on. In the aspect of algorithm arrangement, the workflow management module is implemented in the business logic layer, and the business logic of workflow orchestration is provided. The interface between user and web page is realized by interactive component module in presentation layer. The system test results show that the system can ensure the stable operation of all functions and achieve a simple and friendly algorithm layout operation. At the same time, the algorithm performance of the system also meets the expected performance requirements.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.09;TP311.13;TP181

【相似文献】