基于大数据的辅助专家分析系统的设计与实现

发布时间：2018-06-19 23:52

本文选题：大数据分析 + MapReduce　；参考：《浙江大学》2017年硕士论文

【摘要】：大数据时代,各行各业越来越多的企业或组织将受益于大数据这个发展机遇,大数据分析能够实现大数据规模效应下的数据计算和全面的数据洞察,帮助企业提高生产力、提升竞争力、促进企业革新。然而大数据分析对于许多企业或组织来说存在着许多难题,在数据库软件、大数据采集和规约、发展企业专业分析人员等方面仍存在着一些瓶颈。为适应企业的大数据分析需求,本文设计了基于大数据分析的辅助专家分析系统,集成了数据采集、数据存储、数据分析、数据可视化等功能,并提供了友好的人机交互界面,屏蔽大数据分析算法细节,以帮助企业人员简单高效地使用大数据分析技术进行高并发快速查询,建立复杂数据处理模型,自定义可视化模块来展现大数据的价值,促进企业的业务提升、支持企业决策。本文的辅助专家分析系统综合利用了阿里云的ADS和MaxCompute两个PaaS服务,后者是基于MapReduce框架的。MapReduce是在多节点上并行处理大数据的高可扩展的、具有容错能力的编程框架,本文分类设计了基于MapReduce的基本操作模型,以供用户自由自合配置复杂的数据处理逻辑。专家还可以根据领域经验向平台加入可重用可扩展计算分析模块以扩充系统的数据分析处理能力,方便用户进行迭代增量式数据探索。
[Abstract]:In the era of big data, more and more enterprises or organizations from all walks of life will benefit from the development opportunity of big data. Big data analysis can realize the data calculation and comprehensive data insight under the big data scale effect, and help enterprises improve their productivity. Enhance competitiveness and promote enterprise innovation. However, there are many difficulties in big data analysis for many enterprises or organizations. There are still some bottlenecks in database software, big data collection and specification, the development of enterprise professional analysts and so on. In order to meet the needs of big data analysis of enterprises, this paper designs an assistant expert analysis system based on big data analysis, which integrates the functions of data acquisition, data storage, data analysis and data visualization, and provides a friendly man-machine interface. Shielding the details of big data analysis algorithm to help enterprise personnel to use big data analysis technology to carry out high concurrency and fast query, set up complex data processing model, and customize the visualization module to show the value of big data. Promote business promotion and support enterprise decision-making. The auxiliary expert analysis system in this paper synthesizes two PaaS services, ads and Max Compute of Ali Cloud. The latter is a highly extensible, fault-tolerant programming framework based on MapReduce framework to process big data in parallel on multiple nodes. In this paper, the basic operation model based on MapReduce is designed and classified to allow users to freely self-configure complex data processing logic. Experts can also add reusable scalable computing and analysis module to the platform according to domain experience to expand the data analysis and processing ability of the system so as to facilitate the user to explore the data iteratively and incrementally.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】