基于Hadoop的校园一卡通消费分析系统的设计与实现

发布时间：2018-02-04 07:55

本文关键词： 大数据 Hadoop 一卡通消费分析　出处：《内蒙古农业大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着大数据技术的高速发展,与之结合的领域亦是越来越多,商业网购的用户数据分析、生物信息学的基因比对、关于系统运维的日志分析等等,大数据技术的应用价值在众多方面得到了体现。而校园一卡通管理系统,在大学信息化管理方面,做出了十分重要的贡献,但对于校园一卡通系统里存储的大量数据及这数据里边蕴含着的大量的商业信息与管理信息,现有的分析研究却是较为缓慢的。大数据技术的到来,无疑给了研究人员研究校园一卡通里的这些数据带来了全新的技术支持,用大数据技术去分析这些数据,并在其中找出潜在的联系和规律,对于校园建设与管理工作,有着难以估量的决策参考作用。所以,建立一个基于大数据平台的校园一卡通消费分析系统具有现实意义。把校园一卡通系统里的交易数据作为分析研究对象,运用Hadoop生态系统的大数据相关技术对其展开分析研究,建立起辅助于校园一卡通管理系统的消费分析系统。首先阐述了 Hadoop平台的核心应用HDFS分布式存储系统和MapReduce分布式并行计算处理机制、Hive数据仓库工具和Sqoop数据传递工具。然后对学校决策层和工作人员的探索消费分析的实际工作需求进行了解和分解,得到该消费分析系统的需求分析。并运用常用的Hadoop业务处理框架对需求分析进行设计与实现,使用Sqoop工具将一卡通交易数据从关系型数据库中导入HDFS存储系统,运用MapReduce技术对数据进行预处理,再运用Hive工具对数据进行统计分析并再次导入MySQL数据库进行web展示。论文还使用MapReduce技术对交易数据进行了另外的分析技术尝试和探索。最后初步得到了可提供于学校决策层和工作人员的关于消费信息的建设性建议。论文的工作表明基于Hadoop的校园一卡通消费分析系统的设计与实现是非常有意义的,同时,现在的工作也为明天的学校综合信息分析和数据挖掘工作奠定了基础,为下一步构造完整的分析平台和提供更深远的决策建议进行了有效的探索。
[Abstract]:With the rapid development of big data technology, there are more and more fields combined with it, such as data analysis of commercial online shopping, gene alignment of bioinformatics, log analysis of system operation and maintenance, and so on. The application value of big data technology has been reflected in many aspects. And campus one-card management system, in the university information management, has made a very important contribution. But for the campus card system stored a large number of data and the data contained in a large number of business information and management information, the existing analysis and research is relatively slow. The arrival of big data technology. It undoubtedly gives researchers new technical support to study the data in campus cards, using big data technology to analyze the data, and to find out the potential links and patterns in the data. For campus construction and management work, there is an incalculable role of decision-making reference. It is of practical significance to establish a campus one-card consumption analysis system based on the big data platform. The transaction data in the campus one-card system is taken as the research object. Using the big data technology of Hadoop ecosystem to analyze and study it. A consumption analysis system is established to assist the campus card management system. The core of Hadoop platform is HDFS distributed storage system and MapReduce distributed parallel computing mechanism. Hive data Warehouse tools and Sqoop data transfer tools. Then the school decision-making level and staff exploration consumption analysis of the actual work needs to understand and decompose. Get the demand analysis of the consumption analysis system, and use the commonly used Hadoop business processing framework to design and implement the demand analysis. Using the Sqoop tool to import the card transaction data from the relational database into the HDFS storage system, using MapReduce technology to pre-process the data. Then we use the Hive tool to analyze the data and import the MySQL database to show the web again. This paper also uses the MapReduce technology to carry on the other analysis technology to the transaction data. Finally, some constructive suggestions on consumption information for school decision makers and staff are obtained. The work of this paper shows the design of campus one-card consumption analysis system based on Hadoop. Planning and implementation is very meaningful. At the same time, the present work has also laid the foundation for the school comprehensive information analysis and the data mining work tomorrow, has carried on the effective exploration for the next step constructs the complete analysis platform and provides the more profound decision suggestion.
【学位授予单位】：内蒙古农业大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【参考文献】