基于Hadoop的校园卡数据挖掘的研究与实现

发布时间：2018-06-24 01:43

本文选题：校园卡 + hadoop　；参考：《南昌航空大学》2017年硕士论文

【摘要】：随着高校内的各种业务系统不断增加,高校内积累的师生数据急剧的增长,已经形成了典型的大数据环境。校园卡作为数字校园的一部分,存储着所有师生的各种校内活动的记录,包括食堂餐饮消费记录、开水消费记录、超市购物记录、图书馆出入记录、电费缴纳记录、图书借阅记录、体育场馆使用记录等。这些记录当中隐藏着大量有价值的信息,但我们很难凭借直观的感觉发现它,必须通过数据挖掘的方法挖掘出来。通过对这些数据的深入挖掘,发现其中的有价值的信息,学校管理者就能对师生的消费规律、学习情况有一个更理性、清晰的认识。这将为高校资源的合理分配,校园的规划建设及师生的管理等工作提供有价值的参考。本文基于校园卡近几年来产生的大量数据,采用主流的Hadoop生态下的大数据处理框架进行校园卡数据的清洗、分析、挖掘等工作。首先,本文分析了挖掘校园卡数据的重要性及其相关技术的研究现状。然后对数据挖掘中使用的Hadoop相关技术(HDFS文件系统,Hive数据仓库,MapReduce分布式计算框架)、FP-Growth算法及决策树算法进行了介绍。最后,采用sqoop、Hive等技术,对校园卡数据建立以校园消费为主题的数据仓库。在此数据仓库之上做了以下三项工作:第一、统计各个时间段中各食堂就餐人数,发现了在校就餐人数的周期性变化,对学校的早中晚就餐高峰时间也有了一个更直观的认识。第二、统计学生的各类消费金额,使用C4.5决策树算法建立学生贫困程度预测模型,通过剪枝等优化手段后评估准确率达到85.4%,对学校的贫困生评定有一定的参考价值。第三、统计学生常去商家,运用FP-Growth算法挖掘出大量频繁模式,得出大量学生与商户之间、商户与商户之间的关联规则,使得学校及商户对学生的消费习惯有更清晰的认识。目前大多数高校的信息化平台还只关注在建立事务管理系统,对数据挖掘的运用还不多见。相信随着大数据、机器学习等技术的不断发展,校园数据的分析挖掘在辅助学校管理中将会扮演越来越重要的角色。
[Abstract]:With the increasing of various business systems in colleges and universities, the accumulation of data between teachers and students in colleges and universities is increasing rapidly, which has formed a typical big data environment. As part of the digital campus, the campus card stores records of all kinds of campus activities of teachers and students, including dining hall consumption records, boiling water consumption records, supermarket shopping records, library entry and exit records, and electricity payment records. Books borrowing records, stadiums use records, etc. There is a lot of valuable information hidden in these records, but it is difficult to find it by intuitive sense, which must be mined by the method of data mining. Through the deep mining of these data, find out the valuable information, the school administrator can have a more rational and clear understanding of the law of the consumption of teachers and students and the situation of learning. This will provide valuable reference for the rational allocation of university resources, the planning and construction of campus and the management of teachers and students. Based on a large number of data generated by campus cards in recent years, this paper adopts the big data processing framework under the mainstream Hadoop ecology to clean, analyze and mine campus card data. Firstly, this paper analyzes the importance of mining campus card data and the research status of related technologies. Then the FP-Growth algorithm and decision tree algorithm are introduced for Hadoop related technology (HDFS file system Hive data warehouse / MapReduce distributed computing framework) used in data mining. Finally, using sqoop Hive and other technologies, the campus card data warehouse with the theme of campus consumption is established. The following three tasks have been done on this data warehouse: first, the number of dining rooms in each time period has been counted, the periodic changes of the number of diners in school have been found, and a more intuitive understanding of the peak time of the morning, middle and late dining in the school has also been obtained. Secondly, statistics of all kinds of consumption amount of students, using C4.5 decision tree algorithm to establish the model of student poverty degree prediction, through pruning and other optimization means, the accuracy of evaluation reached 85.40.It has a certain reference value for the evaluation of poor students in schools. Thirdly, statistics students often go to business, use FP-Growth algorithm mining out a large number of frequent patterns, get a large number of students and merchants, business and business between the association rules, so that schools and businesses have a clearer understanding of students' consumption habits. At present, the information platform of most colleges and universities only pays attention to the establishment of transaction management system, and the application of data mining is rare. It is believed that with the development of big data, machine learning and other technologies, the analysis and mining of campus data will play a more and more important role in assisting school management.
【学位授予单位】：南昌航空大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：G647;TP311.13

【参考文献】