基于大数据的高考志愿数据分析关键技术研究

发布时间：2019-06-08 08:34

【摘要】：随着高考网上志愿填报系统的普及,招生管理单位掌握了大量的考生填报数据。然而由于缺乏高效的分析平台,造成这些数据未能得到有效的利用。而大数据技术的出现,使得这一问题能够得以解决。在此背景下,本文就目前流行的Hadoop分布式处理平台进行深入研究,并围绕高考志愿填报数据分析开展了一系列研究工作,主要研究内容及创新点如下:(1)对分布式处理平台的架构和原理进行了分析和研究,并对Hadoop分布式处理平台的架构进行阐述,分析了Hadoop文件系统,MapReduce计算模型的结构和设计思想。(2)针对大数据处理基本流程,结合高考志愿填报数据的特点并在借鉴了Hadoop设计模式的基础上,设计了一个由核心控制节点、数据预处理节点、计算节点和监控节点组成的分布式数据处理模型,以满足高考志愿数据分析的需求。(3)提出并实现了一种基于遗传算法的任务调度算法,将任务的执行时间和成本纳入到考虑范围,降低任务消耗时间,节约运行成本。对比实验表明,该算法同Hadoop平台采用的FIFO调度算法相比,在任务总响应时间和任务执行成本方面有显著降低。(4)提出了一个改进的协同过滤高考志愿推荐算法,并实现了算法的并行化。实验结果表明,该算法可以为高考考生提供准确的志愿推荐。通过对比串行算法和并行算法的执行效率,验证了算法在不同数量节点下的运行效率。
[Abstract]:With the popularity of the online voluntary filling system for college entrance examination, the enrollment management unit has mastered a large number of candidates to fill in the data. However, due to the lack of an efficient analysis platform, these data can not be used effectively. With the emergence of big data technology, this problem can be solved. Under this background, this paper makes an in-depth study on the popular Hadoop distributed processing platform, and carries out a series of research work around the analysis of the voluntary filling data of the college entrance examination. The main research contents and innovations are as follows: (1) the architecture and principle of distributed processing platform are analyzed and studied, and the architecture of Hadoop distributed processing platform is described, and the Hadoop file system is analyzed. The structure and design idea of MapReduce computing model. (2) according to the basic process of big data processing, combined with the characteristics of college entrance examination voluntary filling data, and on the basis of drawing lessons from Hadoop design pattern, a core control node is designed. A distributed data processing model composed of data preprocessing node, computing node and monitoring node is proposed and implemented to meet the needs of voluntary data analysis of college entrance examination. (3) A task scheduling algorithm based on genetic algorithm is proposed and implemented. The execution time and cost of the task are taken into account to reduce the task consumption time and save the operation cost. The experimental results show that compared with the FIFO scheduling algorithm used in Hadoop platform, the total task response time and task execution cost of this algorithm are significantly reduced. (4) an improved cooperative filtering voluntary recommendation algorithm for college entrance examination is proposed. The parallelization of the algorithm is realized. The experimental results show that the algorithm can provide accurate voluntary recommendation for college entrance examination candidates. By comparing the execution efficiency of serial algorithm and parallel algorithm, the running efficiency of the algorithm under different number of nodes is verified.
【学位授予单位】：长春理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：G637;TP311.13

【参考文献】