教育部数据中心数据管理优化策略研究

发布时间：2018-03-18 06:27

本文选题：教育数据管理　切入点：成本优化　出处：《北京科技大学》2017年博士论文　论文类型：学位论文

【摘要】：针对教育数据的快速增长和巨大分析需求,考虑信息安全和基础网络、软硬件资源等因素制约,本文研究教育数据存储的优化管理、任务调度、容错部署等问题,以降低教育数据在管理上的总体成本,提升系统效率和可靠性。论文的主要研究成果有：(1)针对教育数据中心的传统竖井式建设模式成本高、无差别数据存储造成存储空间利用率不高的问题,选取在成本和横向扩展能力上更合适的分布式并行处理方式,提出了基于相关性的列组合聚类及列组合配置策略。在界定搜索查询热度值的基础上构建带有查询热度值的数据列组合,考虑集群中节点负载情况,提出了最小存储成本策略,在降低存储空间成本的同时有效提高教育数据查询分析效率。(2)针对教育数据查询分析面对的巨大压力和响应要求,对教育数据分析任务进行量化和优先级划分,综合考虑服务成本,量化节点分析承载能力,提出了“基于综合效用的教育数据调度模型”。基于系统的整体负载率,提出了优化的任务调度优化策略。(3)针对教育数据的列组合特性,提出数据重要性的界定方法,基于节点负载和数据重要性提出了数据选择性恢复策略,提高了系统的可用性。(4)针对教育分析系统分布式数据节点故障问题,提出了节点故障下资源优化部署模型和部署策略,保证数据的完整性和可用性,并给出了教育数据中心的运维保障策略,以提高系统的服务可靠性。
[Abstract]:In view of the rapid growth of educational data and the huge demand for analysis, considering the constraints of information security and basic network, hardware and software resources, this paper studies the optimization management of educational data storage, task scheduling, fault-tolerant deployment, and so on. In order to reduce the overall cost of educational data management, improve the efficiency and reliability of the system. Due to the low utilization of storage space caused by the nondifferential data storage, a more suitable distributed parallel processing method in cost and horizontal expansion ability is chosen. In this paper, a column combination clustering and column combination configuration strategy based on correlation is proposed. Based on defining the search query heat value, a data column combination with query heat value is constructed. Considering the node load in the cluster, a minimum storage cost strategy is proposed. In order to reduce the cost of storage space and improve the efficiency of educational data query and analysis, the paper quantifies and prioritizes the task of educational data analysis, aiming at the enormous pressure and response demand of educational data query and analysis. Considering the cost of service and quantifying the load carrying capacity of nodes, this paper puts forward a "educational data scheduling model based on comprehensive utility", which is based on the overall load rate of the system. This paper proposes an optimized task scheduling optimization strategy. (3) according to the column combination characteristics of educational data, a method to define the importance of data is proposed, and a data selective recovery strategy based on node load and data importance is proposed. Aiming at the problem of distributed data node failure in educational analysis system, this paper proposes a resource optimal deployment model and deployment strategy under node failure to ensure the integrity and availability of data. In order to improve the reliability of the system, the operation and maintenance strategy of the educational data center is given.
【学位授予单位】：北京科技大学
【学位级别】：博士
【学位授予年份】：2017
【分类号】：G40-058

【相似文献】