MapReduce环境下的性能异常检测和资源调度方法

发布时间：2018-05-28 10:03

本文选题：云计算 + MapReduce　；参考：《北京邮电大学》2013年硕士论文

【摘要】：MapReduce是由Google提出的一个广为人知的编程框架,Hadoop开源实现了这一框架。因为MapReduce适合处理大规模数据,许多企业都采用其进行数据挖掘,数据存储等。MapReduce需要一个调度策略来决定工作如何执行以及工作执行过程中的资源分配,目前许多调度策略主要是为了提高集群资源利用率,而没有充分考虑一个工作对于完成时间的要求。此外,MapReduce是一个架构在廉价设备上的十分复杂的系统,经常会有异常发生,能否及时检测到系统的异常并进行处理对于系统的正常高效运行十分重要。本文针对以上的两点问题进行了研究： 1)针对资源调度问题,本文提出了一种调度机制以保证集群中运行的每个工作都能够按时完成,从而达到其性能要求。和其他的调度策略相比,本文的方法能够预测一个工作的运行状况,并根据预测结果合理地分配资源给每个工作,以尽量避免不必要的资源浪费。调度策略在一个仿真环境中进行了评估,结果表明本文的方法能够保证工作在其预期时间内完成并能够节省资源。 2)针对异常检测问题,本文提出并分析了一种MapReduce环境下的异常检测方法。该方法基于相似节点理论,通过运用密度聚类的方法实时分析系统的性能指标来检测异常。本文还对相似节点理论和异常检测算法进行了实验验证。和现有的其他方法相比,本文提出的方法具有处理过程简单、算法复杂度低、检测灵敏度高且适于在线和离线检测的优点。
[Abstract]:MapReduce is a well-known programming framework proposed by Google. Because MapReduce is suitable for large-scale data processing, many enterprises use it for data mining, data storage, and so on. MapReduce requires a scheduling strategy to determine how work is performed and how resources are allocated during work execution. At present, many scheduling strategies are mainly aimed at improving the utilization of cluster resources, without fully considering the completion time requirement of a single task. In addition, MapReduce is a very complex system based on cheap devices, and there are often exceptions. It is very important to detect and deal with the anomalies in time for the normal and efficient operation of the system. In this paper, the above two problems are studied: 1) aiming at the resource scheduling problem, this paper proposes a scheduling mechanism to ensure that every task running in the cluster can be completed on time, so as to meet its performance requirements. Compared with other scheduling strategies, the proposed method can predict the running status of a job and allocate resources to each task reasonably according to the prediction results, so as to avoid unnecessary waste of resources as far as possible. The scheduling policy is evaluated in a simulation environment. The results show that the proposed method can ensure that the work is completed within the expected time and can save resources. 2) aiming at the problem of anomaly detection, an anomaly detection method in MapReduce environment is proposed and analyzed. This method is based on the theory of similar nodes and detects anomalies by using density clustering method to analyze the performance of the system in real time. The theory of similar nodes and the algorithm of anomaly detection are also verified experimentally in this paper. Compared with other existing methods, the proposed method has the advantages of simple processing, low algorithm complexity, high detection sensitivity and suitable for on-line and off-line detection.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP338.6

【相似文献】