开放式海量数据处理服务的计算完整性研究

发布时间：2018-04-21 22:25

本文选题：开放式海量数据处理服务 + 计算完整性　；参考：《国防科学技术大学》2014年博士论文

【摘要】：开放式海量数据处理服务在大数据处理中发挥着越来越重要的作用。然而,由于开放式服务可能面临来自服务方主观意图以及内部分布式计算环境客观安全因素等威胁,如何保证服务的计算完整性成为重要问题。现有研究主要对海量数据处理的内部计算框架进行研究,通过多副本技术对计算节点提供的结果进行检测,从而保证计算任务的计算完整性。由于基于多副本的技术将会带来很大的计算开销,因此限制了方法的实际可用性;并且这类针对内部计算节点的检测机制并不能有效解决服务方主观欺骗造成的计算完整性问题。针对开放式海量数据处理服务计算完整性问题,本文从计算完整性检测与计算完整性保障两个方面展开研究。计算完整性检测问题关注从用户角度对服务方的计算行为以及结果的完整性检测,属于计算完整性的事后检查;计算完整性保障问题则从服务方角度研究如何组织可信的计算资源得到满足计算完整性要求的计算结果,属于计算完整性的主动保护。本文以当前海量数据处理的主流计算模式Map Reduce为研究对象,结合海量数据处理的计算特点,从提高方法的可用性与计算效能出发,以降低计算完整性检测机制的性能开销、提高计算完整性保障力度为优化目标,系统的研究了开放式海量数据处理服务计算完整性方面的若干重要问题。本文的主要研究内容及创新点包括以下几方面:第一,研究了基于第三方的计算完整性检测问题。云服务模式下,建立可控的云计算安全监管体系是云服务可信研究要解决的重要挑战,对服务进行基于第三方的审计是其中的重要手段。在Map Reduce计算机制中,Map计算为对用户原始输入的处理,是计算的重要组成部分。本文提出了“基于第三方可信抽样的Map阶段计算完整性检测”机制,通过可信第三方对Map Reduce中间结果进行抽样检测,以少量的检测开销来检测服务方Map阶段的计算完整性,并且针对服务商可能存在的主观上的不配合问题,利用Merkle树技术对检测结果进行组织,防止服务方为应付审计而进行欺骗行为,保证检测结果真实可靠。第二,研究了计算完整性用户自主检测问题。在云服务监管体系的建立完善之前,研究服务方不感知的用户自主检测方法也成为解决问题的有效途径。本文研究了“基于监控探针的计算完整性用户自主检测方法”,根据Map Reduce计算问题的类型构建预先知道计算结果的监控探针并注入到输入数据集合中,通过探针数据的计算结果来以一定概率检测整体计算任务是否满足计算完整性要求。由于该方法与具体的计算类型相关,因此,本文重点对该方法进行建模,对其重要性质进行研究,并针对几种典型的Map Reduce计算类型研究监控探针的构造方法。该方法可以对Map与Reduce的全部计算阶段进行完整性检测,并且无需服务方的配合即可得出检测结果;同时由于该方法基于抽样检测思想,检测的开销可以被用户所接受。第三,研究了可信开放式Map Reduce系统构建问题。在服务方内部,当服务方利用开放式计算资源来组织计算系统时,由于计算资源可能来自不同的可信域,因此,需要对各节点的计算结果进行检测,只有通过检测的结果才能被采纳。当前的检测手段主要通过多副本检测,而对于多副本机制在抵御共谋攻击方面的弱点,则缺乏高效的解决手段。本文提出了“抗共谋的开放式环境下可信Map Reduce系统构建”方法,该方法无需针对共谋攻击设计额外的检测机制,仅利用多副本检测的历史信息,即可定位共谋与非共谋攻击模式下的恶意节点。该方法利用完整性证明图来描述系统内的节点间多副本检测关系,并基于完整性证明图的最大团分析来精确定位恶意节点。并且,该方法还提出了基于完整性证明图指导的节点检测对选择启发式算法,指导检测副本对的选择,提高恶意节点检测的效率。第四,研究了计算节点可信性评估问题。在海量数据处理中,尽管多副本技术的检测精度高,但是由于参与计算的节点数量大,如果完全应用多副本技术来进行检测,会导致巨大的计算开销。如果能够利用很小的检测成本,对节点的可信性进行预评估,然后在此基础上,在实际运行的系统中针对可信度低的节点再应用多副本检测,将大大提高检测机制的计算效率。本文提出了“基于监控探针的计算节点可信性评估方法”,通过探针数据的计算结果判断探针在系统中是否被正确执行,并结合Map Reduce的Shuffle机制,确定监控探针的执行路径,通过信誉机制对各参与计算的节点进行可信性评估。该方法工作在应用级,无需对计算框架进行修改。而通过对可信节点的评估,可以对那些低可信性排名的节点进行精度更高的多副本检测,从而有效降低检测机制的计算资源需求。综上所述,本文对开放式海量数据处理服务的计算完整性问题进行了深入的研究,提出了具有高可用性、高检测率、低开销的解决方案,并通过理论分析和大量的实验验证了所提出方法的有效性和性能,对于建立诚实可信的开放式海量数据处理服务环境具有一定的理论意义和应用价值。
[Abstract]:Open mass data processing services play a more and more important role in large data processing. However, because open services may face threats from the subjective intention of the service parties and the objective security factors in the internal distributed computing environment, how to ensure the integrity of the service has become an important problem. The internal computing framework of data processing is studied to detect the results of computing nodes by multi replica technology to ensure the computational integrity of computing tasks. Because the multi replica based technology will bring a lot of computing overhead, the practical availability of the method is limited; and this kind of inspection for internal computing nodes. The measurement mechanism can not effectively solve the computing integrity problem caused by the subjective deception of the service party. Aiming at the integrity problem of the open mass data processing service, this paper studies the two aspects of the integrity detection and the security of computing integrity. And the integrity detection of the results, it belongs to the ex post examination of computing integrity; the problem of computing integrity protection is to study how to organize the trusted computing resources to meet the computing integrity requirements from the point of view of the service side, which belongs to the active protection of computing integrity. This paper is based on the mainstream computing model Map of the current mass data processing. Reduce is the research object. Combining with the computing characteristics of mass data processing, starting from improving the availability and computing efficiency of the method, it reduces the performance overhead of the computing integrity detection mechanism and improves the strength of computing integrity as an optimization goal. The system has studied several important aspects of the integrity of open mass data processing services. The main research contents and innovation points of this paper include the following aspects: first, it studies the problem of computing integrity detection based on third parties. Under the cloud service mode, the establishment of a controlled cloud computing security supervision system is an important challenge to solve the cloud service trust research. The audit of the service based on the third party is important. Means. In the Map Reduce computer system, Map is an important part of the computation for the processing of the original input to the user. This paper proposes a "Map phase computation integrity detection based on third party trusted sampling", which is sampled by a trusted third party to the intermediate result of Map Reduce, and detects the clothing with a small amount of detection overhead. The computing integrity of the Map phase, and the possible subjective non coordination of the service providers, using the Merkle tree technology to organize the detection results, to prevent the service parties from cheating on the audit, and to ensure the true and reliable results of the detection. Second, the problem of independent testing of the computing integrity users is studied. Before the establishment and perfection of the supervision system, the method of user independent detection which is not perceived by the service party is also an effective way to solve the problem. This paper studies the method of "computing integrity user independent detection based on monitoring probe". According to the type of Map Reduce computing problem, the monitoring probe that pre know the results is built and injected into the monitoring system. In the input data set, the calculation results of the probe data are used to determine whether the overall computing task is satisfied with the computing integrity requirements. Because the method is related to the specific type of calculation, this paper focuses on the modeling of the method, studies its important properties, and aims at several typical Map Reduce computing types. Study the construction method of monitoring probe. This method can carry out integrity detection for all calculation stages of Map and Reduce, and can get the detection results without the need of the cooperation of the service side. At the same time, the detection cost can be accepted by the user because of the method based on sampling detection. Third, the construction of the trusted open Map Reduce system is studied. In the service side, when the service party organizes the computing system with open computing resources, the computing resource may come from different trusted domains. Therefore, it needs to detect the results of each node, only through the result of detection. The replica mechanism is lack of efficient solution to the weakness of conspiracy attack. This paper proposes a "trusted Map Reduce system construction under an anti conspiracy open environment". This method does not need to design an additional detection mechanism for conspiracy attack and only uses the historical information of multi copy detection to locate conspiracy and non conspiracy. A malicious node under attack mode. This method uses integrity proof graph to describe the multiple copy detection relationship among nodes in the system, and accurately locates the malicious nodes based on the maximum group analysis of the integrity proof graph. Furthermore, this method also proposes a heuristic algorithm based on the integrity proof diagram to guide the detection of the selection. The selection of copy pairs improves the efficiency of detection of malicious nodes. Fourth, the reliability evaluation of computing nodes is studied. In the process of massive data processing, although the detection precision of multi copy technology is high, the number of nodes involved in the computation is large, and if the multiple copy technology is used to detect it, it will lead to huge computational overhead. It can make use of small detection cost and pre evaluate the credibility of nodes. Then, using multi copy detection for low reliability nodes in the actual running system, it will greatly improve the computing efficiency of the detection mechanism. This paper proposes a method for evaluating the credibility of the computing nodes based on the monitoring probe. The calculation results of the probe data determine whether the probe is properly executed in the system, and combines the Shuffle mechanism of the Map Reduce to determine the execution path of the monitoring probe. The credibility mechanism is used to evaluate the credibility of the nodes involved in the calculation. In this paper, the computational integrity of the open mass data processing service is deeply studied, and the solutions with high availability, high detection rate and low overhead are proposed. Through theoretical analysis and a large number of experiments, the effectiveness and performance of the proposed method are verified. It has a certain theoretical significance and application value for the establishment of an honest and credible open mass data processing service environment.

【学位授予单位】：国防科学技术大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：TP393.08

【相似文献】