基于志愿计算的大规模网络分布式分析架构研究

发布时间：2019-04-11 08:55

【摘要】：近些年来,随着复杂网络科学的不断发展,学术界对于复杂网络的研究逐渐深入,因为复杂网络的研究对很多领域的研究都具有指导意义,像其在社会学中展现出的价值或者对于传播学的研究意义,其应用也越来越广泛。但是随着社会进步以及科技发展,网络规模呈现出指数型增长,数据规模更加庞大,同时也面临着数据处理缓慢甚至计算不出的问题,如何更快更高效的处理这些数据成为了近期复杂网络研究的重大挑战之一。并行计算技术为复杂网络的高效计算提供了可能。当前的一些主流计算框架由于具有一定的限制,比如MapReduce在迭代次数多的情况下并没有展现出很好的优势,而复杂网络的计算的特点之一就是具有较多的迭代次数,因此在复杂网络的计算方面并没有显示出很好的优势。志愿计算的核心思想就是将存在于网络中的空闲资源利用,共同参与分布式计算。本文基于志愿计算的思想,以ICE中间件作为通信媒介,构建了针对于大规模复杂网络计算的松耦合分布式计算框架,用于对复杂网络的分析和计算,并将其命名为DCBV框架。框架主要采用了任务队列的思想和方法,在Master/Worker模式的基础上进行设计,增加中间层MiddleWare,将网络内的多台空闲机器动态的结合起来共同参与复杂网络的分析和计算。除此之外,基于对平均最短路径算法的改进,对设计的DCBV计算框架的原型实现进行了实验和评估。实验结果表明,本文设计的框架可以更加高效的计算出复杂网络的相关参数,并且有很好的容错性,可以随时调整计算节点的个数,同时可以随时动态调整每个计算节点分配的线程数以及根据其分配的线程数为不同计算节点分配任务集。改进的平均最短路径算法相比于之前的算法更加高效,并将改进后的算法在松耦合的计算网络框架中实现,获得良好的加速比。
[Abstract]:In recent years, with the continuous development of complex network science, the academic research on complex network is gradually in-depth, because the study of complex network is of guiding significance to many fields of research. Such as its value in sociology or the significance of communication research, its application is becoming more and more extensive. However, with the progress of society and the development of science and technology, the scale of the network is increasing exponentially, and the scale of the data is even larger. At the same time, it is also facing problems that data processing is slow or even impossible to calculate. How to process these data faster and more efficiently has become one of the major challenges in recent complex network research. Parallel computing technology provides the possibility for efficient computing of complex networks. Some of the current mainstream computing frameworks have some limitations, such as MapReduce does not show a good advantage in the case of a large number of iterations, and one of the characteristics of complex network computing is that it has a large number of iterations. Therefore, the computation of complex network does not show a good advantage. The core idea of voluntary computing is to utilize the free resources existing in the network and participate in distributed computing together. Based on the idea of voluntary computing, this paper constructs a loosely coupled distributed computing framework for large-scale complex network computing using ICE middleware as a communication medium. It is used to analyze and calculate complex networks, and it is named DCBV framework. The framework mainly adopts the idea and method of task queue, designs on the basis of Master/Worker pattern, and adds the middle layer MiddleWare, to participate in the analysis and calculation of complex network together with the dynamic combination of many idle machines in the network. In addition, based on the improvement of the average shortest path algorithm, the prototype implementation of the designed DCBV computing framework is tested and evaluated. The experimental results show that the framework designed in this paper can calculate the parameters of the complex network more efficiently, and has good fault tolerance, and can adjust the number of nodes at any time. At the same time, the number of threads allocated by each computing node can be adjusted dynamically at any time, and the task set for different computing nodes can be assigned according to the number of threads allocated by each computing node. Compared with the previous algorithm, the improved average shortest path algorithm is more efficient, and the improved algorithm is implemented in the loosely coupled computing network framework, and a good speedup is obtained.
【学位授予单位】：北京化工大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：O157.5

【参考文献】