面向分布式内存计算的缓存数据管理系统

发布时间：2018-03-22 00:37

本文选题：内存计算　切入点：分布式处理　出处：《华中科技大学》2016年硕士论文　论文类型：学位论文

【摘要】：面对海量数据,利用图计算或机器学习等具有迭代特征的复杂应用对其进行处理很常见。通过在迭代之间共享缓存数据,典型的分布式内存计算系统如Spark极大地提升了迭代程序处理的速度,在业界使用广泛。然而,在处理的数据量较大时缓存数据无法全部放入内存,需要对缓存数据进行管理。迭代后期再次使用被置换的数据时需要进行重获,此时引入的重获开销影响了性能。传统的置换策略如LRU或FIFO没有考虑到缓存数据重获开销之间存在差异,因而无法保证应用运行过程中整体的重获开销最小。实验发现,在分布式内存计算系统中,缓存数据的重获开销间存在着明显的差异。因此提出并在Spark上实现了一种考虑了重获开销的缓存数据管理系统,来保证数据置换带来的影响最小。首先,依据分布式系统中应用待执行的逻辑是已知的、数据之间有联系的特点,对执行逻辑进行了分析得出了缓存数据之间的依赖关系;然后,定义了重获开销表征置换发生后重获缓存数据所需的代价,定义了重用度表征缓存数据在程序后期再次使用到的次数,在执行程序时实时收集信息并计算这两个指标;最后,设计了置换策略,计算出缓存数据经重用度加权后的单位大小重获开销值,并维护出置换顺序,保证了在置换发生后对应用引入的额外开销最小。在相同的实验环境下,将考虑了重获开销的缓存数据管理系统与Spark中默认的管理系统进行了对比。实验结果表明,考虑了重获开销的缓存数据管理系统在内存空间不足时,因缓存数据置换带来的额外开销显著减少,应用的整体运行时间可以减少30%至50%。
[Abstract]:In the face of massive data, it is very common to process complex applications with iterative characteristics, such as graph computing or machine learning, by sharing cached data between iterations. Typical distributed memory computing systems such as Spark greatly improve the speed of iterator processing and are widely used in the industry. However, cache data cannot be fully stored in memory when the amount of data processed is large. The cached data needs to be managed. When the replaced data is used again at the end of the iteration, The traditional replacement strategies such as LRU or FIFO do not take into account the difference between cache data recovery overhead, so it can not guarantee the minimum overall recovery cost in the running process of the application. In the distributed memory computing system, there are obvious differences between the cost of cache data recovery. Therefore, a cache data management system based on Spark is proposed and implemented. Firstly, according to the fact that the logic of application to be executed in distributed system is known and the data is related, the execution logic is analyzed and the dependency between cache data is obtained. Then, the cost of retrieving the cached data after the replacement is represented is defined, the number of times the cache data is reused in the later stage of the program is defined, the information is collected in real time while executing the program and the two indexes are calculated. Finally, the permutation strategy is designed to calculate the reclaimed cost value of the unit size of the cache data weighted by the degree of reuse, and to maintain the replacement order, which ensures the minimum extra cost to the application after the replacement occurs. In the same experimental environment, This paper compares the cache data management system with the default management system in Spark. The experimental results show that the cache data management system considering the reclaim overhead has insufficient memory space. The overall running time of the application can be reduced by 30% to 50% because of the significant reduction in the extra cost of cache data replacement.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP333;TP315

【相似文献】