面向渲染应用的分布式数据管理与访问优化

发布时间：2018-06-16 15:56

本文选题：分布式数据管理 + 分布式渲染　；参考：《华中科技大学》2013年硕士论文

【摘要】：随着影视行业的蓬勃发展，从传统的2D平面电影时代发展到风靡全球的3D数字化电影时代，而达到这种效果的关键技术之一就是渲染技术。随着单机越来越无法满足渲染应用的处理需求，分布式集群下的渲染应用也逐渐普遍起来，强大的集群计算能力带给渲染应用性能上的巨大提升，但同时由于渲染应用频繁的数据IO访问，使得磁盘IO成为了提升分布式渲染应用性能的瓶颈。基于内存存储的分布式数据缓存系统通过构建一个分布式的共享内存区域的方式，来存放所有分布式渲染应用过程中产生的数据。在高效的分布式内存资源管理模块中，利用内存池状态信息管理机制来实现对共享内存的实时监控；利用内存数据信息管理机制来记录所有存储在共享内存中文件数据的元数据信息，以方便快速查找到它们的位置；利用内存空间申请回收管理机制为上层渲染用户提供内存资源申请与释放的服务，让渲染用户可以自由的使用内存资源。同时在传统分布式数据访问方式的基础上，针对渲染系统的特点提出了两种数据访问优化策略。双层多粒度锁机制，在避免了多个渲染用户的数据访问冲突的前提下，有效的减少了等待延迟的开销。在用户读数据之前的预处理过程中，数据预取优化策略有效的避免了对同一个存储节点重复数据传输情况的发生，减少了网络开销，提高了数据访问的效率。实验表明，面向渲染应用下的分布式数据缓存系统能够很好的满足了渲染应用数据存储和访问需求。完备的分布式内存数据管理为渲染应用提供了可靠的数据保障；针对渲染系统特点的数据访问优化策略，，在满足渲染应用需求的基础上，给渲染应用数据读取带来了性能上的提升。
[Abstract]:With the rapid development of the film and television industry, from the traditional 2D plane film era to the global 3D digital film era, rendering technology is one of the key technologies to achieve this effect. As the single machine is unable to meet the processing requirements of rendering applications, rendering applications in distributed clusters are becoming more and more popular. The powerful computing power of cluster brings great improvement to the performance of rendering applications. However, due to frequent data IO access in rendering applications, disk IO becomes the bottleneck to improve the performance of distributed rendering applications. A distributed data cache system based on memory storage can store all the data generated in distributed rendering applications by constructing a distributed shared memory region. In the efficient distributed memory resource management module, the memory pool state information management mechanism is used to realize the real-time monitoring of shared memory. Using the memory data information management mechanism to record all the metadata information stored in the shared memory file data in order to quickly find their location; Using the memory space request recycling management mechanism to provide the memory resource application and release service for the upper layer rendering user, so that the rendering user can use the memory resource freely. At the same time, based on the traditional distributed data access method, two kinds of data access optimization strategies are proposed according to the characteristics of the rendering system. Double-layer multi-granularity locking mechanism can effectively reduce the overhead of waiting delay while avoiding the collision of multiple rendering users' data access. In the preprocessing process before the user reads the data, the data prefetching optimization strategy effectively avoids the repeated data transmission to the same storage node, reduces the network overhead, and improves the efficiency of data access. Experimental results show that the distributed data cache system for rendering applications can meet the requirements of data storage and access in rendering applications. The complete distributed memory data management provides reliable data guarantee for the rendering application, and the data access optimization strategy for the characteristics of the rendering system, on the basis of satisfying the requirements of the rendering application, It improves the performance of rendering application data reading.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.41;TP333

【参考文献】