基于国产超级计算机的渲染农场的设计与开发

发布时间：2018-03-21 06:24

本文选题：超级计算机　切入点：集群渲染　出处：《山东大学》2016年硕士论文　论文类型：学位论文

【摘要】：随着影视产业的快速发展,电影的画面正变得越来越绚丽、逼真。但是有很多的电影画面无法通过正常的拍摄完成,只能通过后期的建模、特效制作、渲染来完成。电影分辨率的提高带来的单帧渲染时间的增长,以普通的4K电影为例,使用主流的工作站,渲染一帧的时间从几小时到十几个小时,遇上光源众多、场景复杂、材质特殊的场景甚至会到几十个小时。而且现在许多的电影都是3D效果,这时候就得分为左右眼分别进行渲染。虽然单机计算能力也在快速增长,但已经无法满足电影产业的需求,于是集群渲染渲染开始变得流行,集群渲染系统通常被称为渲染农场。目前国内外构建的渲染平台大都基于普通的小型集群,同时也很少去考虑调度策略,更多的是由工作人员进行手动划分资源和分配任务,并且没有针对渲染这一特殊的应用进行文件读写的优化,只是每一个结点单独的渲染一帧。并没有考虑渲染这一应用的特性：渲染既是一个高密度计算型的应用,也是一个数据密集型的应用,而且内存的占用率也很高,而且同一个镜头的渲染任务存在着大量的共享数据。考虑到渲染的这些特性,基于超级计算机来搭建渲染平台就特别合适,因为超级计算机的结点间互访的能力远远地强于集群。本文主要工作是在国产超级计算机的环境下,搭建一个高效能的渲染平台。集群的使用具有一定的复杂性,于是我们基于GridSphere框架开发了一个渲染系统的门户网站,门户网站的设计参考网络渲染的一般流程：上传场景描述文件,提交渲染作业,查看渲染过程,下载结果。门户网站的设计还能隐藏集群操作,完成对用户的管理,使用计费等。同时对镜头级别的任务进行第一级的调度。本文针对同一镜头内的渲染任务存在大量的共享数据的特点,重新设计并实现了渲染单元和数据管理软件,渲染单元为数据管理软件的运行提供了一个框架,而且直接调度渲染引擎,数据管理软件对这些共享数据的读写进行优化,减少文件服务器的访问次数和访问数据量,避免I/O拥堵,而且给引擎提供透明的访问。同时完成镜头内渲染任务的第二级调度,从而实现整体系统多级调度,以此适应超大规模的渲染任务并发。渲染单元与数据管理软件的设计是为了使渲染引擎能够更加高效的运行在超级计算机这一平台上。本文针对整个系统进行往国产超级计算机展开了移植工作。系统的开发是山东大学高性能中心进行,但是超级计算机在操作系统,处理器架构和库文件等方面和普通的集群具有较大的差异性,所以我们在无锡超级计算中心的太湖之光和广州超级计算中心的天河二号进行了移植工作,并分别进行了大于30万核心并行规模的性能测试。本文中提到的渲染引擎是由山东大学自主研发的符合RenderMan规范的渲染引擎RenderWing。本文所做的工作为国家863项目《真实感动漫渲染系统应用与研究》的一部分。
[Abstract]:With the rapid development of the film industry, the film is becoming more and more beautiful, lifelike. But there are a lot of pictures not through normal shooting, only through the latter part of the modeling, special effects production, rendering to complete. It improved resolution of single frame film brings the dyeing time of growth, with ordinary 4K film with the use of mainstream workstations, rendering a frame from a few hours to ten hours, the light source is numerous, in complex scenes, material special scenes even dozens of hours. But now many of the films are 3D effect, this time on the score of left and right eyes were rendered. Although single computing ability also in the rapid growth, but has been unable to meet the demand of the film industry, so the cluster render started to become popular, cluster rendering system is often called the render farm. At home and abroad to build the rendering platform Taiwan are small clusters based on the common, and there are few to consider scheduling strategy, more by the staff manual division of resources and distribution of tasks, and not optimized for reading and writing files for rendering this special application, but each node separate rendering frame. Without considering the characteristics of this rendering application: rendering is a type of high density computing applications, application is a data intensive, but the memory occupancy rate is also high, but with a shot of the rendering task is shared by a large amount of data. Considering these characteristics render, based on super computer to build a special rendering platform right, because of the ability of super computer nodes is far stronger than cluster visits. The main work of this paper is made in the super computer environment, build a high-performance cluster rendering platform. With a certain complexity, so we based on the GridSphere framework for the development of a rendering system portal, the design process of the reference network rendering portal: upload scene description file, render operation, check the rendering process. The design can also download portal hidden cluster operation, to complete the user management. Use billing. At the same time, lens level tasks for the first level scheduling. In this paper, the rendering task for the same camera inside the existing shared data characteristics, we design and implement a rendering unit and data management software, provides a framework for running a rendering unit for data management software, but also directly scheduling the rendering engine data management software, on the shared data read and write optimized file server, reduce the number of accesses and the amount of data access, to avoid congestion I/O, And the engine to provide transparent access. Second level scheduling complete lens rendering tasks at the same time, in order to achieve the overall multilevel scheduling system, in order to adapt to the rendering task. Large scale concurrent design rendering unit and the data management software is to make the rendering engine to more efficiently run on supercomputers on this platform. This paper to the national super computer launched the transplant work for the entire system. The development of the system is the Shandong University high performance center, but in the super computer operating system, processor architecture and library files and other common cluster has great difference, so we at the Wuxi Supercomputing Center of Taihu light and the Guangzhou Supercomputing Center of the Milky way two the number of transplantation, and were more than 300 thousand core parallel performance test scale. Rendering mentioned in this article The engine is developed by Shandong University independently, which is in line with the RenderMan specification of the rendering engine RenderWing.. The work in this paper is part of the National 863 project, the application and research of the realistic animation rendering system.

【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP338

【相似文献】