Hadoop集群性能测试框架的插件配置管理与可视化测试模块的设计与实现

发布时间：2018-05-29 06:30

本文选题：Hadoop + 性能测试　；参考：《南京大学》2015年硕士论文

【摘要】：近年来,“大数据”这个词被越来越多地提及。传统的软件架构在处理企业大数据时越来越力不从心。在这样的背景下,受Google Big Table技术的启发,Hadoop应运而生。如今,各种各样的基于Hadoop的产品或者开源技术层出不穷,技术发展如火如荼。然而,Hadoop产品的测试却仅限于Hadoop产品内置的测试工具与第三方组织开发的benchmark,鲜有对Hadoop各个组件进行可靠性测试的测试框架,这就使得用户在使用Hadoop产品时,往往由于负载大而遇到系统宕机等不可预知的情况,这其实是Hadoop产品测试人员的一种失职。在这样的背景下,某公司设计开发了Hadoop集群性能框架,而本人作为开发者一员,完成了性能测试框架的插件配置管理与可视化测试模块的设计与实现。在设计的时候,使用“插件”的概念,使得框架调度与具体测试代码实现物理上的分离。使用Maven技术来管理项目构建与依赖管理,简化项目管理,同时还可以利用Maven生命周期在项目构建过程中加入定制化的功能,例如类生成与资源处理；使用JAXB技术来实现配置类的自动生成与配置的验证解析,简化Model类的编写与复杂的文件解析；项目广泛使用spring技术,来管理各种Bean的自动装配与创建,通过Sprin g的Application Context管理不同插件运行时的不同环境。插件配置管理模块作为服务提供模块,将插件从物理Bundle转换到逻辑概念,供调度系统调用,使得底层信息获取与文件操作对于上层来说是透明的；可视化测试模块则是为测试用户提高测试效率,将测试人员从手动配置系统的繁琐的linux操作中解放出来。利用可视化测试模块,用户可以新建测试,查看测试进度与监控资源状态,并可以实时查看测试日志,测试结束时,用户可以查看测试历史记录,下载测试报告与日志。利用可视化测试模块,测试用户的测试工作变得非常简单。
[Abstract]:In recent years, the word "big data" has been increasingly mentioned. Traditional software architecture is more and more difficult to deal with enterprise big data. Under this background, Google Big Table technology inspired Hadoop came into being. Today, a variety of Hadoop-based products or open-source technologies emerge in endlessly, technological development is in full swing. However, the testing of Hadoop products is limited to the test tools built into Hadoop products and benchmark developed by third-party organizations. There are few test frameworks for testing the reliability of Hadoop components, which makes users use Hadoop products. Often because of the heavy load and encounter system outages and other unpredictable situations, this is in fact a Hadoop product testers dereliction of duty. Under this background, a company has designed and developed the Hadoop cluster performance framework, and as a developer, I have completed the design and implementation of plug-in configuration management and visual test module of the performance testing framework. In the design, the concept of "plug-in" is used to separate the frame scheduling from the specific test code. Using Maven technology to manage project construction and dependency management, simplify project management, and add customized functions, such as class generation and resource processing, in the process of project construction using Maven lifecycle. The JAXB technology is used to realize the automatic generation of configuration classes and the verification and parsing of configuration, which simplifies the writing of Model classes and complex file parsing. Spring technology is widely used in the project to manage the automatic assembly and creation of various Bean. Through Sprin g's Application Context, different environments of different plug-in runtime are managed. As a service providing module, plug-in configuration management module converts plug-in from physical Bundle to logical concept for scheduling system call, which makes the access and file operation of the underlying information transparent to the upper layer. The visual test module is designed to improve the test efficiency for the test users and free the testers from the tedious linux operation of the manual configuration system. With visual test module, users can build new tests, check test progress and monitor resource status, and can view test log in real time. At the end of test, users can view test history and download test report and log. Using visual test module, testing user test work becomes very simple.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2015
【分类号】：TP311.52

【参考文献】