基于属性分析的测试用例集优化技术研究

发布时间：2018-09-10 16:19

【摘要】：软件测试是软件质量保障的最重要手段之一。软件测试包括测试用例设计、测试用例执行和测试结果审查等任务。为了减少软件测试的开销,提高软件测试效率,人们提出并实现了一系列自动化测试技术,如自动化测试输入生成、测试用例自动化执行、测试结果自动审查等。自动化测试输入生成往往会生成大量冗余测试用例给后期测试用例结果审查和分析带来大量开销。测试用例集优化技术意在通过减少需要运行、审查和分析的测试用例数量来减少这种开销。本论文研究了现存测试用例集优化技术面临的两个问题。在测试用例的预期输出结果无法自动化获取和比对时,基于程序行为聚类的测试用例集优化技术插桩程序获得程序行为信息,通过对程序行为信息进行聚类以优化测试用例集,减少需要人工审查测试用例结果的开销。这种方法的问题是该类方法分析程序行为时把所有的程序元素同等看待,使得聚类效果并不理想。另一个问题是在测试用例预期输出可以自动获取并比对时,基于程序行为覆盖的测试用例集优化技术可以收集程序执行行为的覆盖并进行分析以优化测试用例集。但本论文在经过对华为的案例调研后发现在测试大型工业系统时出于安全及执行正确性考虑,开发测试人员并不会对系统进行插桩收集程序执行行为,在这种情况下只能分析测试输入和测试用例执行成功和失败的信息以优化测试用例集。本论文提出相应的方法以面对以上两个问题。为了克服基于程序行为聚类的测试用例集优化技术的问题,本论文提出了一种基于属性加权的测试用例集优化技术(Weighted Attributes Based Strategy,WAS)。错误定位技术使用执行剖面和选择的测试用例的执行成功失败信息计算每一个程序元素的可疑值。WAS使用错误定位技术计算得到的可疑值调整初始执行剖面并构造一个加权的测试用例执行剖面。构造的执行剖面用来作为下一轮聚类的输入。本论文使用Crosstab,Jaccard,Ochiai,Tarantula,H3c和H3b这6种广泛研究与使用的基于频谱的错误定位技术作为执行剖面加权方式。实验待测程序为7个广泛使用的开源程序(make,ant,sed,flex,grep,gzip和space)。WAS与4种经典的聚类过滤技术进行对比(one per cluster,n per cluster,adaptive sampling和ESBS)。184个包含单故障和多故障的程序错误版本在实验中用于评估。实验结果表明提出的WAS比其它4种过滤技术在Recall和Precision指标上都有更优异的表现。为了在程序无法插桩时进行测试用例集优化,本论文基于在华为进行的案例研究提出了基于属性选择的测试用例集优化技术(Category Selection Based Adaptive Random Testing,CSBART)。CSBART 在线性自适应随机测试(Linear-Order Algorithm for Adaptive RandomTesting,LART)的基础上使用category 选择方法以消除不重要的category的影响,提高LART选择能够发现故障的测试用例的效率。CSBART使用两种category选择方法。一种是Input Profile(IP)方法。IP的思想是把出现在失败的测试中更频繁的choice定义为失败相关的choice。IP通过统计失败相关的choice出现频率鉴别出与测试用例执行失败密切相关的category。第二种方法为基于互信息(Mutual Information,MI)的category 选择方法。MI基于信息熵的理论计算出不同的category中包含关于测试用例执行成功或失败的信息量。信息量越大的category和测试用例成功或是失败执行相关性越密切。本论文在华为的工业环境中对CSBART进行了实验研究并和随机测试(Random Testing,RT),LART,以及基于聚类的技术n per cluster sampling和adaptive sampling进行对比,结果表明CSBART比其它技术在发现能够揭示故障的测试用例上有更优异的表现。本论文基于WAS和CSBART提出基于属性分析的测试用例集优化方法。WAS方法通过加权技术分析程序执行行为中的不同属性(语句)以实现测试用例集的优化。CSBART方法通过分析测试输入的属性(category)选择重要的属性以实现测试用例集的优化。综合以上两种方法本论文提出了一种基于属性分析的测试用例集优化原型框架,相应工具在华为公司的工业研究案例中实现并通过了专业测试人员的评审。
[Abstract]:Software testing is one of the most important means of software quality assurance. Software testing includes test case design, test case execution and test result review. In order to reduce the cost of software testing and improve the efficiency of software testing, a series of automatic testing techniques are proposed and implemented, such as automatic test input generation and test application. Automated test input generation often generates a large number of redundant test cases and incurs a large amount of overhead for later test case review and analysis. Test case set optimization techniques are intended to reduce this overhead by reducing the number of test cases that need to be run, reviewed, and analyzed. When the expected output of test cases can not be automatically acquired and compared, the test case set optimization technique based on program behavior clustering is used to get the program behavior information. By clustering the program behavior information, the test case set is optimized and the requirement is reduced. The problem with this method is that it treats all program elements equally when analyzing program behavior, making clustering unsatisfactory. Another problem is that the test suite optimization technique based on program behavior coverage is used when the expected output of test cases can be automatically obtained and compared. The coverage of program execution behavior can be collected and analyzed to optimize the test suite. However, after a case study of Huawei, this paper finds that developers and testers do not collect program execution behavior for the sake of safety and correctness in testing large-scale industrial systems. In this case, only In order to overcome the problem of test suite optimization based on program behavior clustering, a test suite optimization technique based on attribute weighting (Weig) is proposed in this paper. Error location techniques use execution profiles and selected test case execution success failure information to compute suspicious values for each program element. WAS uses suspicious values computed by error location techniques to adjust the initial execution profile and construct a weighted test case execution profile. This paper uses Crosstab, Jaccard, Ochiai, Tarantula, H3c and H3b as the input of the next round of clustering. Six widely studied and used spectrum-based error location techniques are used as the weighting method of the execution profile. The tested programs are seven widely used open source programs (make, ant, sed, flex, grep, gzip and space). WAS and four Classical clustering filtering techniques are compared (one per cluster, N per cluster, adaptive sampling and ESBS). 184 error versions of programs with single and multiple faults are evaluated in experiments. The experimental results show that the proposed WAS performs better than the other four filtering techniques in Recall and recision indices. In this paper, based on a case study in Huawei, we propose a Category Selection Based Adaptive Random Testing (CSBART). CSBART is a Linear-Order Algorithm for Adaptive Random Testing (LART). CSBART uses two category selection methods. One is the Input Profile (IP) method. The idea of IP is to define a more frequent choice that occurs in a failed test as a failure-related choice. Ce.IP identifies categories that are closely related to test case execution failure by counting the frequency of choice occurrences associated with failure. The second method is a category selection method based on Mutual Information (MI). The greater the amount of information, the closer the correlation between the category and the success or failure of test cases. In this paper, CSBAR T was studied in Huawei's industrial environment and compared with Random Testing (RT), LART, clustering-based technology n per cluster sampling and adaptive sampling. The results show that CSBAR T performs better than other techniques in discovering test cases that reveal faults. This paper proposes a test case set optimization method based on attribute analysis based on WAS and CSBART. In this paper, a prototype framework of test case set optimization based on attribute analysis is proposed. The corresponding tools are implemented in Huawei's industrial research case and passed the review of professional testers.
【学位授予单位】：南京大学
【学位级别】：博士
【学位授予年份】：2017
【分类号】：TP311.53

【相似文献】