Experimental study to compare the use of metamorphic testing and assertion checking

Abstract

A test oracle in software testing is a mechanism for checking whether the program under test behaves correctly for any execution. In some practical situations, oracles can be unavailable or too expensive to apply. Metamorphic testing (MT) was proposed to alleviate this problem so that software can be delivered under the time-to-market pressure. However, the effectiveness of MT has not been studied adequately. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT. The fault detection capability and time cost of MT are compared with the standard assertion checking method. The results show that MT has potentials to detect more faults than the assertion checking method. The experimental results also show a trade-off between the two testing methods: MT can be less efficient but more effective, and can be defined at a coarser level of granularity than the assertion checking method. © by Institute of Software, the Chinese Academy of Sciences. All rights reserved.

在軟件測試中,測試預言是一種用于檢查程序在測試中是否正常運行的機制.然而在某些實際情況下,還無法制定測試預言或者難以有效地應用測試預言.針對此類測試預言問題,蛻變測試于近年應運而生,但蛻變測試的效率問題還沒有被充分地加以研究.作者用控制實驗的方法研究了使用蛻變測試的成本及效率,進而將蛻變測試和常用的斷言檢查兩種方法的錯誤檢測率和時間成本進行了比較和分析.實驗結果表明,相比于斷言檢查方法,蛻變測試具有檢測到更多錯誤的潛力.通過分析蛻變測試的效率和性能,與斷言測試相比,蛻變測試的錯誤檢測率更高效而效率有待提高,可適用于較為粗粒度的測試需求.