Even the very wise cannot see all ends: Many facets of the test oracle problem

Abstract

In software testing, we assume the existence of a test oracle, which verifies whether the equation “actual result = expected outcome” is satisfied for every test case. However, the verification process is often impossible or too difficult in real life, in which case we say that there is a test oracle problem. What exactly is the problem? We usually mean that the expected outcome is not available. Metamorphic testing is a popular approach to alleviate this setback. We verify the consistency between multiple test results and their expected relations. Unfortunately, the oracle problem does not end here. In pervasive computing, for example, the actual results may be too short-lived to be observed. In component-based software, the actual results may be encapsulated and hidden. Hence, problems on the left hand side of the verification equation should not be ignored. Furthermore, in object-oriented computing, the equal sign between the actual result and the expected outcome is a difficult issue because determining the observation equivalence of two objects involves infinite number of checks for every test case. In this talk, we revisit the oracle problem and present research challenges and opportunities for all components of the verification equation “actual result = expected outcome”.