Article: How well does test case prioritization integrate with statistical fault localization?
| Title | How well does test case prioritization integrate with statistical fault localization? |
|---|---|
| Authors | Jiang, B5 Zhang, Z3 Chan, WK2 Tse, TH1 Chen, TY4 |
| Keywords | Adaptive random testing Continuous integration Coverage Software process integration Statistical fault localization Test case prioritization |
| Issue Date | 2012 |
| Publisher | Elsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsof |
| Citation | Information And Software Technology, 2012, v. 54 n. 7, p. 739-758 [How to Cite?] DOI: http://dx.doi.org/10.1016/j.infsof.2012.01.006 |
| Abstract | Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved. |
| ISSN | 0950-5849 2011 Impact Factor: 1.25 2011 SCImago Journal Rankings: 0.055 |
| DOI | http://dx.doi.org/10.1016/j.infsof.2012.01.006 |
| References | References in Scopus |
| dc.contributor.author | Jiang, B | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dc.contributor.author | Zhang, Z | ||||||||||||
| dc.contributor.author | Chan, WK | ||||||||||||
| dc.contributor.author | Tse, TH | ||||||||||||
| dc.contributor.author | Chen, TY | ||||||||||||
| dc.date.accessioned | 2012-05-23T05:53:39Z | ||||||||||||
| dc.date.available | 2012-05-23T05:53:39Z | ||||||||||||
| dc.date.issued | 2012 | ||||||||||||
| dc.description.abstract | Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved. | ||||||||||||
| dc.description.nature | postprint | ||||||||||||
| dc.identifier.citation | Information And Software Technology, 2012, v. 54 n. 7, p. 739-758 [How to Cite?] DOI: http://dx.doi.org/10.1016/j.infsof.2012.01.006 | ||||||||||||
| dc.identifier.citeulike | 10367104 | ||||||||||||
| dc.identifier.doi | http://dx.doi.org/10.1016/j.infsof.2012.01.006 | ||||||||||||
| dc.identifier.epage | 758 | ||||||||||||
| dc.identifier.hkuros | 199409 | ||||||||||||
| dc.identifier.isi | WOS:000304387600006
Funding Information: This research is supported in part by a grant of the Basic Research Fund of Beihang University, a grant of the Natural Science Foundation of China (Project No. 61003027), a strategy research grant of City University of Hong Kong (Project No. 7002673), a grant of the General Research Fund of the Research Grants Council of Hong Kong (Project No. 717308), and a discovery grant of the Australian Research Council (Project No. DP120104773). | ||||||||||||
| dc.identifier.issn | 0950-5849 2011 Impact Factor: 1.25 2011 SCImago Journal Rankings: 0.055 | ||||||||||||
| dc.identifier.issue | 7 | ||||||||||||
| dc.identifier.scopus | eid_2-s2.0-84862821558 | ||||||||||||
| dc.identifier.spage | 739 | ||||||||||||
| dc.identifier.uri | http://hdl.handle.net/10722/147019 | ||||||||||||
| dc.identifier.volume | 54 | ||||||||||||
| dc.language | eng | ||||||||||||
| dc.publisher | Elsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsof | ||||||||||||
| dc.publisher.place | Netherlands | ||||||||||||
| dc.relation.ispartof | Information and Software Technology | ||||||||||||
| dc.relation.references | References in Scopus | ||||||||||||
| dc.rights | NOTICE: this is the author’s version of a work that was accepted for publication in Information and Software Technology. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Information and Software Technology, 2012, v. 54 n. 7, p. 739-758. DOI: 10.1016/j.infsof.2012.01.006 | ||||||||||||
| dc.rights | Creative Commons: Attribution 3.0 Hong Kong License | ||||||||||||
| dc.subject | Adaptive random testing | ||||||||||||
| dc.subject | Continuous integration | ||||||||||||
| dc.subject | Coverage | ||||||||||||
| dc.subject | Software process integration | ||||||||||||
| dc.subject | Statistical fault localization | ||||||||||||
| dc.subject | Test case prioritization | ||||||||||||
| dc.title | How well does test case prioritization integrate with statistical fault localization? | ||||||||||||
| dc.type | Article |
Author Affiliations
- The University of Hong Kong
- City University of Hong Kong
- Chinese Academy of Sciences
- Swinburne University of Technology
- Beijing University of Aeronautics and Astronautics