File Download
  Links for fulltext
     (May Require Subscription)
Supplementary

Article: How well does test case prioritization integrate with statistical fault localization?

TitleHow well does test case prioritization integrate with statistical fault localization?
Authors
KeywordsAdaptive random testing
Continuous integration
Coverage
Software process integration
Statistical fault localization
Test case prioritization
Issue Date2012
PublisherElsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsof
Citation
Information And Software Technology, 2012, v. 54 n. 7, p. 739-758 How to Cite?
AbstractContext: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved.
Persistent Identifierhttp://hdl.handle.net/10722/147019
ISSN
2014 Impact Factor: 1.046
2013 SCImago Journal Rankings: 1.072
ISI Accession Number ID
Funding AgencyGrant Number
Beihang University
Natural Science Foundation of China61003027
City University of Hong Kong7002673
Research Grants Council of Hong Kong717308
Australian Research CouncilDP120104773
Funding Information:

This research is supported in part by a grant of the Basic Research Fund of Beihang University, a grant of the Natural Science Foundation of China (Project No. 61003027), a strategy research grant of City University of Hong Kong (Project No. 7002673), a grant of the General Research Fund of the Research Grants Council of Hong Kong (Project No. 717308), and a discovery grant of the Australian Research Council (Project No. DP120104773).

References

 

DC FieldValueLanguage
dc.contributor.authorJiang, Ben_HK
dc.contributor.authorZhang, Zen_HK
dc.contributor.authorChan, WKen_HK
dc.contributor.authorTse, THen_HK
dc.contributor.authorChen, TYen_HK
dc.date.accessioned2012-05-23T05:53:39Z-
dc.date.available2012-05-23T05:53:39Z-
dc.date.issued2012en_HK
dc.identifier.citationInformation And Software Technology, 2012, v. 54 n. 7, p. 739-758en_HK
dc.identifier.issn0950-5849en_HK
dc.identifier.urihttp://hdl.handle.net/10722/147019-
dc.description.abstractContext: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved.en_HK
dc.languageengen_US
dc.publisherElsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsofen_HK
dc.relation.ispartofInformation and Software Technologyen_HK
dc.rightsNOTICE: this is the author’s version of a work that was accepted for publication in Information and Software Technology. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Information and Software Technology, 2012, v. 54 n. 7, p. 739-758. DOI: 10.1016/j.infsof.2012.01.006-
dc.rightsCreative Commons: Attribution 3.0 Hong Kong License-
dc.subjectAdaptive random testingen_HK
dc.subjectContinuous integrationen_HK
dc.subjectCoverageen_HK
dc.subjectSoftware process integrationen_HK
dc.subjectStatistical fault localizationen_HK
dc.subjectTest case prioritizationen_HK
dc.titleHow well does test case prioritization integrate with statistical fault localization?en_HK
dc.typeArticleen_HK
dc.identifier.emailTse, TH: thtse@cs.hku.hken_HK
dc.identifier.authorityTse, TH=rp00546en_HK
dc.description.naturepostprint-
dc.identifier.doi10.1016/j.infsof.2012.01.006en_HK
dc.identifier.scopuseid_2-s2.0-84862821558en_HK
dc.identifier.hkuros199409en_US
dc.relation.referenceshttp://www.scopus.com/mlt/select.url?eid=2-s2.0-84862821558&selection=ref&src=s&origin=recordpageen_HK
dc.identifier.volume54en_HK
dc.identifier.issue7en_HK
dc.identifier.spage739en_HK
dc.identifier.epage758en_HK
dc.identifier.isiWOS:000304387600006-
dc.publisher.placeNetherlandsen_HK
dc.identifier.scopusauthoridJiang, B=35199818000en_HK
dc.identifier.scopusauthoridZhang, Z=36198974100en_HK
dc.identifier.scopusauthoridChan, WK=23967779900en_HK
dc.identifier.scopusauthoridTse, TH=54971059300en_HK
dc.identifier.scopusauthoridChen, TY=35247605500en_HK
dc.identifier.citeulike10367104-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats