File Download
 
Links for fulltext
(May Require Subscription)
 
Supplementary

Article: How well does test case prioritization integrate with statistical fault localization?
  • Basic View
  • Metadata View
  • XML View
TitleHow well does test case prioritization integrate with statistical fault localization?
 
AuthorsJiang, B5
Zhang, Z1
Chan, WK3
Tse, TH2
Chen, TY4
 
KeywordsAdaptive random testing
Continuous integration
Coverage
Software process integration
Statistical fault localization
Test case prioritization
 
Issue Date2012
 
PublisherElsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsof
 
CitationInformation And Software Technology, 2012, v. 54 n. 7, p. 739-758 [How to Cite?]
DOI: http://dx.doi.org/10.1016/j.infsof.2012.01.006
 
AbstractContext: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved.
 
ISSN0950-5849
2013 Impact Factor: 1.328
2013 SCImago Journal Rankings: 1.072
 
DOIhttp://dx.doi.org/10.1016/j.infsof.2012.01.006
 
ISI Accession Number IDWOS:000304387600006
Funding AgencyGrant Number
Beihang University
Natural Science Foundation of China61003027
City University of Hong Kong7002673
Research Grants Council of Hong Kong717308
Australian Research CouncilDP120104773
Funding Information:

This research is supported in part by a grant of the Basic Research Fund of Beihang University, a grant of the Natural Science Foundation of China (Project No. 61003027), a strategy research grant of City University of Hong Kong (Project No. 7002673), a grant of the General Research Fund of the Research Grants Council of Hong Kong (Project No. 717308), and a discovery grant of the Australian Research Council (Project No. DP120104773).

 
ReferencesReferences in Scopus
 
DC FieldValue
dc.contributor.authorJiang, B
 
dc.contributor.authorZhang, Z
 
dc.contributor.authorChan, WK
 
dc.contributor.authorTse, TH
 
dc.contributor.authorChen, TY
 
dc.date.accessioned2012-05-23T05:53:39Z
 
dc.date.available2012-05-23T05:53:39Z
 
dc.date.issued2012
 
dc.description.abstractContext: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. © 2012 Elsevier B.V. All rights reserved.
 
dc.description.naturepostprint
 
dc.identifier.citationInformation And Software Technology, 2012, v. 54 n. 7, p. 739-758 [How to Cite?]
DOI: http://dx.doi.org/10.1016/j.infsof.2012.01.006
 
dc.identifier.citeulike10367104
 
dc.identifier.doihttp://dx.doi.org/10.1016/j.infsof.2012.01.006
 
dc.identifier.epage758
 
dc.identifier.hkuros199409
 
dc.identifier.isiWOS:000304387600006
Funding AgencyGrant Number
Beihang University
Natural Science Foundation of China61003027
City University of Hong Kong7002673
Research Grants Council of Hong Kong717308
Australian Research CouncilDP120104773
Funding Information:

This research is supported in part by a grant of the Basic Research Fund of Beihang University, a grant of the Natural Science Foundation of China (Project No. 61003027), a strategy research grant of City University of Hong Kong (Project No. 7002673), a grant of the General Research Fund of the Research Grants Council of Hong Kong (Project No. 717308), and a discovery grant of the Australian Research Council (Project No. DP120104773).

 
dc.identifier.issn0950-5849
2013 Impact Factor: 1.328
2013 SCImago Journal Rankings: 1.072
 
dc.identifier.issue7
 
dc.identifier.scopuseid_2-s2.0-84862821558
 
dc.identifier.spage739
 
dc.identifier.urihttp://hdl.handle.net/10722/147019
 
dc.identifier.volume54
 
dc.languageeng
 
dc.publisherElsevier BV. The Journal's web site is located at http://www.elsevier.com/locate/infsof
 
dc.publisher.placeNetherlands
 
dc.relation.ispartofInformation and Software Technology
 
dc.relation.referencesReferences in Scopus
 
dc.rightsNOTICE: this is the author’s version of a work that was accepted for publication in Information and Software Technology. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Information and Software Technology, 2012, v. 54 n. 7, p. 739-758. DOI: 10.1016/j.infsof.2012.01.006
 
dc.rightsCreative Commons: Attribution 3.0 Hong Kong License
 
dc.subjectAdaptive random testing
 
dc.subjectContinuous integration
 
dc.subjectCoverage
 
dc.subjectSoftware process integration
 
dc.subjectStatistical fault localization
 
dc.subjectTest case prioritization
 
dc.titleHow well does test case prioritization integrate with statistical fault localization?
 
dc.typeArticle
 
<?xml encoding="utf-8" version="1.0"?>
<item><contributor.author>Jiang, B</contributor.author>
<contributor.author>Zhang, Z</contributor.author>
<contributor.author>Chan, WK</contributor.author>
<contributor.author>Tse, TH</contributor.author>
<contributor.author>Chen, TY</contributor.author>
<date.accessioned>2012-05-23T05:53:39Z</date.accessioned>
<date.available>2012-05-23T05:53:39Z</date.available>
<date.issued>2012</date.issued>
<identifier.citation>Information And Software Technology, 2012, v. 54 n. 7, p. 739-758</identifier.citation>
<identifier.issn>0950-5849</identifier.issn>
<identifier.uri>http://hdl.handle.net/10722/147019</identifier.uri>
<description.abstract>Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging. &#169; 2012 Elsevier B.V. All rights reserved.</description.abstract>
<language>eng</language>
<publisher>Elsevier BV. The Journal&apos;s web site is located at http://www.elsevier.com/locate/infsof</publisher>
<relation.ispartof>Information and Software Technology</relation.ispartof>
<rights>NOTICE: this is the author&#8217;s version of a work that was accepted for publication in Information and Software Technology. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Information and Software Technology, 2012, v. 54 n. 7, p. 739-758. DOI: 10.1016/j.infsof.2012.01.006</rights>
<rights>Creative Commons: Attribution 3.0 Hong Kong License</rights>
<subject>Adaptive random testing</subject>
<subject>Continuous integration</subject>
<subject>Coverage</subject>
<subject>Software process integration</subject>
<subject>Statistical fault localization</subject>
<subject>Test case prioritization</subject>
<title>How well does test case prioritization integrate with statistical fault localization?</title>
<type>Article</type>
<description.nature>postprint</description.nature>
<identifier.doi>10.1016/j.infsof.2012.01.006</identifier.doi>
<identifier.scopus>eid_2-s2.0-84862821558</identifier.scopus>
<identifier.hkuros>199409</identifier.hkuros>
<relation.references>http://www.scopus.com/mlt/select.url?eid=2-s2.0-84862821558&amp;selection=ref&amp;src=s&amp;origin=recordpage</relation.references>
<identifier.volume>54</identifier.volume>
<identifier.issue>7</identifier.issue>
<identifier.spage>739</identifier.spage>
<identifier.epage>758</identifier.epage>
<identifier.isi>WOS:000304387600006</identifier.isi>
<publisher.place>Netherlands</publisher.place>
<identifier.citeulike>10367104</identifier.citeulike>
<bitstream.url>http://hub.hku.hk/bitstream/10722/147019/1/Content.pdf</bitstream.url>
</item>
Author Affiliations
  1. Institute of Software Chinese Academy of Sciences
  2. The University of Hong Kong
  3. City University of Hong Kong
  4. Swinburne University of Technology
  5. Beijing University of Aeronautics and Astronautics