An Empirical Study on the Use of Defect Prediction for Test Case Prioritization

D. Paterson, J. Campos, R. Abreu, G. M. Kapfhammer, G. Fraser, P. McMinn
International Conference on Software Testing, Verification and Validation (ICST), 2019

Abstract

Test case prioritization has been extensively researched as a means for reducing the time taken to discover regressions in software. While many different strategies have been developed and evaluated, prior experiments have shown them to not be effective at prioritizing test suites to find real faults. This paper presents a test case prioritization strategy based on defect prediction, a technique that analyzes code features – such as the number of revisions and authors — to estimate the likelihood that any given Java class will contain a bug. Intuitively, if defect prediction can accurately predict the class that is most likely to be buggy, a tool can prioritize tests to rapidly detect the defects in that class. We investigated how to configure a defect prediction tool, called Schwa, to maximize the likelihood of an accurate prediction, surfacing the link between perfect defect prediction and test case prioritization effectiveness. Using 6 real-world Java programs containing 395 real faults, we conducted an empirical evaluation comparing this paper’s strategy, called G-clef, against eight existing test case prioritization strategies. The experiments reveal that using defect prediction to prioritize test cases reduces the number of test cases required to find a fault by on average 9.48% when compared with existing coverage-based strategies, and 10.5% when compared with existing history-based strategies.

Bibtex

@article{Paterson:ICST:2019,
  author = {Paterson, David, and Campos, Jos{\'e} and Abreu, Rui and M. Kapfhammer, Gregory and Fraser, Gordon and McMinn, Phil},
  title = {Parallel Many-Objective Search for Unit Tests},
  booktitle = {Proceedings of the 2019 IEEE 12th International Conference on Software Testing, Verification and Validation},
  series = {ICST '19},
  year = {2019},
  isbn = {},
  pages = {},
  numpages = {12},
  url = {},
  doi = {},
  acmid = {},
  publisher = {IEEE Computer Society},
  address = {Washington, DC, USA},
}