Using mutation testing to measure behavioural test diversity

  title={Using mutation testing to measure behavioural test diversity},
  author={Francisco Gomes de Oliveira Neto and Felix Dobslaw and Robert Feldt},
  journal={2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)},
  • F. G. O. Neto, Felix Dobslaw, R. Feldt
  • Published 1 October 2020
  • Computer Science
  • 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Diversity has been proposed as a key criterion to improve testing effectiveness and efficiency. It can be used to optimise large test repositories but also to visualise test maintenance issues and raise practitioners’ awareness about waste in test artefacts and processes. Even though these diversitybased testing techniques aim to exercise diverse behavior in the system under test (SUT), the diversity has mainly been measured on and between artefacts (e.g., inputs, outputs or test scripts). Here… 

Figures and Tables from this paper

Evaluating the Trade-offs of Diversity-Based Test Prioritization: An Experiment
The results show that SS increases test coverage for system-level tests, and the differences in failure detection rate of each diversity increase as more prioritised tests execute, including Jaccard’s Index, Levenshtein, Normalized Compression Distance, and Semantic Similarity.
Mutation Testing Techniques in Software Testing: A Review
The various schemes which are based on the mutation testing are reviewed in this paper and they are regarded as a dominant scheme to quantify the quality of test suite.
Equivalent mutant identification using hybrid wavelet convolutional rain optimization
A novel hybrid strategy known as the hybrid wavelet convolutional rain optimization (HWCRO) to classify the equivalent mutants present in the source codes accurately and exactly identifies the mutated code is proposed.


Visualizing Test Diversity to Support Test Optimisation
The key result is that test similarity maps, based on pair-wise diversity calculations, helped industrial practitioners identify issues with their test repositories and decide on actions to improve and it is concluded that the visualisation of diversity information can assist testers in their maintenance and optimisation activities.
How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults
There are large differences between the tools’ effectiveness and it is demonstrated that no tool is able to subsume the others and overall, PITRV achieves the best results, by finding 6% more faults than the other tools combined.
Achieving scalable model-based testing through test case diversity
A family of similarity-based test case selection techniques for test suites generated from state machines is introduced and a method to identify optimal tradeoffs between the number of test cases to run and fault detection is proposed.
Prioritizing test cases for resource constraint environments using historical test case performance data
The aim in this paper has been to prioritize test cases during software regression test, and a new equation is presented that considers historical effectiveness of the test cases in fault detection, each test case's execution history in regression test and finally the last priority assigned to the test case.
A Theoretical and Empirical Study of Diversity-Aware Mutation Adequacy Criterion
A novel, diversity-aware mutation adequacy criterion is proposed, which is fully satisfied when each of the considered mutants can be identified by the set of tests that kill it, thereby encouraging inclusion of more diverse range of tests.
A history-based test prioritization technique for regression testing in resource constrained environments
This work proposes a new technique that bases prioritization on historical execution data, and conducts an experiment to assess its effects on the long run performance of resource constrained regression testing.
Scalable Approaches for Test Suite Reduction
A family of novel very efficient approaches for similaritybased test suite reduction that apply algorithms borrowed from the big data domain together with smart heuristics for finding an evenly spread subset of test cases and yield a fault detection loss comparable to state-of-the-art techniques.
Mutation-based test-case prioritization in software evolution
  • Yiling LouDan HaoLu Zhang
  • Computer Science
    2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)
  • 2015
A novel test-case prioritization approach for software evolution is proposed, which first uses mutation faults on the difference between the early version and the latter version to simulate real faults occurred in software evolution, and then schedules the execution order of test cases based on their fault-detection capability, which is defined based on mutation faults.
Test Set Diameter: Quantifying the Diversity of Sets of Test Cases
This work proposes a new metric to measure the diversity of sets of tests: the test set diameter (TSDm), which extends earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets.
Evaluating Mutation Testing Alternatives: A Collateral Experiment
In this paper several second order mutation testing strategies are introduced, assessed and compared along with weak mutation against strong, suggesting that they both constitute viable alternatives for mutation as they establish considerable effort reductions without greatly affecting the test effectiveness.