Interaction-based test-suite minimization

@article{Blue2013InteractionbasedTM,
  title={Interaction-based test-suite minimization},
  author={Dale Blue and Itai Segall and Rachel Tzoref and Aviad Zlotnick},
  journal={2013 35th International Conference on Software Engineering (ICSE)},
  year={2013},
  pages={182-191}
}
Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. In this work, we propose to use Interaction-based Test-Suite Minimization (ITSM) as a complementary… 

Figures and Tables from this paper

Combining minimization and generation for combinatorial testing
TLDR
This work introduces Minimization Generation CTD, a combination of CTD with ITSM for addressing situations in which CTD is impractical, and ITSM is insufficient, and presents an efficient implementation which is based on binary decision diagrams.
Modify, enhance, select: co-evolution of combinatorial models and test plans
TLDR
This work provides five alternatives for co-evolving the test plan with the combinatorial model, considering tradeoffs between maximizing fine-grained reuse and minimizing total test plan size, all while meeting the required combinatorsial coverage.
A Search-Based Constraint Elicitation in Test Design
TLDR
This paper proposes a constraint elicitation process that helps combinatorial test design and conducts experiments on some test models, demonstrating that some extracted rules match constraints and others helps to define constraints.
A Systematic Review on Test Suite Reduction: Approaches, Experiment’s Quality Evaluation, and Guidelines
TLDR
An initial set of recommendations is provided that are useful for performing well-designed experiments in the field of TSR and a number of future research directions based on current trends in this field of research are provided.
Towards Automatic Constraint Elicitation in Test Design: Preliminary Evaluation Based on Collective Intelligence
  • Hiroyuki Nakagawa, T. Tsuchiya
  • Computer Science
    2015 30th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW)
  • 2015
TLDR
This paper proposes a collective intelligence approach to determining which value combinations are potential constraints in combinatorial test design and proposes an automated part of the constraint elicitation process.
Scalable Approaches for Test Suite Reduction
TLDR
A family of novel very efficient approaches for similaritybased test suite reduction that apply algorithms borrowed from the big data domain together with smart heuristics for finding an evenly spread subset of test cases and yield a fault detection loss comparable to state-of-the-art techniques.
How Do Assertions Impact Coverage-Based Test-Suite Reduction?
TLDR
This paper finds that assertions (i.e., a typical type of test oracles) are significantly correlated with coverage-based test-suite reduction, and proposes an assertion-aware test- Suite reduction technique which outperforms traditional test-Suite reduction in terms of cost-effectiveness.
Model-Based Test-Suite Minimization using Modified Condition/Decision Coverage (MC/DC)
TLDR
A new approach for test-suite minimization is presented in this work, using dissimilarity matrix, which can be well fitted with MC/DC, which is an effective verification technique, and helps to uncover safety faults.
Probability Model-Based Test Suite Reduction
TLDR
This paper model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability models for the software's old and new versions, respectively, and implement a theorem combining set operations and probability models.
...
1
2
3
4
...

References

SHOWING 1-10 OF 15 REFERENCES
Using binary decision diagrams for combinatorial test design
TLDR
This work presents a new and highly effective BDD-based approach for solving CTD, which finds a set of tests that satisfies t-wise coverage by subset selection and applies it to real-life testing problems of varying complexity, and shows its superior performance.
Model-based testing in practice
TLDR
Four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications and their insights into what works in practice and thoughts about obstacles to transferring this technology into testing organizations.
Regression testing minimization, selection and prioritization: a survey
TLDR
This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research.
Test set size minimization and fault detection effectiveness: a case study in a space application
An important question in software testing is whether it is reasonable to apply coverage based criteria as a filter to reduce the size of a test set. An empirical study was conducted using a test set
Empirical studies of test‐suite reduction
TLDR
Results of these studies reveal that the fault‐detection capabilities of test suites can be severely compromised by test‐suite reduction.
Testing across configurations: implications for combinatorial testing
TLDR
A case study that investigates the effects of changing configurations on two types of test suites and shows that test coverage and fault detection effectiveness do not vary much across configurations for entire test suites; however, for individual test cases and certain types of faults, configurations matter.
An empirical study of the effects of minimization on the fault detection capabilities of test suites
TLDR
An experiment in which the costs and benefits of minimizing test suites of various sizes for several programs are compared reveals that the fault detection capabilities of test suites can be severely compromised by minimization.
An evaluation of combination strategies for test case selection
TLDR
The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.
Effect of test set minimization on fault detection effectiveness
TLDR
As the size of a test set is reduced, while the code coverage is kept constant, there is little or no reduction in the fault detection effectiveness of the new test set so generated.
Effect of Test Set Minimization on Fault Detection Effectiveness
TLDR
An empirical study using the block and all-uses criteria as the coverage measures to address the issue of whether the size of T or the coverage of T on P determines the fault detection effectiveness of T for P.
...
1
2
...