Achievements, Open Problems and Challenges for Search Based Software Testing

@article{Harman2015AchievementsOP,
  title={Achievements, Open Problems and Challenges for Search Based Software Testing},
  author={Mark Harman and Yue Jia and Yuanyuan Zhang},
  journal={2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)},
  year={2015},
  pages={1-12}
}
  • M. Harman, Yue Jia, Yuanyuan Zhang
  • Published 13 April 2015
  • Computer Science
  • 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)
Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda, focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of… 

Figures from this paper

Combinatorial testing: using blocking to assign test cases for validating complex software systems

TLDR
This work extends software testing methodology by introducing the concept of blocking factors for a test suite, and provides an example, using an open source software library, to illustrate its extension.

An extensive evaluation of search-based software testing: a review

TLDR
The main objective of this survey is to find the main topics and trends in this emerging field of search-based software testing by examining the methods and the literature of software testing.

Using Defect Prediction to Improve the Bug Detection Capability of Search-Based Software Testing

TLDR
It is argued that SBST needs to be focused to search for test cases in defective areas rather in non-defective areas of the code in order to maximise the likelihood of discovering the bugs.

OCELOT: A Search-Based Test-Data Generation Tool for C

TLDR
It is shown that Ocelot achieves a higher coverage compared to a competitive tool in 81% of the cases and allows practitioners to write skeletons of test cases for their programs and researchers to easily implement and experiment new approaches for automatic test-data generation.

Towards Efficient Data-flow Test Data Generation

TLDR
A hybrid testing framework is introduced that systematically cast DFT as reachability checking in software model checking to complement SE, yielding practical DFT that combines the two techniques' strengths.

Improved evolutionary generation of test data for multiple paths in search-based software testing

TLDR
This paper proposes an improved grouping strategy of target paths to balance the load of each calculation resource and can accelerate the convergence of search process and improve the efficiency of search-based software testing.

Using Defect Prediction to Improve the Bug Detection Capability of Search-Based Software Testing

  • A. Perera
  • Computer Science
    2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)
  • 2020
TLDR
This thesis argues that SBST needs to be focused to search for test cases in defective areas rather in non-defective areas of the code in order to maximise the likelihood of discovering the bugs.

Cost and Effectiveness of Search-Based Techniques for Model-Based Testing: An Empirical Analysis

TLDR
Local SBTs are more appropriate to generate test data when the type of the constraints is simple and possible enhancement using the model-driven approach to detect the infeasible paths and SBTs to achieve optimal success rate is indicated.

The Maturation of Search-Based Software Testing: Successes and Challenges

  • Myra B. Cohen
  • Computer Science
    2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST)
  • 2019
TLDR
Some open challenges that remain for SBST are discussed including the need for new approaches to system testing, automated oracle generation, incorporating humans into the search process, and leveraging learning through hyper-heuristic search.
...

References

SHOWING 1-10 OF 136 REFERENCES

Current challenges in automatic software repair

TLDR
Current work on evolutionary computation approaches is reviewed, focusing on GenProg, which uses genetic programming to evolve a patch to a particular bug, and important open research challenges are outlined that should guide future research in the area.

Search Based Software Engineering

TLDR
This keynote address surveys recent success and momentum in the subfield of automated program repair and argues for the importance of human studies in automated software engineering and challenges the SBSE community to facilitate reproducible research and scientific advancement through benchmark creation.

Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?

TLDR
An empirical study applying a concolic tool, CUTE, and a search based tool, AUSTIN, to the source code of four large open source applications shows that both tools can only obtain at best a modest level of code coverage.

A search-based automated test-data generation framework for safety-critical systems

TLDR
The paper presents four applications of the framework - specification falsification testing, structural testing, exception condition testing and worst-case execution time testing to show that the framework offers useful support in the development safety-critical software systems.

Efficiency and early fault detection with lower and higher strength combinatorial interaction testing

TLDR
It is shown that the highest strength CIT can be achieved in 5.65 minutes on average and that higher strength suites find more faults, while prioritisations using lower strengths are no worse at achieving early fault revelation.

AUSTIN: An open source tool for search based software testing of C programs

Dynamic adaptive Search Based Software Engineering needs fast approximate metrics (keynote)

TLDR
This paper overviews recent results on SBSE for experimental metric validation and discusses the open problem of fast approximate surrogate metrics for dynamic adaptive SBSe.

EvoSuite: automatic test suite generation for object-oriented software

TLDR
EvoSuite is presented, a tool that automatically generates test cases with assertions for classes written in Java code that applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion.

Search-based software engineering: Trends, techniques and applications

TLDR
The work identifies research trends and relationships between the techniques applied and the applications to which they have been applied and highlights gaps in the literature and avenues for further research.

Evacon: a framework for integrating evolutionary and concolic testing for object-oriented programs

TLDR
This work proposes a novel framework called Evacon that integrates evolutionary testing ( used to search for desirable method sequences) and concolic testing (used to generate desirable method arguments) and shows that the tests generated using this framework can achieve higher branch coverage than evolutionary testing or concolics alone.
...