Assessing Test Adequacy for Black-Box Systems without Specifications

@inproceedings{Walkinshaw2011AssessingTA,
  title={Assessing Test Adequacy for Black-Box Systems without Specifications},
  author={Neil Walkinshaw},
  booktitle={ICTSS},
  year={2011}
}
Testing a black-box system without recourse to a specification is difficult, because there is no basis for estimating how many tests will be required, or to assess how complete a given test set is. Several researchers have noted that there is a duality between these testing problems and the problem of inductive inference (learning a model of a hidden system from a given set of examples). It is impossible to tell how many examples will be required to infer an accurate model, and there is no… 
Behaviourally Adequate Software Testing
  • G. Fraser, Neil Walkinshaw
  • Computer Science
    2012 IEEE Fifth International Conference on Software Testing, Verification and Validation
  • 2012
TLDR
A new test generation technique that is founded on behavioural adequacy is presented, which combines a model evaluation framework from the domain of statistical learning theory with search-based white-box test generation strategies.
Assessing and generating test sets in terms of behavioural adequacy
TLDR
The BESTEST approach enables the use of machine learning algorithms to augment standard syntactic testing approaches and shows how search‐based testing techniques can be applied to generate test sets with respect to this criterion.
Black-Box Test Generation from Inferred Models
TLDR
This paper provides a proof-of-concept inference-driven testing framework, along with evidence of the efficacy of its test sets on three programs, and presents an openly available framework to facilitate experimentation in this area.
Improving software model inference by combining state merging and Markov models
TLDR
This thesis proposed EDSM-Markov a passive inference technique that aimed to improve the existing LTS models in the absence of negative traces and to prevent the over-generalization problem.
Model-Based Testing and Model Inference
TLDR
The best known approaches involve the use of models (such as UML interaction diagrams or state machines) as the basis for selecting test inputs that seek to explore the core functionality of the system.
An Insight Into Test Case Optimization: Ideas and Trends With Future Perspectives
TLDR
A review of optimization techniques used in domains, test case generation, selection, minimization, and prioritization of testing, has been presented and the state-of-the-art issues in software testing have been addressed.
Machine Learning for Dynamic Software Analysis: Potentials and Limits
TLDR
It is argued that the optimal choice of an ML method for a particular application should be guided by the type of models one seeks to infer, and the open challenges for reaching the full potential of ML for software engineering are discussed.
PAC Learning-Based Verification and Model Synthesis
We introduce a novel technique for verification and model synthesis of sequential programs. Our technique is based on learning an approximate regular model of the set of feasible paths in a program,
Active Automata Learning in Practice - An Annotated Bibliography of the Years 2011 to 2016
TLDR
The progress that has been made over the past five years is reviewed, the status of active automata learning techniques with respect to applications in the field of software engineering is assessed, and an updated agenda for future research is presented.
An Efficient Model Inference Algorithm for Learning-based Testing of Reactive Systems
Learning-based testing (LBT) is an emerging methodology to automate iterative black-box requirements testing of software systems. The methodology involves combining model inference with model check
...
1
2
...

References

SHOWING 1-10 OF 39 REFERENCES
The Practical Assessment of Test Sets with Inductive Inference Techniques
TLDR
This paper shows how the Probably Approximately Correct (PAC) framework, a well-established approach in the field of inductive inference, can be applied to inductive testing techniques, facilitating a more pragmatic assessment of these techniques by allowing for a degree of error.
Approximate Testing and Its Relationship to Learning
  • K. Romanik
  • Computer Science
    Theor. Comput. Sci.
  • 1997
Assessing Test Data Adequacy through Program Inference
TLDR
As a basis for the development of usable notions of test data adequacy, an abstract definition is proposed and examined, and approximations to this definition are considered.
Strengthening Inferred Specifications using Search Based Testing
  • K. Ghani, John A. Clark
  • Computer Science
    2008 IEEE International Conference on Software Testing Verification and Validation Workshop
  • 2008
TLDR
This paper shows how search based testing techniques can challenge and identify erroneous elements of such inferred specifications, which leads to a much tighter (accurate) inferred specifications.
Increasing Functional Coverage by Inductive Testing: A Case Study
TLDR
This paper presents a flexible implementation of the inductive testing technique, and demonstrates its application with case-study that applies it to the Linux TCP stack implementation.
A Formal Interpretation of Software Testing as Inductive Inference
  • Hong Zhu
  • Computer Science
    Softw. Test. Verification Reliab.
  • 1996
TLDR
This paper investigates the foundation of software testing by interpreting the axioms of test adequacy criteria as properties of inductive inference, and results relating adequate testing to software correctness and reliability are yielded.
Testing by means of inductive program learning
TLDR
The technique for generating an adequate test set is based on the inductive learning of programs from finite sets of input-output examples and is made possible by a program induction procedure which has evolved from recent research in mnchine Iei]rnlng and inductive logic programming.
Principles and methods of testing finite state machines-a survey
TLDR
The fundamental problems in testing finite state machines and techniques for solving these problems are reviewed, tracing progress in the area from its inception to the present and the stare of the art is traced.
An automated framework for software test oracle
A Recursion Theoretic Approach to Program Testing
TLDR
These relationships between program testing and inductive inference are investigated and general recursion theoretic properties of testable sets of functions are exposed.
...
1
2
3
4
...