Programs, tests, and oracles: the foundations of testing revisited

@article{Staats2011ProgramsTA,
  title={Programs, tests, and oracles: the foundations of testing revisited},
  author={Matthew Staats and Michael W. Whalen and Mats Per Erik Heimdahl},
  journal={2011 33rd International Conference on Software Engineering (ICSE)},
  year={2011},
  pages={391-400}
}
In previous decades, researchers have explored the formal foundations of program testing. By exploring the foundations of testing largely separate from any specific method of testing, these researchers provided a general discussion of the testing process, including the goals, the underlying problems, and the limitations of testing. Unfortunately, a common, rigorous foundation has not been widely adopted in empirical software testing research, making it difficult to generalize and compare… 

Figures from this paper

Automated Test Oracles: State of the Art, Taxonomies, and Trends
The Oracle Problem in Software Testing: A Survey
TLDR
This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.
Oracle problem in software testing
TLDR
This thesis work introduces a technique for assessing and improving test oracles by reducing the incidence of both false positives and false negatives, which combines test case generation to revealfalse positives and mutation testing to reveal false negatives.
Techniques for testing scientific programs without an oracle
  • Upulee Kanewala, J. Bieman
  • Computer Science
    2013 5th International Workshop on Software Engineering for Computational Science and Engineering (SE-CSE)
  • 2013
TLDR
This paper examines three techniques that are used to test programs without oracles: metamorphic testing, Run-time Assertions and developing test oracles using machine learning, and there is potential to increase the level of automation of these techniques, thereby reducing the required level of domain knowledge.
Oracle-Centric Test Case Prioritization
TLDR
This work proposes a technique for prioritizing test cases that explicitly takes into account the impact of test oracles on the effectiveness of testing, and demonstrates improvements in rate of fault detection relative to both random and structural coverage based prioritization techniques when applied to faulty versions of three synchronous reactive systems.
Software testing: a research travelogue (2000–2014)
TLDR
The goal of this paper is to provide an accounting of some of the most successful research performed in software testing since the year 2000, and to present what appear to be the most significant challenges and opportunities in this area.
A Comprehensive Survey of Trends in Oracles for Software Testing
TLDR
This paper provides a comprehensive survey of current approaches to the oracle problem and an analysis of trends in this important area of software testing research and practice.
Better testing through oracle selection: (NIER track)
TLDR
It is argued that selecting the test oracle and test inputs together to complement one another may yield improvements testing effectiveness and is illustrated using an example.
Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing
TLDR
Experiments over four industrial examples demonstrate that the proposed method may be a cost-effective approach for producing small, effective oracle data, with fault finding improvements over current industrial best practice of up to 145.8% observed.
Oracle assessment, improvement and placement
TLDR
A technique for assessing and improving test oracles is proposed, which necessarily places the human tester in the loop and is based on reducing the incidence of both false positives and false negatives and results in an increase in the mutual information between the actual and perfect oracles.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 50 REFERENCES
Foundations of software testing: dependability theory
TLDR
It is argued that testing's primary goal should be to measure the dependability of tested software, and a plausible theory of dependability is needed to suggest and prove results about what test methods should be used, and under what circumstances.
A Critical Analysis of Empirical Research in Software Testing
  • L. Briand
  • Computer Science
    First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007)
  • 2007
TLDR
This paper presents a critical analysis of empirical research in software testing and will attempt to highlight and clarify the issues above in a structured and practical manner.
On Testing Non-Testable Programs
TLDR
'The belief that the tester is routinely able to determine whether or not the test output is correct is the oracle assumption.
Specification-based test oracles for reactive systems
TLDR
It is argued that test oracles should be derived from specifications in conjunction with testing criteria, represented in a common form, and their use made integral to the test- ing process.
What test oracle should I use for effective GUI testing?
TLDR
It is shown that faults are detected early in the testing process when using detailed oracle information and complex oracle procedures, although at a higher cost per test case, and that employing expensive oracles results in detecting a large number of faults using relatively smaller number of test cases.
Better testing through oracle selection: (NIER track)
TLDR
It is argued that selecting the test oracle and test inputs together to complement one another may yield improvements testing effectiveness and is illustrated using an example.
Software testing based on formal specifications: a theory and a tool
TLDR
The paper presents the actual procedures used to mechanically produce such test sets, using Horn clause logic, embedded in an interactive system which, given some general hypotheses schemes and an algebraic specification, produces a test set and the corresponding hypotheses.
Assessing and improving state-based class testing: a series of experiments
TLDR
It is shown that in most cases state-based techniques are not likely to be sufficient by themselves to catch most of the faults present in the code, and though useful, they need to be complemented with black-box, functional testing.
Software Testing Techniques
TLDR
The aim is to compare different testing techniques and what efforts it takes to find genuine failures on particular level of test, e.g. unit/component level, and on other test levels, such as integrated and system test level.
Test data adequacy measurement
TLDR
The abstract properties of adequacy criteria are investigated in a formal axiomatic style, independent of the syntax and semantics of specifications and programs, and the dependencies between the axioms are investigated.
...
1
2
3
4
5
...