Finding Bugs with Specification-Based Testing is Easy!

  title={Finding Bugs with Specification-Based Testing is Easy!},
  author={Janice Chin and David Pearce},
  journal={Art Sci. Eng. Program.},
Automated specification-based testing has a long history with several notable tools having emerged. For example, QuickCheck for Haskell focuses on testing against user-provided properties. Others, such as JMLUnit, use specifications in the form of preand post-conditions to drive testing. An interesting (and under-explored) question is how effective this approach is at finding bugs in practice. In general, one would assume automated testing is less effective at bug finding than static… Expand

Figures and Tables from this paper


ArbitCheck: A Highly Automated Property-Based Testing Tool for Java
A new property-based testing tool, called ArbitCheck, which automates object generation by adapting the feedback-directed random test generation technique, which exhaustively generates possible values of user-defined types and tests properties with them, so that it can reveal faults that are hard to find with either manually written tests or existing property- based testing tools. Expand
Your Proof Fails? Testing Helps to Find the Reason
This work proposes a complete methodology where test generation helps to identify the reason of a proof failure and to exhibit a counterexample clearly illustrating the issue, and describes how to transform a formally specified C program into C code suitable for testing, and illustrates the benefits on comprehensive examples. Expand
An Empirical Evaluation of Assertions as Oracles
  • Kavir Shrestha, M. Rutherford
  • Computer Science
  • 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation
  • 2011
Simple, code-based metrics can be used to predict which methods are amenable to the use of assertion-based oracles with a high degree of accuracy by analyzing the data using data mining techniques. Expand
A Simple and Practical Approach to Unit Testing: The JML and JUnit Way
A new approach that makes writing unit tests easier is presented, which uses a formal specification language's runtime assertion checker to decide whether methods are working correctly, thus automating the writing of unit test oracles. Expand
JCrasher: an automatic robustness tester for Java
JCrasher attempts to detect bugs by causing the program under test to ‘crash’, that is, to throw an undeclared runtime exception, to test the behavior of public methods under random data. Expand
Reconciling Manual and Automated Testing: The AutoTest Experience
AutoTest is a testing tool that provides a "best of both worlds" strategy: it integrates developers' test cases into an automated process of systematic contract-driven testing, and to treat the two types of tests in a unified fashion. Expand
FairFuzz: A Targeted Mutation Strategy for Increasing Greybox Fuzz Testing Coverage
This work proposes a two-pronged approach to increase the coverage achieved by American Fuzzy Lop, including a novel mutation mask creation algorithm, which allows mutations to be biased towards producing inputs hitting a given rare branch. Expand
What good are strong specifications?
A methodology that extends Design by Contract to write strong specifications of functional properties in the form of preconditions, postconditions, and invariants is introduced, aimed at being palatable to developers who are not fluent in formal techniques but are comfortable with writing simple specifications. Expand
TestEra: Specification-Based Testing of Java Programs Using SAT
TestEra requires as input a Java method, a formal specification of the pre- and post-conditions of that method, and a bound that limits the size of the test cases to be generated, and automatically generates all nonisomorphic test inputs up to the given bound. Expand
Designing a verifying compiler: Lessons learned from developing Whiley
A technical overview of the verifying compiler is presented and the numerous design decisions made are documented, which hope to provide a useful resource for those building verifying compilers. Expand