RERS 2016: Parallel and Sequential Benchmarks with Focus on LTL Verification

@inproceedings{Geske2016RERS2P,
  title={RERS 2016: Parallel and Sequential Benchmarks with Focus on LTL Verification},
  author={Maren Geske and Marc Jasper and Bernhard Steffen and Falk Howar and Markus Schordan and Jaco van de Pol},
  booktitle={ISoLA},
  year={2016}
}
The 5th challenge of Rigorous Examination of Reactive Systems (RERS 2016) once again provided generated and tailored benchmarks suited for comparing the effectiveness of automatic software verifiers. RERS is the only software verification challenge that features problems with linear temporal logic (LTL) properties in larger sizes that are available in different programming languages. This paper describes the revised rules and the refined profile of the challenge, which lowers the entry hurdle… 
RERS 2018: CTL, LTL, and Reachability
TLDR
Modifications compared to the previous iteration include an enhanced generation of sequential benchmarks, an improved automation of the construction of parallel benchmarks, a redesigned penalty for wrong answers, and the addition of CTL properties.
RERS 2019: Combining Synthesis with Real-World Models
TLDR
This paper highlights the newly added industrial tracks and the changes in response to the discussions at and results of the last RERS Challenge in Cyprus, and improves the benchmark generation infrastructure for parallel programs towards a full automation.
The RERS challenge: towards controllable and scalable benchmark synthesis
TLDR
A way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors is proposed, which gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing.
Property-Preserving Generation of Tailored Benchmark Petri Nets
TLDR
This paper shows how to systematically construct arbitrarily complex Petri Nets with guaranteed safety properties using a top-down parallel decomposition based on lightweight assumption commitment specifications.
The RERS 2017 challenge and workshop (invited paper)
TLDR
The first half of the RERS workshop focused on the 2017 benchmark profiles, the evaluation of the received contributions, and short presentations of each participating team, while the second half comprised discussions about attractive problem scenarios for future benchmarks.
Generating Hard Benchmark Problems for Weak Bisimulation
TLDR
A method to automatically generate arbitrarily complex benchmark problems for bisimulation checking is proposed, a variant of an incremental generation approach for model checking benchmarks where given benchmark scenarios of controllable size are expanded to arbitrarilycomplex benchmark problems.
Synthesizing Subtle Bugs with Known Witnesses
This paper presents a new technique for the generation of verification benchmarks that are automatically guaranteed to be hard, or as we say, to contain subtle bugs/property violations: (i)
Ample Set Partial Order Reduction for Actions
TLDR
This work proposes a native action-based POR, which preserves a class of properties the authors call blank-invariant, and implemented a simple LTS model checker that uses it, and applies it to the RERS 2017 benchmark suite, comparing its performace with that of Spin on the translated models.
The physics of software tools: SWOT analysis and vision
  • B. Steffen
  • Computer Science
    International Journal on Software Tools for Technology Transfer
  • 2016
TLDR
This paper analyzes the situation of today’s software tools from a global perspective in terms of a SWOT (Strength, Weaknesses, Opportunities, Treats) analysis, identifies challenges, and establishes a global vision for overcoming current weaknesses.
Synchronous or Alternating? - LTL Black-Box Checking of Mealy Machines by Combining the LearnLib and LTSmin
TLDR
This work defines and study Mealy-robust LTL properties, which are insensitive to which of these interpretations of LTL is chosen, and investigates the synchronous semantics and the alternating semantics for Mealy machines.
...
...

References

SHOWING 1-10 OF 26 REFERENCES
Rigorous examination of reactive systems
TLDR
The STTT special section on RERS describes the results of the evaluations and the different analysis techniques that were used in the RERS challenges 2012 and 2013.
Status Report on Software Verification - (Competition Summary SV-COMP 2014)
TLDR
The report represents the state of the art in automatic software verification, in terms of effectiveness and efficiency, at the 3rd International Competition on Software Verification (SV-COMP 2014).
Property-driven benchmark generation: synthesizing programs of realistic structure
We present a systematic approach to the automatic generation of platform-independent benchmarks of realistic structure and tailored complexity for evaluating verification tools for reactive systems.
VerifyThis 2015 A program verification competition
TLDR
The VerifyThis 2015 event, the challenges that were posed during the competition, and a high-level overview of the solutions to these challenges are provided.
Reliable and Reproducible Competition Results with BenchExec and Witnesses (Report on SV-COMP 2016)
TLDR
This report presents the results of the competition and includes a special section that describes how SV-COMP ensures that the experiments are reliably executed, precisely measured, and organized such that the results can be reproduced later.
VerifyThis 2012
TLDR
An overview of the VerifyThis competition series, an account of related activities in the area, and an overview of solutions submitted to the organizers both during and after the 2012 competition are provided.
Model Checking
TLDR
Model checking is applied concurrently with system design, and particularly in its early stages when systems are modelled at a high level of abstraction, because the payoff of finding bugs at that stage is highest whereas the costs are low.
Model-Based Testing of Reactive Systems
TLDR
This chapter focuses on formal objects used to select test suites, so-called coverage criteria, and focuses on the fault detection ability of a test suite satisfying a given coverage criterion.
Software model checking: extracting verification models from source code †
TLDR
It is argued, though, that the method was originally developed to allow for a thorough testing of parts of the software of a new commercial telephone switch also has broad applicability to distributed software systems design in general.
The SPIN Model Checker - primer and reference manual
TLDR
This is the most comprehensive reference guide to SPIN, written by the principal designer of the tool, and gives detailed advice on methods for tackling the most complex software verification problems.
...
...