RERS 2016: Parallel and Sequential Benchmarks with Focus on LTL Verification

  title={RERS 2016: Parallel and Sequential Benchmarks with Focus on LTL Verification},
  author={Maren Geske and Marc Jasper and Bernhard Steffen and Falk Howar and Markus Schordan and Jaco van de Pol},
The 5th challenge of Rigorous Examination of Reactive Systems (RERS 2016) once again provided generated and tailored benchmarks suited for comparing the effectiveness of automatic software verifiers. RERS is the only software verification challenge that features problems with linear temporal logic (LTL) properties in larger sizes that are available in different programming languages. This paper describes the revised rules and the refined profile of the challenge, which lowers the entry hurdle… 
RERS 2018: CTL, LTL, and Reachability
Modifications compared to the previous iteration include an enhanced generation of sequential benchmarks, an improved automation of the construction of parallel benchmarks, a redesigned penalty for wrong answers, and the addition of CTL properties.
RERS 2019: Combining Synthesis with Real-World Models
This paper highlights the newly added industrial tracks and the changes in response to the discussions at and results of the last RERS Challenge in Cyprus, and improves the benchmark generation infrastructure for parallel programs towards a full automation.
The RERS challenge: towards controllable and scalable benchmark synthesis
A way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors is proposed, which gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing.
Property-Preserving Generation of Tailored Benchmark Petri Nets
This paper shows how to systematically construct arbitrarily complex Petri Nets with guaranteed safety properties using a top-down parallel decomposition based on lightweight assumption commitment specifications.
The RERS 2017 challenge and workshop (invited paper)
The first half of the RERS workshop focused on the 2017 benchmark profiles, the evaluation of the received contributions, and short presentations of each participating team, while the second half comprised discussions about attractive problem scenarios for future benchmarks.
Generating Hard Benchmark Problems for Weak Bisimulation
A method to automatically generate arbitrarily complex benchmark problems for bisimulation checking is proposed, a variant of an incremental generation approach for model checking benchmarks where given benchmark scenarios of controllable size are expanded to arbitrarilycomplex benchmark problems.
Synthesizing Subtle Bugs with Known Witnesses
This paper presents a new technique for the generation of verification benchmarks that are automatically guaranteed to be hard, or as we say, to contain subtle bugs/property violations: (i)
Ample Set Partial Order Reduction for Actions
This work proposes a native action-based POR, which preserves a class of properties the authors call blank-invariant, and implemented a simple LTS model checker that uses it, and applies it to the RERS 2017 benchmark suite, comparing its performace with that of Spin on the translated models.
The physics of software tools: SWOT analysis and vision
  • B. Steffen
  • Computer Science
    International Journal on Software Tools for Technology Transfer
  • 2016
This paper analyzes the situation of today’s software tools from a global perspective in terms of a SWOT (Strength, Weaknesses, Opportunities, Treats) analysis, identifies challenges, and establishes a global vision for overcoming current weaknesses.
Synchronous or Alternating? - LTL Black-Box Checking of Mealy Machines by Combining the LearnLib and LTSmin
This work defines and study Mealy-robust LTL properties, which are insensitive to which of these interpretations of LTL is chosen, and investigates the synchronous semantics and the alternating semantics for Mealy machines.


Rigorous examination of reactive systems
The STTT special section on RERS describes the results of the evaluations and the different analysis techniques that were used in the RERS challenges 2012 and 2013.
Status Report on Software Verification - (Competition Summary SV-COMP 2014)
The report represents the state of the art in automatic software verification, in terms of effectiveness and efficiency, at the 3rd International Competition on Software Verification (SV-COMP 2014).
Property-driven benchmark generation: synthesizing programs of realistic structure
We present a systematic approach to the automatic generation of platform-independent benchmarks of realistic structure and tailored complexity for evaluating verification tools for reactive systems.
VerifyThis 2015 A program verification competition
The VerifyThis 2015 event, the challenges that were posed during the competition, and a high-level overview of the solutions to these challenges are provided.
Reliable and Reproducible Competition Results with BenchExec and Witnesses (Report on SV-COMP 2016)
This report presents the results of the competition and includes a special section that describes how SV-COMP ensures that the experiments are reliably executed, precisely measured, and organized such that the results can be reproduced later.
VerifyThis 2012
An overview of the VerifyThis competition series, an account of related activities in the area, and an overview of solutions submitted to the organizers both during and after the 2012 competition are provided.
Model Checking
Model checking is applied concurrently with system design, and particularly in its early stages when systems are modelled at a high level of abstraction, because the payoff of finding bugs at that stage is highest whereas the costs are low.
Model-Based Testing of Reactive Systems
This chapter focuses on formal objects used to select test suites, so-called coverage criteria, and focuses on the fault detection ability of a test suite satisfying a given coverage criterion.
Software model checking: extracting verification models from source code †
It is argued, though, that the method was originally developed to allow for a thorough testing of parts of the software of a new commercial telephone switch also has broad applicability to distributed software systems design in general.
The SPIN Model Checker - primer and reference manual
This is the most comprehensive reference guide to SPIN, written by the principal designer of the tool, and gives detailed advice on methods for tackling the most complex software verification problems.