Scenic: A Language for Scenario Specification and Data Generation

  title={Scenic: A Language for Scenario Specification and Data Generation},
  author={Daniel J. Fremont and Edward J. Kim and Tommaso Dreossi and Shromona Ghosh and Xiangyu Yue and Alberto L. Sangiovanni-Vincentelli and Sanjit A. Seshia},
We propose a new probabilistic programming language for the design and analysis of cyber-physical systems, especially those based on machine learning. We consider several problems arising in the design process, including training a system to be robust to rare events, testing its performance under different conditions, and debugging failures. We show how a probabilistic programming language can help address these problems by specifying distributions encoding interesting types of inputs, then… 

Runtime Monitors for Operational Design Domains of Black-Box ML Models

This paper presents a framework for learning runtime monitors for ODDs of autonomous systems with black-box ML components, and investigates the learning of optimal monitors based on counterexample-guided refinement and conformance testing.

ADEPT: A Testing Platform for Simulated Autonomous Driving

A new testing platform ADEPT, aiming to provide practically realistic and comprehensive testing facilities for DNN-based ADS, is reported, based on the virtual simulator CARLA and features two distinguished test scenario generation strategies designed for autonomous driving.

Addressing the IEEE AV Test Challenge with Scenic and VerifAI

A systematic testing framework leveraging previous work on formally-driven simulation for intelligent cyber-physical systems is demonstrated by identifying concrete failure scenarios for an open-source autopilot, Apollo, starting from a variety of realistic traffic scenarios.

Barrier-Based Test Synthesis for Safety-Critical Systems Subject to Timed Reach-Avoid Specifications

An adversarial, time-varying test-synthesis procedure that does not require specific knowledge of the system’s controller, but can still provably identify realizable and maximally dif ficult tests of system behavior.

Simulation-based Validation for Autonomous Driving Systems

This work investigates a rigorous simulation and testing-based validation method for autonomous driving systems that integrates an existing industrial simulator and a formally defined testing environment and shows that the systematic exploration of risky situations has uncovered many flaws in the real simulator that would have been very difficult to discover by a random exploration process.

AmbieGen: A Search-based Framework for Autonomous Systems Testing

AmbieGen is introduced, a search-based test case generation framework for autonomous systems that uses evolutionary search to identify the most critical scenarios for a given system, and has a modular architecture that allows for the addition of new systems under test, algorithms, and search operators.

A Survey on Autonomous Driving System Simulators

Advantages and disadvantages of the commonly used simulators such as CarSim, CarMaker and AirSim are compared from aspects like virtual environment generation, critical scenarios creation, types of supported sensor, as well as control of traffic participants.

Behavior-Tree Based Scenario Specification and Test Case Generation for Autonomous Driving Simulation

A scenario description language, called BTScenario, is proposed, which supports both the high-level description of the spatial layout of map objects and the Behavior Tree-based specification of temporal behaviors of actors (e.g. cars and pedestrians).

Digitization of Chinese Traffic Laws: Methodologies, Quantative Analysis, and Usage for Monitoring Driving Compliance

A layered trigger domain-based traffic law digitization architecture with digitization-classifled discussions and detailed atomic propositions for online moni- toring is introduced.

Learning Monitorable Operational Design Domains for Assured Autonomy

This paper presents a framework for learning runtime monitors that capture the ODDs of black-box systems and investigates the learning of optimal monitors based on counterexample-guided refinement and conformance testing.



Grand Theft Auto IV

This article has two goals: one is to establish techniques for content analysis of a virtual environment, Liberty City, as part of a dynamic textual object, Grand Theft Auto IV (GTA IV). The second

CARLA: An Open Urban Driving Simulator

This work introduces CARLA, an open-source simulator for autonomous driving research, and uses it to study the performance of three approaches to autonomous driving: a classic modular pipeline, an end-to-end model trained via imitation learning, and an end to-end models trained via reinforcement learning.

SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving

SqueezeDet is a fully convolutional neural network for object detection that aims to simultaneously satisfy all of the above constraints, and is very accurate, achieving state-of-the-art accuracy on the KITTI benchmark.

Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?

A method to incorporate photo-realistic computer images from a simulation engine to rapidly generate annotated data that can be used for the training of machine learning algorithms, which offers the possibility of accelerating deep learning's application to sensor-based classification problems like those that appear in self-driving cars.

Probabilistic programming

This paper describes connections this research area called ``Probabilistic Programming" has with programming languages and software engineering, and this includes language design, and the static and dynamic analysis of programs.

Scenic: A language for scenario specification and data generation (2020)

  • 2020

Specifying real-time properties with metric temporal logic

This paper characterize real-time systems by giving a classification of such quantitative temporal properties, and extends the usual models for temporal logic by including a distance function to measure time and analyzes what restrictions should be imposed on such a function.

Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI

A unified approach to rigorous design of safety-critical autonomous systems using the VerifAI toolkit for formal analysis of AI-based systems and the results of falsification and debugging are used to retrain the network, eliminating several failure cases and improving the overall performance of the closed-loop system.

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

The value of the synthesized dataset is demonstrated, by improving performance in certain machine-learning-based scene understanding tasks—depth and surface normal prediction, semantic segmentation, reconstruction, etc.—and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.

Probabilistic programming for procedural modeling and design