Compositional Falsification of Cyber-Physical Systems with Machine Learning Components

@article{Dreossi2018CompositionalFO,
  title={Compositional Falsification of Cyber-Physical Systems with Machine Learning Components},
  author={Tommaso Dreossi and Alexandre Donz{\'e} and Sanjit A. Seshia},
  journal={Journal of Automated Reasoning},
  year={2018},
  pages={1-23}
}
Cyber-physical systems (CPS), such as automotive systems, are starting to include sophisticated machine learning (ML) components. Their correctness, therefore, depends on properties of the inner ML modules. While learning algorithms aim to generalize from examples, they are only as good as the examples provided, and recent efforts have shown that they can produce inconsistent output under small adversarial perturbations. This raises the question: can the output from learning components lead to… 
Compositional Falsification of Cyber-Physical Systems with Machine Learning Components
TLDR
This work proposes a compositional falsification framework where a temporal logic falsifier and a machine learning analyzer cooperate with the aim of finding falsifying executions of the considered model.
Semantic Adversarial Deep Learning
TLDR
It is argued that the semantics and specification of the overall system has a crucial role to play in this line of research and preliminary research results are presented that support this claim.
VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems
TLDR
VerifAI particularly addresses challenges with applying formal methods to ML components such as perception systems based on deep neural networks, as well as systems containing them, and to model and analyze system behavior in the presence of environment uncertainty.
Compositional Verification without Compositional Specification for Learning-Based Systems
TLDR
This work presents an approach involving a system-level verifier communicating with a component-level analyzer wherein the former identifies a subset of environment behaviors that might lead to aSystem-level failure while the latter identifies erroneous behaviors of the machine learning component that might be extended to asystem-level counterexample.
Towards classes of architectural dependability assurance for machine-learning-based systems
TLDR
This paper discusses how this problem can be addressed at architectural rather than algorithmic level to assess systems dependability properties in early development stages and proposes classes of dependability in which ML-based systems may be categorized and discusses which and how assurances can be made for each class.
Challenges of Machine Learning Applied to Safety-Critical Cyber-Physical Systems
TLDR
The comprehensive analysis presented in this paper is intended as a basis for future holistic approaches for safety engineering of ML-based CPS in safety-critical applications, and aims to support the focus on research onto safety hazards that are not yet adequately addressed.
Synthesis-guided Adversarial Scenario Generation for Gray-box Feedback Control Systems with Sensing Imperfections
TLDR
An algorithm is developed that searches for “adversarial scenarios", which can be thought of as the strategy for the adversary representing the noise and disturbances, that lead to safety violations in closed-loop systems with memoryless controllers.
BreachFlows: Simulation-Based Design with Formal Requirements for Industrial CPS (Extended Abstract)
TLDR
Breach is an open source Matlab/Simulink toolbox that implements falsification in a modular and extensible way and is used in academia as well as for industrial applications, in particular in the automotive domain.
Making AI more trustworthy with a formal methods-based approach to AI system verification and validation. BY SANJIT A. SESHIA, DORSA SADIGH, AND S. SHANKAR SASTRY
  • 2022
...
...

References

SHOWING 1-10 OF 46 REFERENCES
Compositional Falsification of Cyber-Physical Systems with Machine Learning Components
TLDR
This work proposes a compositional falsification framework where a temporal logic falsifier and a machine learning analyzer cooperate with the aim of finding falsifying executions of the considered model.
Semantic Adversarial Deep Learning
TLDR
It is argued that the semantics and specification of the overall system has a crucial role to play in this line of research and preliminary research results are presented that support this claim.
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
TLDR
DeepXplore efficiently finds thousands of incorrect corner case behaviors in state-of-the-art DL models with thousands of neurons trained on five popular datasets including ImageNet and Udacity self-driving challenge data.
Efficient Guiding Strategies for Testing of Temporal Properties of Hybrid Systems
TLDR
This paper presents an approach that uses the rapidly exploring random trees (RRT) technique to explore the state-space of a CPS, and shows that it scales to industrial-scale CPSs by demonstrating its efficacy on an automotive powertrain control system.
Safety Verification of Deep Neural Networks
TLDR
A novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT) is developed, which defines safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image.
Towards Evaluating the Robustness of Neural Networks
TLDR
It is demonstrated that defensive distillation does not significantly increase the robustness of neural networks, and three new attack algorithms are introduced that are successful on both distilled and undistilled neural networks with 100% probability are introduced.
C2E2: A Verification Tool for Stateflow Models
TLDR
The C2E2 algorithm is sound and it is guaranteed to terminate if the system is robustly safe or unsafe with respect to perturbations of guards and invariants of the model.
Logical Clustering and Learning for Time-Series Data
TLDR
This work utilizes monotonic Parametric Signal Temporal Logic (PSTL) to design features for unsupervised classification of time series data that enables using off-the-shelf machine learning tools to automatically cluster similar traces with respect to a given PSTL formula.
Logical Clustering and Learning for Time-Series Data
TLDR
This work utilizes monotonic parametric signal temporal logic (PSTL) to design features for unsupervised classification of time series data that enables using off-the-shelf machine learning tools to automatically cluster similar traces with respect to a given PSTL formula.
S-TaLiRo: A Tool for Temporal Logic Falsification for Hybrid Systems
TLDR
S-TaLiRo is a Matlab toolbox that searches for trajectories of minimal robustness in Simulink/Stateflow diagrams using randomized testing based on stochastic optimization techniques including Monte-Carlo methods and Ant-Colony Optimization.
...
...