Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL

  title={Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL},
  author={Hamid Ebabi and Mahshid Helali Moghadam and Markus Borg and Gregory Gay and Afonso Fontes and Kasper Socha},
  journal={2021 IEEE International Conference on Artificial Intelligence Testing (AITest)},
With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the… Expand

Figures from this paper


Testing advanced driver assistance systems using multi-objective search and neural networks
This paper provides a testing approach for ADAS by combining multi-objective search with surrogate models developed based on neural networks, and shows that combining the search algorithm with surrogate modeling improves the quality of the generated test cases, especially under tight and realistic computational resources. Expand
Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms
This work proposes an automated testing algorithm that builds on learnable evolutionary algorithms that outperforms a baseline evolutionary search algorithm and generates 78% more distinct, critical test scenarios compared to the baseline algorithm. Expand
Testing Autonomous Cars for Feature Interaction Failures using Many-Objective Search
A technique to detect feature interaction failures by casting this problem into a search-based test generation problem, and a new search- based test generation algorithm, called FITEST, that is guided by the hybrid test objectives. Expand
Challenges in Autonomous Vehicle Testing and Validation
Software testing is all too often simply a bug hunt rather than a wellconsidered exercise in ensuring quality. A more methodical approach than a simple cycle of system-level test-fail-patch-test willExpand
Simulation-Driven Validation of Advanced Driving- Assistance Systems
Advanced driving-assistance systems (ADAS) have recently received an increasing attention from the car industry. ADAS development represents a considerable effort to obtain efficient and reliableExpand
Generating effective test cases for self-driving cars from police reports
An extensive evaluation, consisting of a user study involving 34 participants and a quantitative analysis of the quality of the generated tests, shows that this work can generate accurate simulations of car crashes in a matter of minutes. Expand
Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study
Results show that simulator-generated datasets are able to yield DNN prediction errors that are similar to those obtained by testing DNNs with real-life datasets. Expand
CARLA: An Open Urban Driving Simulator
This work introduces CARLA, an open-source simulator for autonomous driving research, and uses it to study the performance of three approaches to autonomous driving: a classic modular pipeline, an end-to-end model trained via imitation learning, and an end to-end models trained via reinforcement learning. Expand
Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry
The state-of-the-art in verification and validation of safety-critical systems that rely on machine learning is reviewed, confirming that ISO 26262 largely contravenes the nature of DNNs. Expand
Testing machine learning based systems: a systematic mapping
A systematic mapping study about testing techniques for MLSs driven by 33 research questions and investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Expand