The virtual lands of Oz: testing an agribot in simulation

@article{Robert2020TheVL,
  title={The virtual lands of Oz: testing an agribot in simulation},
  author={Cl{\'e}ment Robert and Thierry Sotiropoulos and J{\'e}r{\'e}mie Guiochet and H{\'e}l{\`e}ne Waeselynck and Simon Vernhes},
  journal={Empirical Software Engineering},
  year={2020},
  volume={25},
  pages={2025-2054}
}
Testing autonomous robots typically requires expensive test campaigns in the field. To alleviate them, a promising approach is to perform intensive tests in virtual environments. This paper presents an industrial case study on the feasibility and effectiveness of such an approach. The subject system is Oz, an agriculture robot for autonomous weeding. Its software was tested with weeding missions in virtual crop fields, using a 3D simulator based on Gazebo. The case study faced several… 
View on Springer
hal.archives-ouvertes.fr
9 Citations
Background Citations
6
Methods Citations
2

9 Citations

Testing a non-deterministic robot in simulation - How many repeated runs ?
TLDR
This paper addresses the research question of how many runs are required to activate a fault with an industrial case study of an agricultural weeding robot developed by Naïo Technologies and concludes that test input diversity is more efficient than repeated runs in order to activate faults.
A Study on the Challenges of Using Robotics Simulators for Testing
TLDR
A principled understanding of the ways developers use simulation in their process, and the challenges they face in doing so is developed to guide the development of more effective simulators and testing techniques for modern robotics development.
Integration of Test Generation Into Simulation-Based Platforms: An Experience Report
TLDR
This paper presents a practitioner's view of the integration of TAF into simulation-based test platforms, through two industrial case studies, for testing an agricultural robot developed by Naïo Technologies and a static perception system by SICK AG.
Simulation for Robotics Test Automation: Developer Perspectives
TLDR
A principled understanding is developed of the ways robotics developers use simulation in their testing processes and the challenges they face and ways in which simulators can be improved for use as a means of verification and validation.
Testing an Underwater Robot Executing Transect Missions in Mayotte
TLDR
This paper proposes five classes of oracle mission properties, used to perform test verification and evaluation: mission phases, time, energy, safety and localization, and applies this methodology on the semi-AUV prototype which executes autonomously biologic observation protocols in the Mayotte lagoon.
GzScenic: Automatic Scene Generation for Gazebo Simulator
TLDR
GzScenic; a tool that automatically generates scenes for simulation in Gazebo, a high-level probabilistic programming language that allows users to specify scenarios for simulation.
TAF: a Tool for Diverse and Constrained Test Case Generation
TLDR
A novel generic tool to generate constrained and diverse test cases from a data model that is defined using an XML-based domain-specific language and generated by combining random sampling with the use of an SMT solver.
Reliability Assessment and Safety Arguments for Machine Learning Components in Assuring Learning-Enabled Autonomous Systems
TLDR
An overall assurance framework for LES is presented with an emphasis on quantitative aspects, e.g., breaking down system-level safety targets to component-level requirements and supporting claims stated in reliability metrics, and a novel model-agnostic Reliability Assessment Model for ML classifiers that utilises the operational profile and robustness verification evidence.
Towards Autonomous Robotic Systems: 21st Annual Conference, TAROS 2020, Nottingham, UK, September 16, 2020, Proceedings

References

SHOWING 1-10 OF 32 REFERENCES
Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study
TLDR
The detailed analysis of the triggers and effects of these bugs shows that most of them can be revealed in low-fidelity simulation, and provides insights into interesting navigation scenarios to test as well as into how to address the test oracle problem.
Virtual Worlds for Testing Robot Navigation: A Study on the Difficulty Level
TLDR
A test framework based on MORSE (Modular Open Robots Simulation Engine), and using the generation of virtual 3D worlds to challenge the navigation service is proposed, and the notion of the difficulty of the generated worlds is elaborate, which is characterized in terms of mission achievement, mission duration and trajectory curves.
Design and use paradigms for Gazebo, an open-source multi-robot simulator
  • N. Koenig, A. Howard
  • Art
    2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566)
  • 2004
TLDR
Gazebo is designed to fill this niche by creating a 3D dynamic multi-robot environment capable of recreating the complex worlds that would be encountered by the next generation of mobile robots.
Crashing Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?
TLDR
An empirical study on bugs that have been fixed in the widely used, open-source ArduPilot system finds that the majority of bugs manifest under simple conditions that can be easily reproduced in software-based simulation, and is used to inform a novel framework for testing for bugs in simulation, consistently and reproducibly.
Modular open robots simulation engine: MORSE
TLDR
The conception principles of the simulator, MORSE, are presented, which gives the possibility to evaluate the algorithms embedded in the software architecture of the robot within which they are to be integrated and some use-case illustrations.
Safety Validation of Sense and Avoid Algorithms Using Simulation and Evolutionary Search
TLDR
A safety validation approach for Sense and Avoid (SAA) algorithms aboard Unmanned Aerial Vehicles (UAVs) by accelerating the process of finding dangerous faults of SAA algorithms and supporting the safety validation process.
Testing Autonomous Robot Control Software Using Procedural Content Generation
TLDR
This work uses procedural content generation, as developed for the film and video game industries, to create a diverse range of test situations, and analyses the three highest rated situations in depth, which revealed weaknesses in the smoothed nearness-diagram control algorithm.
A method for the reproduction of vehicle test drives for the simulation based evaluation of image processing algorithms
  • M. Nentwig, M. Stamminger
  • Computer Science
    13th International IEEE Conference on Intelligent Transportation Systems
  • 2010
TLDR
The reproduction of a real test drive is described and simulation quality is examined by a vehicle detection algorithm and a more detailed introduction to appearance based methods, especially the used Haar-wavelet based approach is given.
Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms
TLDR
This work proposes an automated testing algorithm that builds on learnable evolutionary algorithms that outperforms a baseline evolutionary search algorithm and generates 78% more distinct, critical test scenarios compared to the baseline algorithm.
Test scenario selection for system-level verification and validation of geolocation-dependent automotive control systems
TLDR
An approach to determine the relevance of recorded scenarios and derive efficient sets of test scenarios and a two-step approach starts with a specification-based classification-tree for initial scenario selection.
...
...