Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

@article{Mller2018Sim4CVAP,
  title={Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications},
  author={Matthias M{\"u}ller and Vincent Casser and Jean Lahoud and Neil G. Smith and Bernard Ghanem},
  journal={International Journal of Computer Vision},
  year={2018},
  volume={126},
  pages={902-919}
}
We present a photo-realistic training and evaluation simulator (Sim4CV) (http://www.sim4cv.org) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous… Expand
Augmented LiDAR Simulator for Autonomous Driving
TLDR
This letter proposes a novel LiDAR simulator that augments real point cloud with synthetic obstacles (e.g., vehicles, pedestrians, and other movable objects) and describes the placement of obstacles that is critical for performance enhancement. Expand
AADS: Augmented autonomous driving simulation using data-driven algorithms
TLDR
This work combines augmented real-world pictures with a simulated traffic flow to create photorealistic simulation images and renderings that are ready for training and testing of AD systems from perception to planning. Expand
Unlimited Road-scene Synthetic Annotation (URSA) Dataset
TLDR
This work provides a method for persistent, ground truth, asset annotation of a game world, using open-source tools and resources found in single-player modding communities, and demonstrates realtime, on-demand, groundtruth data annotation capability of this method. Expand
Semantic Segmentation Learning for Autonomous UAVs using Simulators and Real Data
TLDR
A thorough survey of the most recent and popular simulators and synthetic datasets is made, exploring solutions for semantic segmentation on images taken from drones and proposing an extension of the CARLA simulator by introducing an aerial camera. Expand
VIVID: Virtual Environment for Visual Deep Learning
TLDR
A new Virtual Environment for Visual Deep Learning (VIVID) is presented, which offers large-scale diversified indoor and outdoor scenes and leverages the advanced human skeleton system, which enables us to simulate numerous complex human actions. Expand
The RobotriX: An Extremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions
TLDR
The RobotriX is an extremely photorealistic indoor dataset designed to enable the application of deep learning techniques to a wide variety of robotic vision problems and will serve as a new milestone for investigating 2D and 3D robotic vision tasks with large-scale data-driven techniques. Expand
Pavilion: Bridging Photo-Realism and Robotics
  • Fan Jiang, Qi Hao
  • Computer Science
  • 2019 International Conference on Robotics and Automation (ICRA)
  • 2019
TLDR
Pavilion, a novel open-source simulation system, for robot perception and kinematic control based on the Unreal Engine and the Robot Operating System, and a Gazebo-compatible real-time simulation system is developed to enable training and evaluation of a large number of sensor fusion, planning, decision and control algorithms. Expand
Synthetic training data for deep neural networks on visual correspondence tasks
TLDR
This thesis motivates and describes the making of large synthetic datasets for low-level correspondence matching problems, and uses these datasets to train deep neural networks for the fundamental vision tasks of optical flow and stereo disparity estimation, achieving a new state of the art at the time of their publication. Expand
3D Part Guided Image Editing for Fine-Grained Object Understanding
TLDR
This paper proposes an effective training data generation process by fitting a 3D car model with dynamic parts to cars in real images, and demonstrates that the trained network with edited images largely outperforms other baselines in terms of 2D detection and instance segmentation accuracy. Expand
DronePose: Photorealistic UAV-Assistant Dataset Synthesis for 3D Pose Estimation via a Smooth Silhouette Loss
TLDR
A data synthesis pipeline is designed to create a realistic multimodal dataset that includes both the exocentric user view, and the egocentric UAV view and exploits the joint availability of photorealistic and synthesized inputs to train a single-shot monocular pose estimation model. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 79 REFERENCES
VirtualWorlds as Proxy for Multi-object Tracking Analysis
TLDR
This work proposes an efficient real-to-virtual world cloning method, and validate the approach by building and publicly releasing a new video dataset, called "Virtual KITTI", automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. Expand
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
TLDR
A new simulator built on Unreal Engine that offers physically and visually realistic simulations for autonomous vehicles in real world and that is designed from the ground up to be extensible to accommodate new types of vehicles, hardware platforms and software protocols. Expand
A Benchmark and Simulator for UAV Tracking
TLDR
A new aerial video dataset and benchmark for low altitude UAV target tracking, as well as, a photo-realistic UAV simulator that can be coupled with tracking methods to easily extend existing real-world datasets. Expand
The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
TLDR
This paper generates a synthetic collection of diverse urban images, named SYNTHIA, with automatically generated class annotations, and conducts experiments with DCNNs that show how the inclusion of SYnTHIA in the training stage significantly improves performance on the semantic segmentation task. Expand
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image
TLDR
This paper proposes a learning method that they call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models, and shows that it can train a policy that generalizes to thereal world, without requiring the simulator to be particularly realistic or high-fidelity. Expand
Deep Neural Network for Real-Time Autonomous Indoor Navigation
TLDR
A deep learning model, Convolutional Neural Network (ConvNet), is used to learn a controller strategy that mimics an expert pilot's choice of action, and a practical system in which a quadcopter autonomously navigates indoors and finds a specific target by using a single camera. Expand
End to End Learning for Self-Driving Cars
TLDR
A convolutional neural network is trained to map raw pixels from a single front-facing camera directly to steering commands and it is argued that this will eventually lead to better performance and smaller systems. Expand
Playing for Data: Ground Truth from Computer Games
TLDR
It is shown that associations between image patches can be reconstructed from the communication between the game and the graphics hardware, which enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. Expand
Robust real-time vision-based aircraft tracking from Unmanned Aerial Vehicles
TLDR
This paper presents a novel robust visual tracking algorithm for UAVs in the midair to track an arbitrary aircraft at real-time frame rates, together with a unique evaluation system, that is called Adaptive M3 tracker, i.e. AM3. Expand
Semantic Pose Using Deep Networks Trained on Synthetic RGB-D
TLDR
This work proposes to find instances of common furniture classes, their spatial extent, and their pose with respect to generalized class models, and uses a deep, wide, multi-output convolutional neural network that predicts class, pose, and location of possible objects simultaneously. Expand
...
1
2
3
4
5
...