GEM: Glare or Gloom, I Can Still See You – End-to-End Multi-Modal Object Detection

  title={GEM: Glare or Gloom, I Can Still See You – End-to-End Multi-Modal Object Detection},
  author={Osama Mazhar and Robert Babu{\vs}ka and Jens Kober},
  journal={IEEE Robotics and Automation Letters},
Deep neural networks designed for vision tasks are often prone to failure when they encounter environmental conditions not covered by the training data. Single-modal strategies are insufficient when the sensor fails to acquire information due to malfunction or its design limitations. Multi-sensor configurations are known to provide redundancy, increase reliability, and are crucial in achieving robustness against asymmetric sensor failures. To address the issue of changing lighting conditions… Expand

Figures and Tables from this paper


Random Shadows and Highlights: A new data augmentation method for extreme lighting conditions
This paper demonstrates that RSH not only increases the robustness of the models against lighting perturbations, but also reduces over-fitting significantly, thus RSH should be considered essential for all vision related learning systems. Expand
End-to-End Object Detection with Transformers
This work presents a new method that views object detection as a direct set prediction problem, and demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Expand
Focal Loss for Dense Object Detection
This paper proposes to address the extreme foreground-background class imbalance encountered during training of dense detectors by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples, and develops a novel Focal Loss, which focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. Expand
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather
A deep fusion network for robust fusion without a large corpus of labeled training data covering all asymmetric distortions is presented, and a single-shot model that adaptively fuses features, driven by measurement entropy is proposed. Expand
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
A new data augmentation method is introduced which advances the state-of-the-art and outperforms models pretrained with 1000x more labeled data and concludes that future research must study multiple distribution shifts simultaneously. Expand
A Fourier Perspective on Model Robustness in Computer Vision
AutoAugment, a recently proposed data augmentation policy optimized for clean accuracy, achieves state-of-the-art robustness on the CIFAR-10-C benchmark and is observed to use a more diverse set of augmentations than previously observed. Expand
A Survey of Deep Learningbased Object Detection
  • IEEE Access
  • 2019
Borrow From Anywhere: Pseudo Multi-Modal Object Detection in Thermal Imagery
This paper proposes a pseudo-multimodal object detector trained on natural image domain data to help improve the performance of object detection in thermal images and shows that the framework outperforms existing benchmarks without the explicit need for paired training examples from the two domains. Expand
Deep Learning for Generic Object Detection: A Survey
A comprehensive survey of the recent achievements in this field brought about by deep learning techniques, covering many aspects of generic object detection: detection frameworks, object feature representation, object proposal generation, context modeling, training strategies, and evaluation metrics. Expand
LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks
The proposed cross fusion FCN was evaluated on the KITTI road benchmark where it achieved excellent performance, with a MaxF score of 96.03%, ranking it among the top-performing approaches. Expand