Multi-Modal Pedestrian Detection with Large Misalignment Based on Modal-Wise Regression and Multi-Modal IoU

  title={Multi-Modal Pedestrian Detection with Large Misalignment Based on Modal-Wise Regression and Multi-Modal IoU},
  author={Napat Wanchaitanawong and Masayuki Tanaka and Takashi Shibata and M. Okutomi},
  journal={2021 17th International Conference on Machine Vision and Applications (MVA)},
  • Napat Wanchaitanawong, Masayuki Tanaka, +1 author M. Okutomi
  • Published 2021
  • Computer Science
  • 2021 17th International Conference on Machine Vision and Applications (MVA)
The combined use of multiple modalities enables accurate pedestrian detection under poor lighting conditions by using the high visibility areas from these modalities together. The vital assumption for the combination use is that there is no or only a weak misalignment between the two modalities. In general, however, this assumption often breaks in actual situations. Due to this assumption's breakdown, the position of the bounding boxes does not match between the two modalities, resulting in a… Expand

Figures and Tables from this paper


Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection
A novel Aligned Region CNN (AR-CNN) is proposed to handle the weakly aligned multispectral data in an end-to-end way, and a new multimodal fusion method, which performs feature re-weighting to select more reliable features and suppress the useless ones is presented. Expand
Unified multi-spectral pedestrian detection based on probabilistic fusion networks
This paper proposes a novel sensor fusion framework for detecting pedestrians even in challenging real-world environments that outperforms the state-of-the-art methods on the challenging KAIST, CVC-14, and DIML multi-spectral pedestrian datasets. Expand
Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation
A network fusion architecture is proposed, which consists of a multispectral proposal network to generate pedestrian proposals, and a subsequent mult ispectral classification network to distinguish pedestrian instances from hard negatives and significantly outperforms state-of-the-art methods on the KAIST dataset. Expand
Cross-modality interactive attention network for multispectral pedestrian detection
A novel cross-modality interactive attention network that takes full advantage of the interactive properties of multispectral input sources is proposed that achieves state-of-the-art performance with high efficiency. Expand
Learning Cross-Modal Deep Representations for Robust Pedestrian Detection
The proposed approach outperforms the state-of-the-art on the challenging KAIST multispectral pedestrian dataset and it is competitive with previous methods on the popular Caltech dataset. Expand
Pedestrian Detection at Day/Night Time with Visible and FIR Cameras: A Comparison
To assess the accuracy gain of different pedestrian models (holistic, part-based, patch-based) when training with images in the far infrared spectrum, a new dataset is built and an early fusion approach to combine features from both modalities is proposed. Expand
Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection
This paper deeply compares six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Expand
Multispectral pedestrian detection: Benchmark dataset and baseline
This dataset introduces multispectral ACF, which is an extension of aggregated channel features (ACF) to simultaneously handle color-thermal image pairs, and achieves another breakthrough in the pedestrian detection task. Expand
Multi-spectral pedestrian detection based on accumulated object proposal with fully convolutional networks
This framework is shown to surpass state-of-the-art pedestrian detection methods on multi-spectral pedestrian benchmark and inferring the confidence of a pedestrian using deep convolutional neural networks and support vector regression. Expand
Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection
This paper demonstrates illumination information encoded in multispectral images can be utilized to significantly boost performance of pedestrian detection and presents a powerful framework based on multi-task learning of illumination-aware pedestrians detection and semantic segmentation. Expand