Learning to remove multipath distortions in Time-of-Flight range images for a robotic arm setup

  title={Learning to remove multipath distortions in Time-of-Flight range images for a robotic arm setup},
  author={Kilho Son and Ming-Yu Liu and Yuichi Taguchi},
  journal={2016 IEEE International Conference on Robotics and Automation (ICRA)},
Range images captured by Time-of-Flight (ToF) cameras are corrupted with multipath distortions due to interaction between modulated light signals and scenes. The interaction is often complicated, which makes a model-based solution elusive. We propose a learning-based approach for removing the multipath distortions for a ToF camera in a robotic arm setup. Our approach is based on deep learning. We use the robotic arm to automatically collect a large amount of ToF range images containing various… 

Deep Learning for Transient Image Reconstruction from ToF Data

A novel deep learning approach is introduced, which estimates the structure of the time-dependent scene impulse response and from it recovers a depth image with a reduced amount of MPI in ToF cameras by estimating the direct and global components of the incoming light.

Deep Learning for Multi-path Error Removal in ToF Sensors

A novel method for MPI removal and depth refinement exploiting an ad-hoc deep learning architecture working on data from a multi-frequency ToF camera using a Convolutional Neural Network made of two sub-networks.

Deep End-to-End Time-of-Flight Imaging

It is demonstrated that the proposed network can efficiently exploit the spatio-temporal structures of ToF frequency measurements, and validate the performance of the joint multipath removal, denoising and phase unwrapping method on a wide range of challenging scenes.

Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments

A new learning based end-to-end depth prediction network which takes noisy raw I-ToF signals as well as an RGB image and fuses their latent representation based on a multi step approach involving both implicit and explicit alignment to predict a high quality long range depth map aligned to the RGB viewpoint is proposed.

Time-of-Flight Cameras in Space: Pose Estimation with Deep Learning Methodologies

This work focuses on demonstrating the capability of detecting a part of a satellite floating in space, to support in-orbit servicing missions (e.g. for space debris removal) and reveals that deep learning techniques can obtain higher accuracy and robustness w.r.t. classical methods.

RADU: Ray-Aligned Depth Update Convolutions for ToF Data Denoising

This paper proposes an iterative correcting approach operating in 3D space, that is designed to learn on 2.5D data by enabling 3D point convolutions to correct the points’ positions along the view direction, and is able to outperform SOTA methods on several datasets.

A foreground detection algorithm for Time-of-Flight cameras adapted dynamic integration time adjustment and multipath distortions

A pixel-insensitive and threshold-free algorithm that can stably detect foreground under different circumstances including the two mentioned scenarios and is an end-to-end model based on deep learning.

MPI Planar Correction of Pulse Based ToF Cameras

  • Marian-Leontin PopL. Tamás
  • Computer Science
    2022 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR)
  • 2022
This paper focuses on the MPI reduction problem using Feature Pyramid Networks (FPN) which allow the mitigation of this type of artifact for pulse-based ToF cameras, and manages to attenuate theMPI effect on planar surfaces using a learning-based method on real ToF data.

Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning

Experimental results demonstrate that the proposed approach is able to effectively denoise real world data and to outperform state-of-the-art techniques.

iToF2dToF: A Robust and Flexible Representation for Data-Driven Time-of-Flight Imaging

The transient representation is revisited, it can be integrated with different rule-based depth sensing algorithms that are robust to low SNR and can deal with ambiguous scenarios that arise in practice (e.g., specular MPI, optical cross-talk).



Separating true range measurements from multi-path and scattering interference in commercial range cameras

An algorithm applied to both the Mesa Imaging SR-4000 and Canesta Inc.

Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration

An improved multipath model is formed and a compensation process in order to correct the multipath-related errors is designed and a calibration of the ToF illumination supports the process.

Capturing Time-of-Flight data with confidence

This paper proposes an improved per-pixel confidence measure using a Random Forest regressor trained with real-world data and argues that an improved confidence measure leads to superior reconstructions in subsequent steps of traditional scan processing pipelines.

Multipath Interference Compensation in Time-of-Flight Camera Images

  • S. Fuchs
  • Computer Science
    2010 20th International Conference on Pattern Recognition
  • 2010
A multipath model is formulated in order to estimate the interference and correct the measurements of multipath interference in a Time-of-flight camera and the positive impact of these corrections is experimentally demonstrated.

Extrinsic and depth calibration of ToF-cameras

  • S. FuchsG. Hirzinger
  • Computer Science
    2008 IEEE Conference on Computer Vision and Pattern Recognition
  • 2008
In this work a calibration process for ToF-cameras with respect to the intrinsic parameters, the depth measurement distortion and the pose of the camera relative to a robotpsilas end effector is described.

Stereo time-of-flight

Both in simulation and real images the stereo-ToF acquisition produces more accurate depth measurements, and a quantitative evaluation of the performance of the proposed method for different objects and stereo configurations is provided based on a simulation of the ToF cameras.

Technical Foundation and Calibration Methods for Time-of-Flight Cameras

This survey gives an overview over the current state of the art in ToF sensor calibration with a focus on continuous wave intensity modulation approach.

Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging

In correlation-based time-of-flight (C-ToF) imaging systems, light sources with temporally varying intensities illuminate the scene. Due to global illumination, the temporally varying radiance

Lock-in Time-of-Flight (ToF) Cameras: A Survey

This paper reviews the state-of-the art in the field of lock-in time-of-flight (ToF) cameras, their advantages, their limitations, the existing calibration methods, and the way they are being used,