Rolling-Shutter-Aware Differential SfM and Image Rectification

  title={Rolling-Shutter-Aware Differential SfM and Image Rectification},
  author={Bingbing Zhuang and Loong Fah Cheong and Gim Hee Lee},
  journal={2017 IEEE International Conference on Computer Vision (ICCV)},
In this paper, we develop a modified differential Structure from Motion (SfM) algorithm that can estimate relative pose from two consecutive frames despite of Rolling Shutter (RS) artifacts. In particular, we show that under constant velocity assumption, the errors induced by the rolling shutter effect can be easily rectified by a linear scaling operation on each optical flow. We further propose a 9-point algorithm to recover the relative pose of a rolling shutter camera that undergoes constant… 

Figures from this paper

IMU-Assisted Learning of Single-View Rolling Shutter Correction
A deep neural network is proposed to predict depth and row-wise pose from a single image for rolling shutter correction, to incorporate inertial measurement unit (IMU) data into the pose refinement process, which greatly enhances the pose prediction.
Occlusion-Aware Rolling Shutter Rectification of 3D Scenes
A first-of-its-kind pipeline to recover the latent image of a 3D scene from a set of such RS distorted images by sequentially recovering both the camera motion and scene structure while accounting for RS and occlusion effects is developed.
Image Stitching and Rectification for Hand-Held Cameras
A new differential homography that can account for the scanline-varying camera poses in Rolling Shutter (RS) cameras is derived, and its application to carry out RS-aware image stitching and rectification at one stroke is demonstrated.
Neural Global Shutter: Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature
This paper investigates using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos and develops a novel algorithm incorporating spatial and temporal designs to correct the spatial-varying RSGR distortion.
Rolling Shutter Pose and Ego-Motion Estimation Using Shape-from-Template
A new method for the absolute camera pose problem (PnP) which handles Rolling Shutter effects which outperforms existing RS pose estimation techniques in terms of accuracy and stability of performance in various configurations.
Learning Structure-And-Motion-Aware Rolling Shutter Correction
This paper first makes a theoretical contribution by showing that RS two-view geometry is degenerate in the case of pure translational camera motion, and proposes a Convolutional Neural Network (CNN)-based method which learns the underlying geometry from just a single RS image and performs RS image correction.
Learning Rolling Shutter Correction from Real Data without Camera Motion Assumption
A deep neural network is proposed to predict depth and camera poses for single-frame rolling shutter correction and it is enabled by training on real images captured by rolling shutter cameras instead of synthetic ones generated with certain motion assumption.
Context-Aware Video Reconstruction for Rolling Shutter Cameras
An approximated bilateral motion model is derived, which can serve as an alternative to provide a simple but effective GS frame initialization for related tasks and achieves superior performance over state-of-the-art methods in terms of objective metrics and subjective visual quality.
3D Vision Geometry for Rolling Shutter Cameras
This thesis proposes a general framework for modelling and solving structure from motion (SfM) with RS cameras, and presents a novel RS correction method which uses line features and a new RS Homography matrix based on point correspondences from an RS pair.
Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video
  • Bin Fan, Yuchao Dai
  • Computer Science
    2021 IEEE/CVF International Conference on Computer Vision (ICCV)
  • 2021
The first RS temporal super-resolution network in a cascaded structure to extract high framerate global shutter video and explores the underlying spatio-temporal geometric relationships within a deep learning framework, where no extra supervision besides the middle-scanline ground truth GS image is needed.


Sparse to Dense 3D Reconstruction from Rolling Shutter Images
A cost function for Bundle Adjustment (BA) is proposed that models the rolling shutter effect, incorporates GPS/INS readings, and enforces pairwise smoothness between neighboring poses and introduces a novel interpolation scheme for therolling shutter plane sweep stereo algorithm.
Rolling Shutter Stereo
It is shown that naive 3D reconstructions (assuming global shutter) will deliver biased geometry already for very mild assumptions on vehicle speed and resolution, and rolling shutter dense multiview stereo algorithms that solve for time of exposure and depth at the same time are developed.
A minimal solution to the rolling shutter pose estimation problem
This work derives a minimal solution using the Gröbner Basis that solves for the absolute pose as well as the motion of a rolling shutter camera, and shows that the minimal problem requires 5-point correspondences and gives up to 8 real solutions.
CTSR 2011-03 Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes
A robust, real-time video stabilization and rolling shutter correction technique based on commodity gyroscopes based on a novel framework for automatically calibrating the gyroscope and camera outputs from a single video capture.
Rolling shutter bundle adjustment
A bundle adjustment (BA) method that obtains accurate structure and motion from rolling shutter video sequences: RSBA, which succeeds in cases where competing methods diverge, and consistently produces more accurate results.
High Quality Structure from Small Motion for Rolling Shutter Cameras
A novel 3D reconstruction method from narrow-baseline image sequences that effectively handles the effects of a rolling shutter that occur from most of commercial digital cameras is introduced and consistently generates better 3D depth maps than those by the state-of-the-art method.
Efficient Video Rectification and Stabilisation for Cell-Phones
A method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras by smoothing a sequence of reference rotations from the estimated curve, to obtain a high-quality image stabilisation.
Geometric Models of Rolling-Shutter Cameras
A general projection equation for a rolling shutter camera is developed and it is demonstrated how the rolling shutter effects the projective geometry of the camera and in turn the structure-from-motion.
Calibration-free rolling shutter removal
This work presents a novel algorithm for efficient removal of rolling shutter distortions in uncalibrated streaming videos that is computationally efficient, easy to parallelize, and robust to challenging artifacts introduced by various cameras with differing technologies.
Rolling Shutter Camera Relative Pose: Generalized Epipolar Geometry
The rolling shutter essential matrix is introduced, and its link to existing models such as the push-broom cameras is established, summarized in a clean hierarchy of multi-perspective cameras.