• Corpus ID: 233231721

Decoupled Spatial-Temporal Transformer for Video Inpainting

@article{Liu2021DecoupledST,
  title={Decoupled Spatial-Temporal Transformer for Video Inpainting},
  author={R. Liu and Hanming Deng and Yangyi Huang and Xiaoyu Shi and Lewei Lu and Wenxiu Sun and Xiaogang Wang and Jifeng Dai and Hongsheng Li},
  journal={ArXiv},
  year={2021},
  volume={abs/2104.06637}
}

• R. Liu, Hanming Deng, +6 authors Hongsheng Li
• Published 14 April 2021
• Computer Science
• ArXiv

Video inpainting aims to fill the given spatiotemporal holes with realistic appearance but is still a challenging task even with prosperous deep learning approaches. Recent works introduce the promising Transformer architecture into deep video inpainting and achieve better performance. However, it still suffers from synthesizing blurry texture as well as huge computational cost. Towards this end, we propose a novel Decoupled Spatial-Temporal Transformer (DSTT) for improving video inpainting with exceptional efficiency. Our proposed DSTT disentangles the task of learning spatial-temporal attention into 2 sub-tasks: one is for attending temporal object movements on different frames at same spatial locations, which is achieved by temporally-decoupled Transformer block, and the other is for attending similar background textures on same frame of all spatial positions, which is achieved by spatiallydecoupled Transformer block. The interweaving stack of such two blocks makes our proposed model attend background textures and moving objects more precisely, and thus the attended plausible and temporally-coherent appearance can be propagated to fill the holes. In addition, a hierarchical encoder is adopted before the stack of Transformer blocks, for learning robust and hierarchical features that maintain multi-level local spatial structure, resulting in the more representative token vectors. Seamless combination of these two novel designs forms a better spatial-temporal attention scheme and our proposed model achieves better performance than state-of-theart video inpainting approaches with significant boosted efficiency. Training code and pretrained models are available at https://github.com/ruiliu-ai/DSTT.