Deep Learning compatible Differentiable X-ray Projections for Inverse Rendering

  title={Deep Learning compatible Differentiable X-ray Projections for Inverse Rendering},
  author={Karthik Shetty and Annette I. Birkhold and Norbert Strobel and Bernhard Egger and Srikrishna Jaganathan and Markus Kowarschik and Andreas K. Maier},
  booktitle={Bildverarbeitung f{\"u}r die Medizin},
Many minimally invasive interventional procedures still rely on 2D fluoroscopic imaging. Generating a patient-specific 3D model from these X-ray projection data would allow to improve the procedural workflow, e.g. by providing assistance functions such as automatic positioning. To accomplish this, two things are required. First, a statistical human shape model of the human anatomy and second, a differentiable X-ray renderer. In this work, we propose a differentiable renderer by deriving the… Expand
1 Citations

Figures from this paper

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective
The impact of machine learning on 2D/3D registration is reviewed to systematically summarize the recent advances made by introduction of this novel technology and offers a perspective on the most pressing needs, significant open problems, and possible next steps. Expand


Fast Generation of Virtual X-ray Images for Reconstruction of 3D Anatomy
A novel GPU-based approach to render virtual X-ray projections of deformable tetrahedral meshes that will improve treatments in orthopedics, where 3D anatomical information is essential, and significantly boosts the deformation/projection performance. Expand
X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery
This work presents a method to automatically detect anatomical landmarks in X-ray images independent of the viewing direction and shows that its intra-operative landmark detection together with pre-operative CT enables X-rays pose estimation which, ultimately, benefits initialization of image-based 2D/3D registration. Expand
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
This work proposes a truly differentiable rendering framework that is able to directly render colorized mesh using differentiable functions and back-propagate efficient supervision signals to mesh vertices and their attributes from various forms of image representations, including silhouette, shading and color images. Expand
Physics-Driven Learning of X-ray Skin Dose Distribution in Interventional Procedures.
The combination of deep neural networks and MC simulation of particle physics has the potential to decrease the computational complexity of accurate skin dose estimation and can provide dose distributions in under one second when running on high-end hardware. Expand
A comparison of similarity measures for use in 2-D-3-D medical image registration
Results show that the introduction of soft-tissue structures and interventional instruments into the phantom image can have a large effect on the performance of some similarity measures previously applied to 2-D-3-D image registration. Expand
Simulation of X-ray Attenuation on the GPU
The results show that the GPU implementation with full X-ray point precision is faster by a factor of about 60 to 65 than the CPU implementation, without any sign of loss of accuracy, and the increase inperformance achieved with GPU calculations opens up new perspectives. Expand
Accelerating 3D deep learning with PyTorch3D
1. Accelerating 3D Deep Learning with PyTorch3D, arXiv 2007.08501 2. Mesh R-CNN, ICCV 2019 3. SynSin: End-to-end View Synthesis from a Single Image, CVPR 2020 4. Fast Differentiable Raycasting forExpand
SMPL: a skinned multi-person linear model
The Skinned Multi-Person Linear model (SMPL) is a skinned vertex-based model that accurately represents a wide variety of body shapes in natural human poses that is compatible with existing graphics pipelines and iscompatible with existing rendering engines. Expand
Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration
The first accurately annotated, non-synthetic, dataset of hip fluoroscopy is created by using anatomical annotations produced by a neural network as training data for neural networks and state-of-the-art performance in fluoroscopic segmentation and landmark localization was achieved. Expand
SMPL: A Skinned Multi-Person
  • Linear Model. ACM Trans Graph (Proc SIGGRAPH Asia)
  • 2015