Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

  title={Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries},
  author={Sandy Engelhardt and Raffaele De Simone and Peter M. Full and Matthias Karck and Ivo Wolf},
Current `dry lab' surgical phantom simulators are a valuable tool for surgeons which allows them to improve their dexterity and skill with surgical instruments. These phantoms mimic the haptic and shape of organs of interest, but lack a realistic visual appearance. In this work, we present an innovative application in which representations learned from real intraoperative endoscopic sequences are transferred to a surgical phantom scenario. The term hyperrealism is introduced in this field… 
Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training
A cross-domain conditional generative adversarial network approach (GAN) that aims to generate more consistent stereo pairs and shows substantial improvements in depth perception and realism evaluated.
Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation
A task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains and it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself.
Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs
This paper proposes to apply generative adversarial neural networks trained with a cycle consistency loss, or CycleGANs, to improve realism in ultrasound (US) simulation from computed tomography (CT) scans, and results can contribute to reduce the gap between artificially generated and real US scans.
Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data
A novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes is proposed, which produces consistent translations of arbitrary views and thus enables long-term consistent video synthesis.
OfGAN: Realistic Rendition of Synthetic Colonoscopy Videos
The Optical Flow Generative Adversarial Network (OfGAN) is developed to transform simulated colonoscopy videos into realistic ones while preserving annotation, and it is demonstrated that the performance of the OfGAN overwhelms the baseline method in relative tasks through both qualitative and quantitative evaluation.
MedGAN: Medical Image Translation using GANs
A new framework, named MedGAN, is proposed for medical image-to-image translation which operates on the image level in an end- to-end manner and outperforms other existing translation approaches.
Robotic Instrument Segmentation With Image-to-Image Translation
This letter proposes to alleviate the problem of reliance on large labelled data by training deep learning models on datasets that are synthesised using image-to-image translation techniques and investigates different methods to perform this process optimally.
Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies
This survey provides a comprehensive review of adversarial models for image synthesis, and summarizes the synthetic image generation methods, and discusses the categories including image-to-image translation, fusion image generation, label- to-image mapping, and text-to -image translation.
Realistic endoscopic image generation method using virtual-to-real image-domain translation
A realistic image generation method for visualisation in endoscopic simulation systems is proposed in this study and improves the reality of the virtual endoscopic images using a virtual-to-real image-domain translation technique.
CaDIS: Cataract dataset for surgical RGB-image segmentation
A dataset for semantic segmentation of cataract surgery videos complementing the publicly available CATARACTS challenge dataset is introduced and the performance of several state-of-the-art deep learning models for semantic segmentsation is benchmarked on the presented dataset.


Deep monocular 3D reconstruction for assisted navigation in bronchoscopy
A decoupled deep learning architecture is proposed that projects input frames onto the domain of CT renderings, thus allowing offline training from patient-specific CT data, and shows a novel architecture to perform real-time monocular depth estimation without losing patient specificity in bronchoscopy.
Image-to-Image Translation with Conditional Adversarial Networks
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.
OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool
The OpenHELP phantom proved to be feasible and accurate, consecutively applied frequently in the field of computer-assisted surgery at the authors' institutions and is accessible as an open-source project at for the academic community.
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
A novel dual-GAN mechanism is developed, which enables image translators to be trained from two sets of unlabeled images from two domains, and can even achieve comparable or slightly better results than conditional GAN trained on fully labeled data.
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
This work presents an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples, and introduces a cycle consistency loss to push F(G(X)) ≈ X (and vice versa).
Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy
It is shown that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images, using a joint deep convolutional neural network‐conditional random field (CNN‐CRF) framework for monocular endoscopy depth estimation.
Elastic Mitral Valve Silicone Replica Made from 3D-Printable Molds Offer Advanced Surgical Training
Software is developed to automatically produce a 3D-printable casting mold for silicone material, as this mimics properties of the valve tissue in terms of stitching and cutting, and the usefulness of these models is evaluated by an experienced cardiac surgeon, successfully conducting reconstructive surgery on pathological silicone valves.
A Survey of Augmented Reality
  • Ronald T. Azuma
  • Computer Science
    Presence: Teleoperators & Virtual Environments
  • 1997
The characteristics of augmented reality systems are described, including a detailed discussion of the tradeoffs between optical and video blending approaches, and current efforts to overcome these problems are summarized.
A Taxonomy of Mixed Reality Visual Displays
Paul Milgram received the B.A.Sc. degree from the University of Toronto in 1970, the M.S.E.E. degree from the Technion (Israel) in 1973 and the Ph.D. degree from the University of Toronto in 1980.