Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

@inproceedings{Engelhardt2018ImprovingST,
  title={Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries},
  author={Sandy Engelhardt and Raffaele De Simone and Peter M. Full and Matthias Karck and Ivo Wolf},
  booktitle={MICCAI},
  year={2018}
}
Current `dry lab' surgical phantom simulators are a valuable tool for surgeons which allows them to improve their dexterity and skill with surgical instruments. These phantoms mimic the haptic and shape of organs of interest, but lack a realistic visual appearance. In this work, we present an innovative application in which representations learned from real intraoperative endoscopic sequences are transferred to a surgical phantom scenario. The term hyperrealism is introduced in this field… Expand
Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training
TLDR
A cross-domain conditional generative adversarial network approach (GAN) that aims to generate more consistent stereo pairs and shows substantial improvements in depth perception and realism evaluated. Expand
Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation
TLDR
A task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains and it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself. Expand
Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs
TLDR
This paper proposes to apply generative adversarial neural networks trained with a cycle consistency loss, or CycleGANs, to improve realism in ultrasound (US) simulation from computed tomography (CT) scans, and results can contribute to reduce the gap between artificially generated and real US scans. Expand
Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data
TLDR
A novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes is proposed, which produces consistent translations of arbitrary views and thus enables long-term consistent video synthesis. Expand
OfGAN: Realistic Rendition of Synthetic Colonoscopy Videos
TLDR
The Optical Flow Generative Adversarial Network (OfGAN) is developed to transform simulated colonoscopy videos into realistic ones while preserving annotation, and it is demonstrated that the performance of the OfGAN overwhelms the baseline method in relative tasks through both qualitative and quantitative evaluation. Expand
MedGAN: Medical Image Translation using GANs
TLDR
A new framework, named MedGAN, is proposed for medical image-to-image translation which operates on the image level in an end- to-end manner and outperforms other existing translation approaches. Expand
Robotic Instrument Segmentation With Image-to-Image Translation
TLDR
This letter proposes to alleviate the problem of reliance on large labelled data by training deep learning models on datasets that are synthesised using image-to-image translation techniques and investigates different methods to perform this process optimally. Expand
Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies
TLDR
This survey provides a comprehensive review of adversarial models for image synthesis, and summarizes the synthetic image generation methods, and discusses the categories including image-to-image translation, fusion image generation, label- to-image mapping, and text-to -image translation. Expand
Realistic endoscopic image generation method using virtual-to-real image-domain translation
TLDR
A realistic image generation method for visualisation in endoscopic simulation systems is proposed in this study and improves the reality of the virtual endoscopic images using a virtual-to-real image-domain translation technique. Expand
CaDIS: Cataract dataset for surgical RGB-image segmentation
TLDR
A dataset for semantic segmentation of cataract surgery videos complementing the publicly available CATARACTS challenge dataset is introduced and the performance of several state-of-the-art deep learning models for semantic segmentsation is benchmarked on the presented dataset. Expand
...
1
2
3
...

References

SHOWING 1-9 OF 9 REFERENCES
Deep monocular 3D reconstruction for assisted navigation in bronchoscopy
TLDR
A decoupled deep learning architecture is proposed that projects input frames onto the domain of CT renderings, thus allowing offline training from patient-specific CT data, and shows a novel architecture to perform real-time monocular depth estimation without losing patient specificity in bronchoscopy. Expand
Image-to-Image Translation with Conditional Adversarial Networks
TLDR
Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Expand
OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool
TLDR
The OpenHELP phantom proved to be feasible and accurate, consecutively applied frequently in the field of computer-assisted surgery at the authors' institutions and is accessible as an open-source project at www.open-cas.org for the academic community. Expand
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
TLDR
A novel dual-GAN mechanism is developed, which enables image translators to be trained from two sets of unlabeled images from two domains, and can even achieve comparable or slightly better results than conditional GAN trained on fully labeled data. Expand
Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks
TLDR
This work presents an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples, and introduces a cycle consistency loss to push F(G(X)) ≈ X (and vice versa). Expand
Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy
TLDR
It is shown that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images, using a joint deep convolutional neural network‐conditional random field (CNN‐CRF) framework for monocular endoscopy depth estimation. Expand
Elastic Mitral Valve Silicone Replica Made from 3D-Printable Molds Offer Advanced Surgical Training
TLDR
Software is developed to automatically produce a 3D-printable casting mold for silicone material, as this mimics properties of the valve tissue in terms of stitching and cutting, and the usefulness of these models is evaluated by an experienced cardiac surgeon, successfully conducting reconstructive surgery on pathological silicone valves. Expand
A Survey of Augmented Reality
  • Ronald T. Azuma
  • Computer Science
  • Presence: Teleoperators & Virtual Environments
  • 1997
TLDR
The characteristics of augmented reality systems are described, including a detailed discussion of the tradeoffs between optical and video blending approaches, and current efforts to overcome these problems are summarized. Expand
A Taxonomy of Mixed Reality Visual Displays
Paul Milgram received the B.A.Sc. degree from the University of Toronto in 1970, the M.S.E.E. degree from the Technion (Israel) in 1973 and the Ph.D. degree from the University of Toronto in 1980.Expand