SurgeonAssist-Net: Towards Context-Aware Head-Mounted Display-Based Augmented Reality for Surgical Guidance

  title={SurgeonAssist-Net: Towards Context-Aware Head-Mounted Display-Based Augmented Reality for Surgical Guidance},
  author={Michael J. Doughty and Karan Singh and Nilesh R. Ghugre},
We present SurgeonAssist-Net: a lightweight framework making action-and-workflow-driven virtual assistance, for a set of predefined surgical tasks, accessible to commercially available optical seethrough head-mounted displays (OST-HMDs). On a widely used benchmark dataset for laparoscopic surgical workflow, our implementation competes with state-of-the-art approaches in prediction accuracy for automated task recognition, and yet requires 7.4× fewer parameters, 10.2× fewer floating point… Expand

Figures and Tables from this paper


EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos
This paper proposes a novel method for phase recognition that uses a convolutional neural network (CNN) to automatically learn features from cholecystectomy videos and that relies uniquely on visual information. Expand
Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis
A novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both surgical tool presence detection and phase recognition. Expand
SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
Based on the phase transition-sensitive predictions from the SV-RCNet, a simple yet effective inference scheme, namely the prior knowledge inference (PKI), by leveraging the natural characteristic of surgical video is proposed, which improves the consistency of results and largely boosts the recognition performance. Expand
Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures
An overview of the development of integrated AR solutions in the context of minimally invasive surgery, integrating preoperative and intraoperative anatomic and functional data, improving visual perception in a mixed environment, and developing new user interaction paradigms for taking full advantage of the virtual data, while overlaid onto the real scene are provided. Expand
Real-Time Task Recognition in Cataract Surgery Videos Using Adaptive Spatiotemporal Polynomials
A new algorithm for recognizing surgical tasks in real-time in a video stream based on adaptive spatiotemporal polynomials is introduced, particularly suited to characterize deformable moving objects with fuzzy borders, which are typically found in surgical videos. Expand
Laparoscopy with augmented reality adaptations
A design and implementation of a multimodal interface to enhance the laparoscopic procedure and results show that surgeons consider that the prototype can improve surgeon-to-surgeon communication using head pose as a means of pointing and enhance hand-eye coordination. Expand
Surgeons blinded by enhanced navigation: the effect of augmented reality on attention
Using the detection rate of unexpected findings, whether inattentional blindness is significant in a surgical context and the impact of on-screen navigational cuing with augmented reality are assessed. Expand
Learning convolutional action primitives for fine-grained action recognition
The Latent Convolutional Skip Chain Conditional Random Field (LC-SC-CRF) time series model learns a set of interpretable and composable action primitives from sensor data and performs well without requiring hand-crafted features or intricate domain knowledge. Expand
Augmented reality in neurosurgery: a systematic review
Current literature confirms that AR is a reliable and versatile tool when performing minimally invasive approaches in a wide range of neurosurgical diseases, although prospective randomized studies are not yet available and technical improvements are needed. Expand
A Survey of Calibration Methods for Optical See-Through Head-Mounted Displays
This paper provides insights into the fundamentals of calibration techniques, and presents an overview of both manual and automatic approaches, as well as evaluation methods and metrics, which identifies opportunities for future research. Expand