Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot

  title={Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot},
  author={Minho Hwang and Daniel Seita and Brijen Thananjeyan and Jeffrey Ichnowski and Samuel Paradis and Danyal Fer and Thomas Low and Ken Goldberg},
  journal={2020 International Symposium on Medical Robotics (ISMR)},
Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera. In this study, we explore the potential of depth sensing for efficient and reliable automation of surgical subtasks. We consider a monochrome (all red) version of the peg transfer task from the Fundamentals of Laparoscopic Surgery training suite implemented with the da Vinci Research Kit (dVRK). We use calibration… 
Superhuman Surgical Peg Transfer Using Depth-Sensing and Deep Recurrent Neural Networks
This is the first fully automated system to achieve “superhuman” performance in both speed and success on peg transfer, and an approach that leverages emerging innovations in depth sensing, deep learning, and Peiper's method for computing inverse kinematics with time-minimized joint motion is presented.
Efficiently Calibrating Cable-Driven Surgical Robots With RGBD Sensing, Temporal Windowing, and Linear and Recurrent Neural Network Compensation
A novel approach to efficiently calibrate a dVRK by placing a 3D printed fiducial coordinate frame on the arm and end-effector that is tracked using RGBD sensing is proposed and it is suggested that the resulting model can reduce the mean tracking error of the physical robot.
Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles
This work proposes HOUSTON: Handoff of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that uses a learned active sensing policy with a stereo camera to localize and align the needle into a visible and accessible pose for the other arm to study handover of unmodified surgical needles.
Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation
Results on a da Vinci Research Kit suggest that combining the coarse policy with half a second of corrections from the learned policy during each high-precision segment improves the success rate on the Fundamentals of Laparoscopic Surgery peg transfer task.
Analysis of Laser Sensors and Camera Vision in the Shoe Position Inspection System
The main aim of this work is to conduct analyses of sensory systems for inspection systems and their possibilities for further work mainly based on the resolution and quality of obtained data, including dependency on scanning complex surfaces and the achieved resolution of scanned surfaces.
ActiveZero: Mixed Domain Learning for Active Stereovision with Zero Annotation
  • Isabella Liu, Edward Yang, +5 authors Hao Su
  • Computer Science
  • 2021
A new framework, ActiveZero, is presented, which is a mixed domain learning solution for active stereovision systems that requires no real world depth annotation and introduces a novel selfsupervised loss called temporal IR reprojection to increase the robustness and accuracy of the reprojections in hardto-perceive regions.
SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning
  • Jiaqi Xu, Bin Li, Bo Lu, Yunhui Liu, Q. Dou, P. Heng
  • Computer Science, Engineering
    2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
  • 2021
The designed SurRoL integrates a user-friendly RL library for algorithm development and a real-time physics engine, which is able to support more PSM/ECM scenarios and more realistic physical interactions, and achieves better transferability in the real world.
Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments
A new learning based end-to-end depth prediction network which takes noisy raw I-ToF signals as well as an RGB image and fuses their latent representation based on a multi step approach involving both implicit and explicit alignment to predict a high quality long range depth map aligned to the RGB viewpoint is proposed.
Efficiently Calibrating Cable-Driven Surgical Robots With RGBD Fiducial Sensing and Recurrent Neural Networks
A novel approach to efficiently calibrate robotic surgical assistants by placing a 3D printed fiducial coordinate frames on the arm and end-effector that is tracked using RGBD sensing and considering 13 approaches to modeling to measure the coupling and history-dependent effects between joints.
Autonomous task planning and situation awareness in robotic surgery*
A framework to implement surgical task automation consisting of a task-level reasoning module based on answer set programming, a low-level motion planning moduleBased on dynamic movement primitives, and a situation awareness module that generates explainable plans and recovers from failure conditions.


Learning 2D Surgical Camera Motion From Demonstrations
A probabilistic model that identifies image features that “dwell” close to the camera's focal point in expert demonstrations is proposed, and the extent to which the system correctly segments candidate debridement targets (box accuracy) and correctly ranks those targets (rank accuracy) is evaluated.
Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure
A novel two-phase coarse-to-fine calibration method for debridement of raisins and pumpkin seeds as fragment phantoms and finds that without calibration, position errors average 4.55mm and the combination of Phase I and Phase II can reduce average error to 1.08mm.
Transferring Know-How for an Autonomous Camera Robotic Assistant
Robotic platforms are taking their place in the operating room because they provide more stability and accuracy during surgery. Although most of these platforms are teleoperated, a lot of research is
SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics
This letter proposes a novel surgical perception framework, SuPer, for surgical robotic control that successfully completed soft tissue manipulation tasks with high accuracy and was implemented on the da Vinci Surgical System in real-time with an end-effector controller.
Automated Extraction of Surgical Needles from Tissue Phantoms
We consider the surgical subtask of automated extraction of embedded suturing needles from silicone phantoms and propose a four-step algorithm consisting of calibration, needle segmentation, grasp
Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization
Initial results suggest that the dVRK can perform suturing at 30% of human speed while completing 86% suture throws attempted, and the Suture Needle Angular Positioner (SNAP) results in a 3x error reduction in the needle pose estimate in comparison with the standard actuator.
Automated Pick-Up of Suturing Needles for Robotic Surgical Assistance
This paper presents a vision-guided needle grasping method for automatically grasping the needle that has been inserted into the patient prior to anastomosis, and shows examples of successful autonomous grasping that has the potential to simplify and decrease the operational time in RALP by assisting a small component of urethrovesical anASTomosis.
An interchangeable surgical instrument system with application to supervised automation of multilateral tumor resection
Experiments suggest that an Intuitive Surgical system using these interchangeable instruments can perform a multi-step tumor resection procedure that uses a novel haptic probe to localize the tumor, standard scalpel to expose the tumors, standard grippers to extract the subcutaneous tumor, and a fluid injection tool to seal the wound.
Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms
Using the da Vinci Research Kit (DVRK) robotic surgical assistant, this work explores a “Learning By Observation” (LBO) approach where the robot identifies, segment, and parameterize motion sequences and sensor conditions to build a finite state machine (FSM) for each subtask.
Autonomous Laparoscopic Robotic Suturing with a Novel Actuated Suturing Tool and 3D Endoscope
  • H. Saeidi, H. Le, +5 authors A. Krieger
  • Computer Science, Medicine
    2019 International Conference on Robotics and Automation (ICRA)
  • 2019
The authors' test results indicate that STAR can reach 2.9 times better consistency in suture spacing compared to manual method and also eliminate suture repositioning and adjustments, and the consistency of suture bite sizes obtained by STAR matches with those obtained by manual suturing.