A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks

@article{Mayer2006ASF,
  title={A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks},
  author={Hermann Georg Mayer and Faustino J. Gomez and Daan Wierstra and Istv{\'a}n Nagy and Alois Knoll and J{\"u}rgen Schmidhuber},
  journal={2006 IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year={2006},
  pages={543-548}
}
Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between… 

Figures from this paper

A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks
TLDR
Results obtained using long short-term memory RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control.
A Novel Method for Knot-Tying in Autonomous Robotic assisted Surgery Using Deep Learning
TLDR
A novel method using deep learning based on Variational Autoencoders for robotic assistants to learn knot-tying with the Data Set of manual operation, instead of learning from video demonstrations is proposed.
Dynamic trajectory planning for robotic knot tying
  • Bo Lu, H. Chu, Li Cheng
  • Medicine
    2016 IEEE International Conference on Real-time Computing and Robotics (RCAR)
  • 2016
TLDR
A new knot-tying method is proposed to enhance the quality of robotic Knot tying practice with low supervision and results confirm that suture loops can be successfully winded around the instrument without suture slippage.
A Preliminary Investigation Into the Feasibility of Semi-Autonomous Surgical Path Planning for a Mastoidectomy Using LSTM-Recurrent Neural Networks
TLDR
Almost captured surgical trajectories from trained surgeons were used to train a set of artificial neural networks (ANNs) using recorded surgeons' motions when manipulating a surgical instrument during procedure training using a surgery simulator, and it was found that even when trained on a small number of datasets, the ANNs converged and could generate output trajectories that were still assessed to be safe even when slight changes in the fiducial marker placement locations were given.
Human-Machine Collaborative surgery using learned models
  • N. Padoy, Gregory Hager
  • Medicine, Computer Science
    2011 IEEE International Conference on Robotics and Automation
  • 2011
TLDR
A novel surgical Human-Machine Collaborative system in which portions of a surgical task are performed autonomously under complete surgeon's control, and other portions manually, provides a safe and acceptable solution for surgical performance enhancement.
Surgical Automation for Multilateral Multi-Throw Suturing
TLDR
A novel mechanical needle guide and a framework for optimizing needle size, trajectory, and control parameters using sequential convex programming are presented and initial results suggest that the dVRK can perform suturing at 30% of human speed while completing 86% suture throws attempted.
Trajectory Planning of Knot-Tying Manipulation in Surgery
TLDR
To improve the surgical knot quality and accelerate healing for patients in surgery, trajectory planning of suture clamping point in knot-tying manipulation using a master-slave robot is discussed based on the surgeon’s manual motion of knot-TYing.
A CNN-based prototype method of unstructured surgical state perception and navigation for an endovascular surgery robot
TLDR
A novel convolutional neural network (CNN)-based framework is proposed to address challenges for navigation of an ES robot based on surgeons’ skill learning, and shows its capability of adapting to different situations and achieves similar success rate and average operating time.
Motion generation of robotic surgical tasks: Learning from expert demonstrations
TLDR
This paper proposes using programming by demonstration to build generative models and generate smooth trajectories that capture the underlying structure of the motion data recorded from expert demonstrations to generate smoother trajectories for reproduction of three common medical tasks.
Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization
TLDR
Initial results suggest that the dVRK can perform suturing at 30% of human speed while completing 86% suture throws attempted, and the Suture Needle Angular Positioner (SNAP) results in a 3x error reduction in the needle pose estimate in comparison with the standard actuator.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 28 REFERENCES
The Endo[PA]R system for minimally invasive robotic surgery
TLDR
This research project tackles haptic feedback and prolonged operation time in minimally invasive robotic surgery by adding high fidelity force-feedback and automating recurrent manipulation tasks into the Endo[PA]R (endoscopic partially-autonomous robot) system, an open evaluation platform for robotic surgery.
Robotic surgical instruments for dexterity enhancement in thoracoscopic coronary artery bypass graft.
TLDR
The use of robotic enhancement technology leads to an efficient performance of sutured coronary artery bypass anastomoses in a plastic model that enhances dexterity, precision, and reduces surgeon's fatigue while preserving the quality of hand suturing.
ROBOTIC ASSISTED SUTURING IN MINIMALLY INVASIVE SURGERY
TLDR
The focus of the research is the development of the EndoBot controller that is capable of a range of MIS operations including manual, shared, and supervised autonomous operations and particular emphasis is placed on its automation.
Manual vs robotically assisted laparoscopic surgery in the performance of basic manipulation and suturing tasks.
TLDR
Laroscopic maneuvering and suturing is faster and just as precise when performed manually as when performed with the prototype robotic system, inversely proportional to the size of the suture.
Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Learning
TLDR
A general framework for sequence learning, EVOlution of recurrent systems with LINear outputs (Evolino), which uses evolution to discover good RNN hidden node weights, while using methods such as linear regression or quadratic programming to compute optimal linear mappings from hidden state to output.
Solving Non-Markovian Control Tasks with Neuro-Evolution
TLDR
This article demonstrates a neuroevolution system, Enforced Sub-populations (ESP), that is used to evolve a controller for the standard double pole task and a much harder, non-Markovian version, and introduces an incremental method that evolves on a sequence of tasks, and utilizes a local search technique (Delta-Coding) to sustain diversity.
Endo[PA]R: An Open Evaluation System for Minimally Invasive Robotic Surgery
TLDR
EndoPAR (Endoscopic Partial-Autonomous Robot) is an open experimental platform for robot assisted minimally invasive heart surgery with haptic feedback that was evaluated by surgeons and it was capable of performing automated knot-tying.
Efficient Reinforcement Learning through Symbiotic Evolution
TLDR
A new reinforcement learning method called SANE (Symbiotic, Adaptive Neuro-Evolution), which evolves a population of neurons through genetic algorithms to form a neural network capable of performing a task, is presented.
Active Guidance for a Finless Rocket Using Neuroevolution
TLDR
Results show that the evolved active guidance controller can greatly increase the final altitude of the rocket, and that ESP can be an effective method for solving real-world, non-linear control tasks.
Learning to Forget: Continual Prediction with LSTM
TLDR
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
...
1
2
3
...