Augmented reality environment with virtual fixtures for robotic telemanipulation in space

  title={Augmented reality environment with virtual fixtures for robotic telemanipulation in space},
  author={Tian Xia and Simon L{\'e}onard and Anton Deguet and Louis L. Whitcomb and Peter Kazanzides},
  journal={2012 IEEE/RSJ International Conference on Intelligent Robots and Systems},
  • Tian Xia, S. Léonard, P. Kazanzides
  • Published 24 December 2012
  • Computer Science
  • 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems
This paper presents an augmented reality framework, implemented on the master console of a modified da Vinci® surgical robot, that enables the operator to design and implement assistive virtual fixtures during teleoperation. Our specific goal is to facilitate teleoperation with large time delays, such as the delay of several seconds that occurs with ground-based control of robotic systems in earth orbit. The virtual fixtures give immediate visual feedback and motion guidance to the operator… 

Figures and Tables from this paper

Model-based telerobotic control with virtual fixtures for satellite servicing tasks

A new delay tolerant control methodology is proposed, using virtual fixtures, hybrid position/force control, task frame formalism, and environment modeling, that is robust to modeling and registration errors.

Augmented virtuality for model-based teleoperation

An augmented virtuality interface for teleoperation is presented, which can solve problems of unintuitive teleoperation and limited camera views by projecting real camera images onto a registered 3D model of the environment and allowing the operator to select any desired viewpoint.

Scene Modeling and Augmented Virtuality Interface for Telerobotic Satellite Servicing

Experiments on a ground-based telerobotic platform indicate that the proposed method leads to better teleoperation performance with 30% better blade alignment and 50% reduction in task execution time compared to the baseline case where visualization is restricted to the available camera views.

Effects of Augmented Reality on the Performance of Teleoperated Industrial Assembly Tasks in a Robotic Embodiment

The effects of augmented reality (AR) are evaluated as a tool to deliver additional information, which helps in overcoming the differences in perception between telepresence and actual presence, and the sense of presence and embodiment for the participant is positively affected by different types of AR.

Model-Augmented Haptic Telemanipulation: Concept, Retrospective Overview, and Current Use Cases

The model-augmented haptic telemanipulation (MATM) approach is discussed in detail, the latest research results of the various technologies encompassed within this approach are presented, and the broad application potential of MATM is demonstrated based on the aforementioned use cases.

Augmented reality-aided tele-presence system for robot manipulation in industrial manufacturing

This work investigates the use of a highly immersive telepresence system for industrial robotics where a remote robot is controlled through operator's movements and muscle contractions captured with a wearable device where a Robot Operating System integrated framework is presented.

A Framework for Interactive Virtual Fixture Generation for Shared Teleoperation in Unstructured Environments

  • Vitalii PruksJ. Ryu
  • Computer Science
    2020 IEEE International Conference on Robotics and Automation (ICRA)
  • 2020
This work introduces a framework for the interactive generation of VF, based on the observation that a human can easily understand just by looking at the remote environment which VF could help in task execution, and proposes a user interface that detects features on camera images and permits interactive selection of the features.

Human-Machine Collaborative Telerobotics: Computer Assistance for Manually Controlled Telesurgery and Teleoperation

New contributions in telerobotics are presented that show the way to improved access and control and a new mixed autonomy method is presented based on a division of the workspace into regions that are either safe for computer assisted control, or require pure human control.

Preliminary study of virtual nonholonomic constraints for time-delayed teleoperation

A nonholonomic virtual fixture (NHVF) that adheres to the VNHC to further reduce the potential for operators to input mistaken commands is described, which was found to decrease performance slightly, though user feedback indicated that a differently implemented virtual fixture and controller may improve performance.

Point cloud augmented virtual reality environment with haptic constraints for teleoperation

A novel methodology for haptic constraints in a point cloud augmented virtual reality environment is proposed to address this human operation limitation and can enhance human operation efficiency and reduce time costs by at least 59% compared with no-haptic-constraint operations.



The use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation.

Abstract : This report reviews the notion of virtual fixtures for force-reflecting telepresence systems that has been introduced in a previous report and describes an additional study which

Constrained control for surgical assistant robots

  • A. KapoorMing LiR. Taylor
  • Medicine
    Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006.
  • 2006
The approach uses a weighted, multi-objective (both linear and nonlinear) constrained optimization framework to formalize a library of virtual fixtures for task primitives and a way to assemble multiple virtual fixture objects.

Teleprogramming: Toward Delay-Invariant Remote Manipulation

The overallteleprogramming control concept is introduced, its main components are described, and the preliminary results using the experimental teleprogramming system are reported on.

Constrained Cartesian motion control for teleoperated surgical robots

A method of formulating desired motions as sets of task goals in any number of coordinate frames (task frames) relevant to the task, optionally subject to additional linear constraints in each of the task frames is presented.

Spatial Motion Constraints Using Virtual Fixtures Generated by Anatomy

The experimental results show that a robotic assistant employing a spatial-motion-constraints-generation approach on spatial motion constraints can assist the user in skilled manipulation tasks, while maintaining desired properties.

Teleprogramming: overcoming communication delays in remote manipulation

  • J. FundaR. Paul
  • Computer Science
    1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings
  • 1990
A novel combination of computer graphics and manipulator programming is proposed to solve the problem of teleoperation in the presence of feedback delays by allowing the operator to kinesthetically interact with a graphical simulation of the remote environment and to interactively teleprogram the remote manipulator through a sequence of elementary symbolic instructions generated automatically by the operator's station software.

Analysis of virtual fixture contact stability for telemanipulation

  • J. AbbottA. Okamura
  • Computer Science
    Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453)
  • 2003
This paper considers the problem of unstable vibrations of the slave against the FRVF for a general class of telemanipulator control architectures, and finds that the discrete state-space method agrees with simulations, and can be easily used to design and analyze the stability and transient behavior of the tele manipulator.

Model-mediated Telemanipulation

This paper presents a user-centered, model-mediated approach to bilateral telemanipulation under large communication delays that mitigates user perception difficulties and allows stable motion and force interactions with the remote environment.

Teleprogramming for subsea teleoperation using acoustic communication

It is argued that the low bandwidth and high delay imposed by acoustic modems makes it unwise to adopt conventional teleoperation techniques and a system is presented which permits subsea teleoperative tasks to be carried out using such limited communication resources.

Asymptotic Stability for Force Reflecting Teleoperators with Time Delay

A bilateral system consists of a local master manipulator and a remotely located slave manipulator. Velocity commands are sent forward from the master to the slave, and force information is "re