Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction

  title={Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction},
  author={Paul Schydlo and Mirko Rakovi{\'c} and Lorenzo Jamone and Jos{\'e} Santos-Victor},
  journal={2018 IEEE International Conference on Robotics and Automation (ICRA)},
Close human-robot cooperation is a key enabler for new developments in advanced manufacturing and assistive applications. Close cooperation require robots that can predict human actions and intent, understanding human non-verbal cues. Recent approaches based on neural networks have led to encouraging results in the human action prediction problem both in continuous and discrete spaces. Our approach extends the research in this direction. Our contributions are three-fold. First, we validate the… 

Ensemble of LSTMs and feature selection for human action prediction

An ensemble of long-short term memory (LSTM) networks are proposed to use for human action prediction and it is suggested that the LSTM model slightly outperforms the gaze baseline in single object picking accuracy, but achieves better accuracy in macro object prediction.

Biologically Inspired Controller of Human Action Behaviour for a Humanoid Robot in a Dyadic Scenario

It is shown that it is possible to model the complete upper body human behaviour during placing and giving interactions, and the generated trajectories from the model give a better approximation of the human-like behaviour in a humanoid robot than the existing inverse kinematics solver.

Action Anticipation: Reading the Intentions of Humans and Robots

The results show that it is possible to model (nonverbal) signals exchanged by humans during interaction, and how to incorporate such a mechanism in robotic systems with the twin goal of being able to “read” human action intentions and acting in a way that is legible by humans.

Action-Agnostic Human Pose Forecasting

The triangular-prism recurrent neural network (TP-RNN) models the hierarchical and multi-scale characteristics of human dynamics and captures the latent hierarchical structure in human pose sequences by encoding temporal dependencies with different time-scales.

FMG- and RNN-Based Estimation of Motor Intention of Upper-Limb Motion in Human-Robot Collaboration

This study demonstrates the potential of FMG-based wearable technologies to enhance human-robot collaboration in industrial settings by employing machine learning algorithms to interpret biosignals acquired from the human user and accordingly planning the robot reaction.

Safe Human Robot Cooperation in Task Performed on the Shared Load

Force myography data are used to develop a control scheme for the robot such that it can work with the human worker while avoiding collisions and show that the proposed scheme can successfully plan the robot motion based on the actions of the human user.

Predicting Human Errors from Gaze and Cursor Movements

This work employs a data-driven approach to predict human errors in a special divided attention task involving timing constraints and requiring focused concentration and frequent shifts of attention, and investigates the trade-off between evaluation metrics and anticipation window.

A Multi-Task Framework for Action Prediction

A novel multi-task deep forest framework, which treats temporal progress analysis as a relevant task to action prediction and takes advantage of observation ratio labels of incomplete videos during training, which is a cascade structure of random forests and multi- task random forests.

Bidirectional Communication Control for Human-Robot Collaboration

A novel communication control architecture is proposed that enables human and robot to set up a bidirectional communication about the task to execute and the human to take an informed decision on the behavior of the robot.



Fast target prediction of human reaching motion for cooperative human-robot manipulation tasks using time series classification

A data-driven approach that synthesizes anticipatory knowledge of both human motions and subsequent action steps in order to predict in real-time the intended target of a human performing a reaching motion is presented.

Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture

A sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams and shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision and recall.

Cost-Based Anticipatory Action Selection for Human–Robot Fluency

This work describes a model for human-robot joint action, and proposes an adaptive action selection mechanism for a robotic teammate, which makes anticipatory decisions based on the confidence of their validity and their relative risk.

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

This work represents each possible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations through object affordances and represents each ATCRF as a particle and represents the distribution over the potential futures using a set of particles.

Robot anticipation of human intentions through continuous gesture recognition

A method to recognize human body movements is proposed and it is combined with the contextual knowledge of human-robot collaboration scenarios provided by an object affordances framework that associates actions with its effects and the objects involved in them.

On Human Motion Prediction Using Recurrent Neural Networks

It is shown that, surprisingly, state of the art performance can be achieved by a simple baseline that does not attempt to model motion at all, and a simple and scalable RNN architecture is proposed that obtains state-of-the-art performance on human motion prediction.

Cooperative human robot interaction systems: IV. Communication of shared plans with Naïve humans using gaze and speech

Results support the cooperative stance in human social cognition, and suggest that cooperative robots should employ joint plans, fully communicate them in order to sustain effective collaboration while being ready to adapt if the human makes a midstream mistake.

Towards a Platform-Independent Cooperative Human Robot Interaction System: III An Architecture for Learning and Executing Actions and Shared Plans

A cooperative human-robot interaction system that has been specifically developed for portability between different humanoid platforms, by abstraction layers at the perceptual and motor interfaces is presented.

Anticipatory robot control for efficient human-robot collaboration

An anticipatory control method is presented that enables robots to proactively perform task actions based on anticipated actions of their human partners, and is implemented into a robot system that monitored its user's gaze, predicted his or her task intent based on observed gaze patterns, and performed anticipatory task actions according to its predictions.

Anticipation and initiative in human-humanoid interaction

Results from human-robot cooperation experiments in the context of a cooperative assembly task are presented and it is demonstrated how these progressive levels render the cooperative interaction more fluid and more rapid.