DronePaint: Swarm Light Painting with DNN-based Gesture Recognition

  title={DronePaint: Swarm Light Painting with DNN-based Gesture Recognition},
  author={Valerii Serpiva and Ekaterina Karmanova and Aleksey Fedoseev and Stepan Perminov and Dzmitry Tsetserukou},
  journal={ACM SIGGRAPH 2021 Emerging Technologies},
We propose a novel human-swarm interaction system, allowing the user to directly control a swarm of drones in a complex environment through trajectory drawing with a hand gesture interface based on the DNN-based gesture recognition. The developed CV-based system allows the user to control the swarm behavior without additional devices through human gestures and motions in real-time, providing convenient tools to change the swarm’s shape and formation. The two types of interaction were proposed… Expand

Figures and Tables from this paper


Gesture based Human-Swarm Interactions for Formation Control using interpreters
A novel interpreter system, which acts as an intermediary between the user and the swarm to simplify the user's role in the interaction, and employs machine learning, Kalman filtering and optimal control techniques to translate the user input into swarm control parameters. Expand
Gesture based human - Multi-robot swarm interaction and its application to an interactive display
A taxonomy for gesture-based interaction between a human and a group (swarm) of robots is described and a depth sensor is used to recognize human gesture, determining the commands sent to a group comprising tens of robots. Expand
SwarmTouch: Guiding a Swarm of Micro-Quadrotors With Impedance Control Using a Wearable Tactile Interface
A novel interaction strategy for a human–swarm communication, which combines impedance control and vibrotactile feedback is proposed, which takes into account the human hand velocity and changes the formation shape and dynamics accordingly using impedance interlinks simulated between quadrotors, which helps to achieve a natural swarm behavior. Expand
DroneGraffiti: autonomous multi-UAV spray painting
A multi-drone graffiti dispatch system tested on up to three actual aerial robots operating simultaneously in real-time proved to be robust and effective and can be transferred to commercial applications that need precise aerial vehicle operation, such as commercial painting, cleaning, and non-destructive control. Expand
PaintCopter: An Autonomous UAV for Spray Painting on Three-Dimensional Surfaces
Experimental results demonstrate autonomous spray painting by the UAV, doing area fill and versatile line painting on a 3D surface. Expand
Implementation of Hand Gesture Recognition Device Applicable to Smart Watch Based on Flexible Epidermal Tactile Sensor Array
This paper presents a new gesture recognition method using a Flexible Epidermal Tactile Sensor based on strain gauges to sense deformation, which significantly outperformed existing methods. Expand
User-defined Swarm Robot Control
This work presents a user-defined set of control interactions for tabletop swarm robots derived from an elicitation study, and investigates the effects of number of robots and proximity on the user's interaction and found significant effects. Expand
Quadcopter-Projected In-Situ Navigation Cues for Improved Location Awareness
Through a user study with 16 participants, it is shown that using in-situ instructions for navigation leads to a significantly higher ability to observe real-world points of interest. Expand
Sparked: a live interaction between humans and quadcopters
Cirque du Soleil, ETH Zürich, and Verity Studios partnered to develop a short film featuring 10 quadcopters in a flying performance where humans and drones move in sync. Expand
The Association for Computing Machinery
As the Association for Computing Machinery enters a new phase of its existence, it seems befitting to review, briefly, the conditions in the computing field just prior to its organization and theExpand