A Self-Organizing Gesture Map for a Voice-Controlled Instrument Interface

  title={A Self-Organizing Gesture Map for a Voice-Controlled Instrument Interface},
  author={Stefano Fasciani and Lonce L. Wyse},
Mapping gestures to digital musical instrument parameters is not trivial when the dimensionality of the sensor-captured data is high and the model relating the gesture to sensor data is unknown. In these cases, a front-end processing system for extracting gestural information embedded in the sensor data is essential. In this paper we propose an unsupervised offline method that learns how to reduce and map the gestural data to a generic instrument parameter control space. We make an… 

Figures and Tables from this paper

Vocal Control of Sound Synthesis Personalized by Unsupervised Machine Listening and Learning
A user-driven adaptive method to control the sonic response of digital musical instruments using information extracted from the timbre of the human voice is described using a minimum-loss mapping that hides any synthesizer-specific parameters and maps the vocal interaction directly to perceptual characteristics of the generated sound.
The Machine Learning Algorithm as Creative Musical Tool
It is shown that, like other interfaces, learning algorithms can be characterised by the ways their affordances intersect with goals of human users, and it is argued that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways.
Design and Evaluation of a Gesture Controlled Singing Voice Installation
A media installation that allows users to perform an aria without real singing and provide the look and feel from a 20th century performance of an opera singer is developed.


Browsing Music and sound using gestures in a Self-Organized 3D Space
This work proposes a new system for browsing music in augmented reality by utilizing sensors capable of sensing motion in 3-dimensions and the use of audio feature extraction and self-organizing maps (SOMs).
A Voice Interface for Sound Generators: adaptive and automatic mapping of gestures to sound
This paper proposes the exploitation of vocal gesture as an extension or alternative to traditional physical controllers, which uses dynamic aspects of vocal sound to control variations in the timbre of the synthesized sound.
Making music through real-time voice timbre analysis: machine learning and timbral control
This thesis develops approaches that can be used with a wide variety of musical instruments by applying machine learning techniques to automatically derive the mappings between expressive audio input and control output, with a focus on timbral control.
Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance
This paper presents ongoing work on gesture mapping strategies and applications to sound synthesis by signal models controlled via a standard MIDI wind controller, considering different mapping strategies in order to achieve "fine" control of additive synthesis by coupling originally independent outputs from the wind controller.
Adapting General Purpose Interfaces to synthesis Engines using Unsupervised Dimensionality Reduction Techniques and inverse Mapping from Features to parameters
Adapt techniques for mapping generic user interfaces to synthesis engines guarantee a linear relationship between control signals and perceptual features, and at the same time, reduces the control space dimensionality maintaining the maximum explorability of the sonic space.
Towards a Model for Instrumental Mapping in Expert Musical Interaction
This paper reviews models of the ways in which performer instrumental actions can be linked to sound synthesis parameters and discusses a general model for instrumental mapping, by separating the mapping layer into two independent parts.
This paper proposes to use the voice as a source of realvalued and time-continuous control signals that can be employed to interact with any multidimensional human-controllable device in real-time.
Mapping Strategies for Musical Performance
An explorative mode of interaction is defined, termed Performance Mode, which is more suitable to real-time control than the many "choice-based" approaches and is considered for the designers of realtime musical interfaces.
Gestural control of sound synthesis
This paper provides a review of gestural control of sound synthesis in the context of the design and evaluation of digital musical instruments and an application of this research to the control of digital audio effects.
Real-time human interaction with supervised learning algorithms for music composition and performance
This thesis presents a general-purpose software system for applying standard supervised learning algorithms in music and other real-time problem domains, called the Wekinator, which supports human interaction throughout the entire supervised learning process, including the generation of training examples and the application of trained models to real- time inputs.