“Did You Hear That?” Learning to Play Video Games from Audio Cues

  title={“Did You Hear That?” Learning to Play Video Games from Audio Cues},
  author={Raluca D. Gaina and Matthew Stephenson},
  journal={2019 IEEE Conference on Games (CoG)},
Game-playing AI research has focused for a long time on learning to play video games from visual input or symbolic information. However, humans benefit from a wider array of sensors which we utilise in order to navigate the world around us. In particular, sounds and music are key to how many of us perceive the world and influence the decisions we make. In this paper, we present initial experiments on game-playing agents learning to play video games solely from audio cues. We expand the Video… 

Figures and Tables from this paper

Agents that Listen: High-Throughput Reinforcement Learning with Multiple Sensory Systems

A new version of VizDoom simulator is introduced to create a highly efficient learning environment that provides raw audio observations and studies the performance of different model architectures in a series of tasks that require the agent to recognize sounds and execute instructions given in natural language.

A Deep Reinforcement Learning Blind AI in DareFightingICE

This paper presents a deep reinforcement learning agent (AI) that uses sound as the input on the DareFightingICE platform and proposes different approaches to process audio data and uses the Proximal Policy optimization algorithm for this blind AI.

DareFightingICE Competition: A Fighting Game Sound Design and AI Competition

This paper presents a new competition-at the 2022 IEEE Conference on Games (CoG)- called DareFightingICE Competition. The competition has two tracks: a sound design track and an AI track. The game

The Aural-Visual Rhetoric in Video Game Tutorials

ABSTRACT This article asserts that auditory cues can be categorized by rhetorical function into the categories of visual rhetoric, defined by Amare and Manning under Peirce’s Ten Classes of Sign,

Who is Real Bob? Adversarial Attacks on Speaker Recognition Systems

This paper conducts the first comprehensive and systematic study of the adversarial attacks on SR systems (SRSs) to understand their security weakness in the practical black-box setting, and proposes an adversarial attack, named FakeBob, to craft adversarial samples.



General Video Game AI: A Multitrack Framework for Evaluating Agents, Games, and Content Generation Algorithms

This survey paper presents the VGDL, the GVGAI framework, existing tracks, and reviews the wide use of GVgaI framework in research, education, and competitions five years after its birth.

The Effects of Musical Experience and Hearing Loss on Solving an Audio-Based Gaming Task

We conducted an experiment using a purposefully designed audio-based game called the Music Puzzle with Japanese university students with different levels of hearing acuity and experience with music

Game Sound Technology and Player Interaction: Concepts and Developments

This chapter treats computer game playing as an affective activity, largely guided by the audio-visual aesthetics of game content (of which, here, we concentrate on the role of sound) and the

New musical organology : the audio-games

The characteristics and the limits of this field are defined and a few of the historical milestones highlighted (abstract cinema, gaming theory in music, graphic actions and scores), and areas for further research to devise analytical tools for these new objects are proposed.

The Influence of Background Music of Video Games on Immersion

The background music has been proved to be an indispensable part in every successful video game. At the same time, all the video game designers and players hope video games could be as immersive as

TagATune: A Game for Music and Sound Annotation

The rationale, design and preliminary results from a pilot study using a prototype of TagATune to label a subset of the FreeSound database, which aims to extract descriptions of sounds and music from human players.

Listening to Fear: A Study of Sound in Horror Computer Games

Computer game sound is as crucial to the creation of the depicted gameworld’s mood as it is in its undeniable support to gameplay. In horror computer games, this role is increased tenfold as sound

Game accessibility: a survey

The current state-of-the-art in research and practice in the accessibility of video games and points out relevant areas for future research are surveyed.

Towards generalized accessibility of video games for the visually impaired

This dissertation, and the identification of a generalized game interaction model, helps provide an understanding of how video games can be designed and modified to improve their accessibility features, and develops the first mainstream game using haptic feedback.

Orchestrating Game Generation

The different creative facets of games are identified, how orchestration can be facilitated in a top-down or bottom-up fashion is proposed, indicative preliminary examples of orchestration are reviewed, and the open questions and challenges ahead are discussed.