Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation
@article{Tang2020LearningAS, title={Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation}, author={Zhenyu Tang and Hsien-Yu Meng and Dinesh Manocha}, journal={2021 IEEE Virtual Reality and 3D User Interfaces (VR)}, year={2020}, pages={835-844} }
We present a novel hybrid sound propagation algorithm for interactive applications. Our approach is designed for dynamic scenes and uses a neural network-based learned scattered field representation along with ray tracing to generate specular, diffuse, diffraction, and occlusion effects efficiently. We use geometric deep learning to approximate the acoustic scattering field using spherical harmonics. We use a large 3D dataset for training, and compare its accuracy with the ground truth…
Figures and Tables from this paper
7 Citations
Acoustic Rendering Based on Geometry Reduction and Acoustic Material Classification
- Physics2022 IEEE Conference on Games (CoG)
- 2022
We present work in progress on a pipeline for audio rendering integrating vision-based systems for acoustic material classification. With a marching cubes algorithm, the pipeline estimates a cuboid…
DeepEigen: Learning-based Modal Sound Synthesis with Acoustic Transfer Maps
- Computer Science, PhysicsArXiv
- 2021
This work combines two network-based solutions to formulate a complete learning-based 3D modal sound model that includes a 3D sparse convolution network as the eigendecomposition solver and an encoder-decoder network for the prediction of the Far-Field Acoustic Transfer maps.
GWA: A Large High-Quality Acoustic Dataset for Audio Processing
- Computer ScienceSIGGRAPH
- 2022
The Geometric-Wave Acoustic (GWA) dataset is presented, a large-scale audio dataset of about 2 million synthetic room impulse responses (IRs) and their corresponding detailed geometric and simulation configurations that is the first data with accurate wave acoustic simulations in complex scenes.
Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse
- Computer Science2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA)
- 2021
This work proposes a practical approach for designing parametric 6-degree-of-freedom object-based interactive audio engines to deliver the perceptually relevant binaural cues necessary for audio/visual and virtual/real congruence in Metaverse experiences.
Improving aircraft performance using machine learning: a review
- Computer ScienceArXiv
- 2022
The basic concepts and the most relevant strategies for ML are presented together with the mostrelevant applications in aerospace engineering, revealing that ML is improving aircraft performance and that these techniques will have a large impact in the near future.
Point-based Acoustic Scattering for Interactive Sound Propagation via Surface Encoding
- Computer ScienceIJCAI
- 2021
This work presents a novel geometric deep learning method to compute the acoustic scattering properties of geometric objects using discrete Laplacian-based surface encoders and approximate the neighborhood of each point using a shared multi-layer perceptron.
An overview of machine learning and other data-based methods for spatial audio capture, processing, and reproduction
- Computer ScienceEURASIP J. Audio Speech Music. Process.
- 2022
This article reviews the most important application domains of data-based spatial audio including well-established methods that employ conventional signal processing while paying special attention to the most recent achievements that make use of machine learning.
References
SHOWING 1-10 OF 62 REFERENCES
Diffraction Kernels for Interactive Sound Propagation in Dynamic Environments
- Computer ScienceIEEE Transactions on Visualization and Computer Graphics
- 2018
A novel method to generate plausible diffraction effects for interactive sound propagation in dynamic scenes is presented and it is found that the auditory perception using the approach is comparable to that of a wave-based sound propagation method.
Fast Acoustic Scattering Using Convolutional Neural Networks
- Mathematics, PhysicsICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2020
This work proposes training a convolutional neural network to map from a convex scatterer’s crosssection to a 2D slice of the resulting spatial loudness distribution, and shows that employing a full-resolution residual network for the resulting image-to-image regression problem yields spatially detailed loudness fields.
Interactive sound propagation for dynamic scenes using 2D wave simulation
- PhysicsComput. Graph. Forum
- 2020
A technique to model wave‐based sound propagation to complement visual animation in fully dynamic scenes and suggest that wave solvers can be a practical approach to real‐time dynamic acoustics is presented.
Wave-ray coupling for interactive sound propagation in large complex scenes
- PhysicsACM Trans. Graph.
- 2013
A novel two-way pressure coupling technique at the interface of near-object and far-field regions that is able to simulate high-fidelity acoustic effects in large, complex indoor and outdoor environments and Half-Life 2 game engine.
Guided Multiview Ray Tracing for Fast Auralization
- Computer ScienceIEEE Transactions on Visualization and Computer Graphics
- 2012
A novel method for tuning geometric acoustic simulations based on ray tracing that efficiently computes early specular paths and first order diffraction with a multiview tracing algorithm and combines it with a fast GPU sound propagation system for interactive simulation.
High-order diffraction and diffuse reflections for interactive sound propagation in large environments
- Physics, Computer ScienceACM Trans. Graph.
- 2014
An incremental approach that combines radiosity and path tracing techniques to iteratively compute diffuse reflections and algorithms for wavelength-dependent simplification and visibility graph computation to accelerate higher-order diffraction at runtime are presented.
WAVE: Interactive Wave-based Sound Propagation for Virtual Environments
- Computer Science, PhysicsIEEE Transactions on Visualization and Computer Graphics
- 2015
An interactive wave-based sound propagation system that generates accurate, realistic sound in virtual environments for dynamic (moving) sources and listeners is presented and a novel algorithm to accurately solve the wave equation is proposed using a combination of precomputation techniques and GPU-based runtime evaluation.
Adaptive Sampling for Sound Propagation
- PhysicsIEEE Transactions on Visualization and Computer Graphics
- 2019
Precomputed sound propagation samples acoustics at discrete scene probe positions to support dynamic listener locations. An offline 3D numerical simulation is performed at each probe and the…
Interactive Sound Rendering on Mobile Devices using Ray-Parameterized Reverberation Filters
- Physics, Computer ScienceArXiv
- 2018
This work combines ray-tracing-based sound propagation with reverberation filters using robust automatic reverb parameter estimation that is driven by impulse responses computed at a low sampling rate to create a new sound rendering pipeline that is able to generate plausible sound propagation effects for interactive dynamic scenes.
Parametric directional coding for precomputed sound propagation
- PhysicsACM Trans. Graph.
- 2018
A novel parametric encoder is proposed that compresses this function within a budget of ~100MB for large scenes, while capturing many salient acoustic effects indoors and outdoors, and is complemented with a lightweight signal processing algorithm whose filtering cost is largely insensitive to the number of sound sources.