Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation

@article{Tang2020LearningAS,
  title={Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation},
  author={Zhenyu Tang and Hsien-Yu Meng and Dinesh Manocha},
  journal={2021 IEEE Virtual Reality and 3D User Interfaces (VR)},
  year={2020},
  pages={835-844}
}
We present a novel hybrid sound propagation algorithm for interactive applications. Our approach is designed for dynamic scenes and uses a neural network-based learned scattered field representation along with ray tracing to generate specular, diffuse, diffraction, and occlusion effects efficiently. We use geometric deep learning to approximate the acoustic scattering field using spherical harmonics. We use a large 3D dataset for training, and compare its accuracy with the ground truth… 
[PDF] Semantic Reader
7 Citations
Background Citations
4
Methods Citations
2

Figures and Tables from this paper

7 Citations

Acoustic Rendering Based on Geometry Reduction and Acoustic Material Classification

We present work in progress on a pipeline for audio rendering integrating vision-based systems for acoustic material classification. With a marching cubes algorithm, the pipeline estimates a cuboid…

DeepEigen: Learning-based Modal Sound Synthesis with Acoustic Transfer Maps

This work combines two network-based solutions to formulate a complete learning-based 3D modal sound model that includes a 3D sparse convolution network as the eigendecomposition solver and an encoder-decoder network for the prediction of the Far-Field Acoustic Transfer maps.

GWA: A Large High-Quality Acoustic Dataset for Audio Processing

The Geometric-Wave Acoustic (GWA) dataset is presented, a large-scale audio dataset of about 2 million synthetic room impulse responses (IRs) and their corresponding detailed geometric and simulation configurations that is the first data with accurate wave acoustic simulations in complex scenes.

Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse

This work proposes a practical approach for designing parametric 6-degree-of-freedom object-based interactive audio engines to deliver the perceptually relevant binaural cues necessary for audio/visual and virtual/real congruence in Metaverse experiences.

Improving aircraft performance using machine learning: a review

The basic concepts and the most relevant strategies for ML are presented together with the mostrelevant applications in aerospace engineering, revealing that ML is improving aircraft performance and that these techniques will have a large impact in the near future.

Point-based Acoustic Scattering for Interactive Sound Propagation via Surface Encoding

This work presents a novel geometric deep learning method to compute the acoustic scattering properties of geometric objects using discrete Laplacian-based surface encoders and approximate the neighborhood of each point using a shared multi-layer perceptron.

An overview of machine learning and other data-based methods for spatial audio capture, processing, and reproduction

This article reviews the most important application domains of data-based spatial audio including well-established methods that employ conventional signal processing while paying special attention to the most recent achievements that make use of machine learning.

References

SHOWING 1-10 OF 62 REFERENCES

Diffraction Kernels for Interactive Sound Propagation in Dynamic Environments

A novel method to generate plausible diffraction effects for interactive sound propagation in dynamic scenes is presented and it is found that the auditory perception using the approach is comparable to that of a wave-based sound propagation method.

Fast Acoustic Scattering Using Convolutional Neural Networks

This work proposes training a convolutional neural network to map from a convex scatterer’s crosssection to a 2D slice of the resulting spatial loudness distribution, and shows that employing a full-resolution residual network for the resulting image-to-image regression problem yields spatially detailed loudness fields.

Interactive sound propagation for dynamic scenes using 2D wave simulation

A technique to model wave‐based sound propagation to complement visual animation in fully dynamic scenes and suggest that wave solvers can be a practical approach to real‐time dynamic acoustics is presented.

Wave-ray coupling for interactive sound propagation in large complex scenes

A novel two-way pressure coupling technique at the interface of near-object and far-field regions that is able to simulate high-fidelity acoustic effects in large, complex indoor and outdoor environments and Half-Life 2 game engine.

Guided Multiview Ray Tracing for Fast Auralization

A novel method for tuning geometric acoustic simulations based on ray tracing that efficiently computes early specular paths and first order diffraction with a multiview tracing algorithm and combines it with a fast GPU sound propagation system for interactive simulation.

High-order diffraction and diffuse reflections for interactive sound propagation in large environments

An incremental approach that combines radiosity and path tracing techniques to iteratively compute diffuse reflections and algorithms for wavelength-dependent simplification and visibility graph computation to accelerate higher-order diffraction at runtime are presented.

WAVE: Interactive Wave-based Sound Propagation for Virtual Environments

An interactive wave-based sound propagation system that generates accurate, realistic sound in virtual environments for dynamic (moving) sources and listeners is presented and a novel algorithm to accurately solve the wave equation is proposed using a combination of precomputation techniques and GPU-based runtime evaluation.

Adaptive Sampling for Sound Propagation

Precomputed sound propagation samples acoustics at discrete scene probe positions to support dynamic listener locations. An offline 3D numerical simulation is performed at each probe and the…

Interactive Sound Rendering on Mobile Devices using Ray-Parameterized Reverberation Filters

This work combines ray-tracing-based sound propagation with reverberation filters using robust automatic reverb parameter estimation that is driven by impulse responses computed at a low sampling rate to create a new sound rendering pipeline that is able to generate plausible sound propagation effects for interactive dynamic scenes.

Parametric directional coding for precomputed sound propagation

A novel parametric encoder is proposed that compresses this function within a budget of ~100MB for large scenes, while capturing many salient acoustic effects indoors and outdoors, and is complemented with a lightweight signal processing algorithm whose filtering cost is largely insensitive to the number of sound sources.
...