Ubicoustics: Plug-and-Play Acoustic Activity Recognition

@article{Laput2018UbicousticsPA,
  title={Ubicoustics: Plug-and-Play Acoustic Activity Recognition},
  author={Gierad Laput and Karan Ahuja and Mayank Goel and Chris Harrison},
  journal={Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology},
  year={2018}
}
Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. [] Key Method We start by taking an existing, state-of-the-art sound labeling model, which we then tune to classes of interest by drawing data from professional sound effect libraries traditionally used in the entertainment industry.
Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition
TLDR
This work built an end-to-end system for self-supervised learning of events labelled through one-shot interaction, and shows that the system can accurately and automatically learn acoustic events across environments, while adhering to users' preferences for non-intrusive interactive behavior.
Ok Google, What Am I Doing?
TLDR
This work explores how off-the-shelf conversational assistants can be enhanced with acoustic-based human activity recognition by leveraging the short interval after a voice command is given to the device.
LASO: Exploiting Locomotive and Acoustic Signatures over the Edge to Annotate IMU Data for Human Activity Recognition
TLDR
This paper proposes LASO, a multimodal approach for automated data annotation from acoustic and locomotive information, and uses pre-trained audio-based activity recognition models for labeling the IMU data while handling the acoustic noises.
Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos
TLDR
A framework for audio-based activity recognition that can make use of millions of embedding features from public online video sound clips is proposed, based on the combination of oversampling and deep learning approaches, that does not require further feature processing or outliers filtering.
ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users
TLDR
ProtoSound is introduced, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fine-grained categories and discussing open challenges in personalizable sound recognition, including the need for better recording interfaces and algorithmic improvements.
Robust Audio Sensing with Multi-Sound Classification
  • Peter Haubrick, Juan Ye
  • Computer Science
    2019 IEEE International Conference on Pervasive Computing and Communications (PerCom
  • 2019
TLDR
The results have demonstrated that the proposed stacked classifier can robustly identify each sound category among mixed acoustic signals, without the need of any a priori knowledge about the number and signature of sounds in the mixed signals.
SoundWatch: Exploring Smartwatch-based Deep Learning Approaches to Support Sound Awareness for Deaf and Hard of Hearing Users
TLDR
A performance evaluation of four low-resource deep learning sound classification models: MobileNet, Inception, ResNet-lite, and VGG-lite across four device architectures: watch-only, watch+phone, watch-+phone+cloud, and watch+cloud finds that the watch+phones architecture provided the best balance between CPU, memory, network usage, and classification latency.
PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition
TLDR
PrivacyMic, a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely, and real-world activity recognition performance is comparable to simulated results, suggesting immediate viability in performing privacy-preserving daily activity recognition.
Accoustate: Auto-annotation of IMU-generated Activity Signatures under Smart Infrastructure
TLDR
It is observed that non-overlapping acoustic gaps exist with a high probability during the simultaneous activities performed by two individuals in the environment’s acoustic context, which helps to resolve the overlapping activity signatures to label them individually.
Non-intrusive Continuous User Identification from Activity Acoustic Signatures
TLDR
This paper develops an automated machine learning-based framework, trained on basic acoustic features, that can identify the users from the acoustic signatures generated by their activities to distinguish individuals performing the activities continuously.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 49 REFERENCES
DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
TLDR
This paper presents DeepEar -- the first mobile audio sensing framework built from coupled Deep Neural Networks (DNNs) that simultaneously perform common audio sensing tasks and shows DeepEar is feasible for smartphones by building a cloud-free DSP-based prototype that runs continuously, using only 6% of the smartphone's battery daily.
SoundSense: scalable sound sensing for people-centric applications on mobile phones
TLDR
This paper proposes SoundSense, a scalable framework for modeling sound events on mobile phones that represents the first general purpose sound sensing system specifically designed to work on resource limited phones and demonstrates that SoundSense is capable of recognizing meaningful sound events that occur in users' everyday lives.
Looking to listen at the cocktail party
TLDR
A deep network-based model that incorporates both visual and auditory signals to solve a single speech signal from a mixture of sounds such as other speakers and background noise, showing clear advantage over state-of-the-art audio-only speech separation in cases of mixed speech.
Audio-based human activity recognition using Non-Markovian Ensemble Voting
TLDR
A novel recognition approach called Non-Markovian Ensemble Voting (NEV) able to classify multiple human activities in an online fashion without the need for silence detection or audio stream segmentation is proposed.
Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers
TLDR
This work describes a method for the recognition of activities that are characterized by a hand motion and an accompanying sound using microphones and three-axis accelerometers mounted at two positions on the user's arms using on-body sensing.
BodyScope: a wearable acoustic sensor for activity recognition
TLDR
A wearable acoustic sensor, called BodyScope, is developed to record the sounds produced in the user's throat area and classify them into user activities, such as eating, drinking, speaking, laughing, and coughing.
Audio-based context recognition
TLDR
This paper investigates the feasibility of an audio-based context recognition system developed and compared to the accuracy of human listeners in the same task, with particular emphasis on the computational complexity of the methods.
NELS - Never-Ending Learner of Sounds
TLDR
This work introduces the Never-Ending Learner of Sounds (NELS), a project for continuously learning of sounds and their associated knowledge, and proposes a system that continuously learns from the web relations between sounds and language.
Reliable detection of audio events in highly noisy environments
Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds
TLDR
This work proposes Zensors, a new sensing approach that fuses real-time human intelligence from online crowd workers with automatic approaches to provide robust, adaptive, and readily deployable intelligent sensors.
...
1
2
3
4
5
...