Corpus ID: 1488871

On Evaluation Validity in Music Autotagging

@article{Gouyon2014OnEV,
  title={On Evaluation Validity in Music Autotagging},
  author={Fabien Gouyon and Bob L. Sturm and Jo{\~a}o Lobato Oliveira and Nuno Hespanhol and Thibault Langlois},
  journal={ArXiv},
  year={2014},
  volume={abs/1410.0001}
}
Music autotagging, an established problem in Music Information Retrieval, aims to alleviate the human cost required to manually annotate collections of recorded music with textual labels by automating the process. Many autotagging systems have been proposed and evaluated by procedures and datasets that are now standard (used in MIREX, for instance). Very little work, however, has been dedicated to determine what these evaluations really mean about an autotagging system, or the comparison of two… Expand
A method of music autotagging based on audio and lyrics
TLDR
A music autotagging system that relies on both audio and lyrics to solve the above problems and supports that a multitask learning classifier that combines audio and lyric information has a better performance than a single-task learning classification method using only audio data than previous studies. Expand
Revisiting Autotagging Toward Faultless Instrumental Playlists Generation
TLDR
This paper posits a new approach that uses the presence probability of a frame’s predicted singing voice to deduce the track tag, i.e., whether the track is an Instrumental or a Song, and proposes a paradigm that focuses on minority classes in musical databases and thus enhances general user satisfaction. Expand
The State of the Art Ten Years After a State of the Art: Future Research in Music Information Retrieval
TLDR
A case study of all published research using the most-used benchmark dataset in MGR during the past decade shows that none of the evaluations in these many works is valid to produce conclusions with respect to recognizing genre, i.e. that a system is using criteria relevant for recognizing genre. Expand
The “Horse” Inside: Seeking Causes Behind the Behaviors of Music Content Analysis Systems
TLDR
This article takes a state-of-the-art music content analysis system and investigates what causes it to achieve exceptionally high performance in a benchmark music audio dataset, and dissects the system to understand its operation, determine its sensitivities and limitations, and predict the kinds of knowledge it could and could not possess about music. Expand
Toward Faultless Content-Based Playlists Generation for Instrumentals
  • Yann Bayle, Pierre Hanna, Matthias Robine
  • Computer Science
  • 2017
TLDR
An industrial real-world musical database that is unevenly distributed between Songs and Instrumentals and bigger than databases used in previous studies is considered, which generates an Instrumental playlist with up to three times less false positives than cutting edge methods. Expand
SATIN: a persistent musical database for music information retrieval and a supporting deep learning experiment on song instrumental classification
TLDR
SATIN is a database of 400k audio-related metadata and identifiers that aims at facilitating reproducibility and comparisons among the Music Information Retrieval algorithms, and SOFT1, the first Set of FeaTures extracted by a company thanks to SATIN is introduced. Expand
Making Explicit the Formalism Underlying Evaluation in Music Information Retrieval Research: A Look at the MIREX Automatic Mood Classification Task
TLDR
The formalism underlying evaluation in music information retrieval research is made explicit, and this formalism is applied to discuss the MIREX automatic mood classification task. Expand
Kara1k: A Karaoke Dataset for Cover Song Identification and Singing Voice Analysis
TLDR
This work introduces Kara1k, a new musical dataset composed of 2,000 analyzed songs thanks to a partnership with a karaoke company, and provides metadata such as the title, genre, original artist, year, International Standard Recording Code and the ground truths for the singer's gender. Expand
SATIN: A Persistent Musical Database for Music Information Retrieval
TLDR
SATIN is a database of 400k audio-related metadata and identifiers that aims at facilitating reproducibility and comparisons among the MIR algorithms that resolves the major problems of accessing more tracks, managing copyrights locks, saving computation time, and guaranteeing consistency over research databases. Expand
New Paths in Music Recommender Systems Research
TLDR
Three use cases are focused on: automatic playlist generation, context-aware music recommendation, and recommendation in the creative process of music making. Expand
...
1
2
...

References

SHOWING 1-10 OF 48 REFERENCES
Three Current Issues In Music Autotagging
TLDR
This paper presents autotagging experiments conducted with two different systems and conducts experiments on the generality of autotagged models, showing that a number of different methods at a similar performance level to the state-of-the-art fail to learn tag models able to generalize to datasets from different origins. Expand
Exploring automatic music annotation with "acoustically-objective" tags
TLDR
This work uses the Swat10k data set, which consists of 10,870 songs annotated using a vocabulary of 475 acoustic tags and 153 genre tags from Pandora's Music Genome Project, to develop an autotagging system and evaluates two new sets of content-based audio features obtained using the publicly-available Echo Nest API. Expand
Automatic Tagging of Audio: The State-of-the-Art
TLDR
This chapter provides a review of the state-of-the-art methods for addressing automatic tagging of audio, divided in the following sections: goal, framework, audio representation, labeled data, classification, evaluation, and future directions. Expand
Autotagger: A Model for Predicting Social Tags from Acoustic Features on Large Music Databases
TLDR
This paper uses a set of 360 classifiers trained using the online ensemble learning algorithm FilterBoost to map audio features onto social tags collected from the Web, allowing for insertion of previously unheard music into a social recommender system. Expand
Evaluation in Music Information Retrieval
TLDR
A survey of past evaluation work in the context of Text IR, from the point of view of validity, reliability and efficiency of the experiments is presented. Expand
Social Tagging and Music Information Retrieval
TLDR
The state of the art in commercial and research social tagging systems for music is described, how tags are collected and used in current systems are described, and some of the issues that are encountered when using tags are explored. Expand
Semantic Annotation and Retrieval of Music and Sound Effects
We present a computer audition system that can both annotate novel audio tracks with semantically meaningful words and retrieve relevant tracks from a database of unlabeled audio content given aExpand
A Simple Method to Determine if a Music Information Retrieval System is a “Horse”
  • Bob L. Sturm
  • Computer Science
  • IEEE Transactions on Multimedia
  • 2014
We propose and demonstrate a simple method to explain the figure of merit (FoM) of a music information retrieval (MIR) system evaluated in a dataset, specifically, whether the FoM comes from theExpand
Improving automatic music tag annotation using stacked generalization of probabilistic SVM outputs
TLDR
It is described how stacked generalization can be used to improve the performance of a state-of-the-art automatic tag annotation system for music based on audio content analysis and report results on two publicly available datasets. Expand
A Generative Context Model for Semantic Music Annotation and Retrieval
TLDR
A Dirichlet mixture model (DMM) is proposed as a second, additional stage in the modeling process, to supplement any auto-tagging system that generates a semantic multinomial over a vocabulary of tags when annotating a song. Expand
...
1
2
3
4
5
...