Conditional LSTM-GAN for Melody Generation from Lyrics

@article{Yu2021ConditionalLF,
  title={Conditional LSTM-GAN for Melody Generation from Lyrics},
  author={Yi Yu and Simon Canales},
  journal={ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)},
  year={2021},
  volume={17},
  pages={1 - 20}
}
  • Yi Yu, Simon Canales
  • Published 2021
  • Computer Science, Engineering
  • ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)
Melody generation from lyrics has been a challenging research issue in the field of artificial intelligence and music, which enables to learn and discover latent relationship between interesting lyrics and accompanying melody. [...] Key Method Most importantly, we propose a novel deep generative model, conditional Long Short-Term Memory - Generative Adversarial Network (LSTM-GAN) for melody generation from lyrics, which contains a deep LSTM generator and a deep LSTM discriminator both conditioned on lyrics.Expand

Paper Mentions

Lyrics-Conditioned Neural Melody Generation
TLDR
By exploiting a large music dataset with 12,197 pairs of English lyrics and melodies, a lyrics-conditioned AI neural melody generation system that consists of three components: lyrics encoder network, melody generation network, and MIDI sequence tuner is developed. Expand
AI-Lyricist: Generating Music and Vocabulary Constrained Lyrics
We propose AI-Lyricist: a system to generate novel yet meaningful lyrics given a required vocabulary and a MIDI file as inputs. This task involves multiple challenges, including automaticallyExpand
Diverse Melody Generation from Chinese Lyrics via Mutual Information Maximization
TLDR
The method, which is called Diverse Melody Generation (DMG), a sequence-to-sequence model learns to generate diverse melodies heavily depending on the input style ids, while keeping the tonality and improving the alignment. Expand
SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint
TLDR
SongMASS is proposed, which leverages masked sequence to sequence (MASS) pre-training and attention based alignment modeling for lyric- to-melody and melody-to-lyric generation and generates lyric and melody with significantly better quality than the baseline method without pre- training or alignment constraint. Expand
Automatic Neural Lyrics and Melody Composition
TLDR
The qualitative and quantitative evaluation measures revealed that the proposed method, AutoNLMC, is indeed capable of generating original lyrics and corresponding melody for composing new songs. Expand
Unsupervised Generative Adversarial Alignment Representation for Sheet music, Audio and Lyrics
  • D. Zeng, Yi Yu, K. Oyama
  • Engineering, Computer Science
  • 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM)
  • 2020
TLDR
The proposed unsupervised generative adversarial alignment representation (UGAAR) model to learn deep discriminative representations shared across three major musical modalities: sheet music, lyrics, and audio, where a deep neural network based architecture on three branches is jointly trained. Expand
TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method
  • Zeqian Ju, Peiling Lu, +7 authors Tie-Yan Liu
  • Computer Science, Engineering
  • ArXiv
  • 2021
TLDR
TeleMelody is developed, a two-stage lyric-to-melody generation system with music template to bridge the gap between lyrics and melodies and generates melodies with higher quality, better controllability, and less requirement on paired lyric-Melody data than previous generation systems. Expand
AI Song Contest: Human-AI Co-Creation in Songwriting
TLDR
Findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression. Expand
MTM Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio?
TLDR
The goal of MTM Dataset collection is to overcome the constraints by extending music notes to sheet music and music audio, and build music-note and syllable fine grained alignment, such that the dataset can be used to learn joint representation across multimodal music data. Expand
Conditional Hybrid GAN for Sequence Generation
TLDR
Through evaluating the task of generating melody (associated with note, duration, and rest) from lyrics, it is demonstrated that the proposed C-Hybrid-GAN outperforms the existing methods in context-conditioned discrete-valued sequence generation. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 57 REFERENCES
Neural Melody Composition from Lyrics
TLDR
A melody composition model that generates lyrics-conditional melody as well as the exact alignment between the generated melody and the given lyrics simultaneously is developed based on the sequence-to-sequence framework. Expand
A Hierarchical Recurrent Neural Network for Symbolic Melody Generation
TLDR
A hierarchical recurrent neural network for melody generation, which consists of three long-short-term-memory (LSTM) subnetworks working in a coarse-to-fine manner along time, which produces better melodies evaluated by humans. Expand
Music Generation by Deep Learning - Challenges and Directions
TLDR
Some limitations of a direct application of deep learning to music generation are selected, why the issues are not fulfilled and how to address them by possible approaches are analyzed. Expand
Deep learning for music generation: challenges and directions
TLDR
Some limitations of a direct application of deep learning to music generation are selected and why the issues are not fulfilled and how to address them by possible approaches are analyzed. Expand
Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval
TLDR
This is the first study that uses deep architectures for learning the temporal correlation between audio and lyrics, involving two-branch deep neural networks for audio modality and text modality (lyrics) and two significant contributions are made in the audio branch. Expand
Generating Polyphonic Music Using Tied Parallel Networks
TLDR
A neural network architecture which enables prediction and composition of polyphonic music in a manner that preserves translation-invariance of the dataset and attains high performance at a musical prediction task and successfully creates note sequences which possess measure-level musical structure. Expand
IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation
TLDR
A novel Introspective Recurrent Convolutional GAN approach, where LSTM cells are integrated with 2D transconvolutional layers to generate plausible videos from given text and proposes mutual-information introspection to semantically align the generated video to text. Expand
Automatic Generation of Melodic Accompaniments for Lyrics
TLDR
This work describes a system that is able to automatically generate and evaluate musical accompaniments for a given set of lyrics, which derives the rhythm for the melodic accompaniment from the cadence of the text. Expand
Automatic song composition from the lyrics exploiting prosody of Japanese language
TLDR
An algorithm that can automatically generate songs from Japanese lyrics by considering composition as an optimal-solution search problem under constraints given by the prosody of the lyrics is presented. Expand
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
TLDR
Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. Expand
...
1
2
3
4
5
...