Audio AI Timeline Here we will keep track of the latest AI models for waveform based audio generation, starting in 2023! 2023 Date Release [Samples] Paper Code Trained Model 14.11 Mustango: Toward Controllable Text-to-Music Generation arXiv GitHub Hugging Face 13.11 Music ControlNet: Multiple Time-varying Controls for Music Generation arXiv - - 02.11 E3 TTS: Easy End-to-End Diffusion-based Text to Speech arXiv - - 01.10 UniAudio: An Audio Foundation Model Toward Universal Audio Generation arXiv GitHub - 24.09 VoiceLDM: Text-to-Speech with Environmental Context arXiv GitHub - 05.09 PromptTTS 2: Describing and Generating Voices with Text Prompt arXiv - - 14.08 SpeechX: Neural Codec Language Model as a Versatile Speech Transformer arXiv - - 10.08 AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining arXiv GitHub Hugging Face 09.08 JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models arXiv - - 03.08 MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies arXiv GitHub - 14.07 Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts arXiv - - 10.07 VampNet: Music Generation via Masked Acoustic Token Modeling arXiv GitHub - 22.06 AudioPaLM: A Large Language Model That Can Speak and Listen arXiv - - 19.06 Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale PDF GitHub - 08.06 MusicGen: Simple and Controllable Music Generation arXiv GitHub Hugging Face Colab 06.06 Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias arXiv - - 01.06 Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis arXiv GitHub - 29.05 Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation arXiv - - 25.05 MeLoDy: Efficient Neural Music Generation arXiv - - 18.05 CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training arXiv - - 18.05 SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities arXiv GitHub - 16.05 SoundStorm: Efficient Parallel Audio Generation arXiv GitHub (unofficial) - 03.05 Diverse and Vivid Sound Generation from Text Descriptions arXiv - - 02.05 Long-Term Rhythmic Video Soundtracker arXiv GitHub - 24.04 TANGO: Text-to-Audio generation using instruction tuned LLM and Latent Diffusion Model PDF GitHub Hugging Face 18.04 NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers arXiv GitHub (unofficial) - 10.04 Bark: Text-Prompted Generative Audio Model - GitHub Hugging Face Colab 03.04 AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models arXiv - - 08.03 VALL-E X: Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling arXiv - - 27.02 I Hear Your True Colors: Image Guided Audio Generation arXiv GitHub - 08.02 Noise2Music: Text-conditioned Music Generation with Diffusion Models arXiv - - 04.02 Multi-Source Diffusion Models for Simultaneous Music Generation and Separation arXiv GitHub - 30.01 SingSong: Generating musical accompaniments from singing arXiv - - 30.01 AudioLDM: Text-to-Audio Generation with Latent Diffusion Models arXiv GitHub Hugging Face 30.01 Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion arXiv GitHub - 29.01 Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models PDF - - 28.01 Noise2Music - - - 27.01 RAVE2 [Samples RAVE1] arXiv GitHub - 26.01 MusicLM: Generating Music From Text arXiv GitHub (unofficial) - 18.01 Msanii: High Fidelity Music Synthesis on a Shoestring Budget arXiv GitHub Hugging Face Colab 16.01 ArchiSound: Audio Generation with Diffusion arXiv GitHub - 05.01 VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers arXiv GitHub (unofficial) (demo) -