[Text] -> [Non-speech Audio] Experiments

[StevenSchrembeck]

We're cracking some cool problems in narrative sound design using the great audio diffusion models created here, starting with text to "sound effect".

"Footsteps in gravel, distant, many" -> sample.wav 🔈
"A train passing through a huge cave" -> sample.wav 🔈
"A sword fight with tubas" -> sample.wav 🔈
"Metal clank, ominous, deep" -> sample.wav 🔈
"Roller-coaster on the moon" -> sample.wav 🔈

Any major updates or results will be posted here in this discussion, but collaboration will be through the Non-speech Audio Diffusion discord server.

--

Project goals:

• [ x ] Set up training environment with default settings
• [ ~ ] Train an AudioDiffusionConditional end to end with an arbitrary dataset
• Build public sound effects dataset
• Modify model input/output layers to accommodate SFX dataset
• First pass training. Iterate on hyperparameters until first result is achieved
• Custom pre-processing pipeline
  • Quality improvements (normalization, standardize lengths, etc)
  • Quantity/variation improvements (noise injection, cropping, audio effects like reverb)

Output improvement ideas:

• Build private sound effects dataset (high quality files we own the license to but cannot share)
• Hydrate text embeddings with embedding values from a model like BERT. Then fine-tune on SFX labels
• Add hyperparameter search algorithm
• Cull dataset of low rated audio samples
• Massively expand dataset with annotated Youtube (or movie / TV) audio, using annotations as text labels