Built for Gemini API Developer Competition
Atmosphere Showcase: atmosphere.pab.dev
Atmosphere Showcase Repository: paul-bokelman/atmosphere-showcase
Atmosphere aims to enhance the storytelling experience by overlaying ambient audio onto audio book segments with strong auditory and sensory scenes. By utilizing the BBC sound effects library, Atmosphere is able to provide a rich and immersive experience for listeners. Atmosphere is built with Python and utilizes the Gemini API to generate immersive audio books.
Atmosphere interacts with the Google Gemini Flash API in 2 independent steps to generate immersive and cohesive audio recordings.
Initially, Atmosphere is provided with the audio recording in its entirety, and is tasked with locating timestamps within the audio book that correspond to scenes that would be enhanced by ambient audio. In addition to providing the timestamp of these scenes, Gemini also produces a concise description of the context of the scene, and salient keywords from the scene that construct the overall tone of the segment.
Using the keywords gathered for each scene identified by Gemini, Atmosphere sifts through the 33,000+ sound effects and their associated keywords in the BBC sound effects library, and acquires all sounds with at least one keyword matching that of the scene. With all sound effects a minimum of one matching keyword compiled into a list, Gemini is prompted to pick the sound from the list that best matches the scene description created by Gemini in step 1.
Once all sounds have been selected for each scene, Atmosphere normalizes, fades, trims then overlays the selected sound effects onto the corresponding audio book segments.
- Clone the Atmosphere repository to your local machine.
- Create and use a virtual environment by running
python3.11 -m venv env && source env/bin/activate
. - Install the required dependencies by running
python3.11 -m pip install -r requirements.txt
. - Add your Gemini API key to the
.env.development
asGEMINI_API_KEY
file in the root of the Atmosphere directory. - Create a
recordings
andout
directory under amedia'
directory in the root of the Atmosphere directory (/media/recordings/
,/media/out
). This is where you will place the audio book you would like to generate an immersive version of. - Place the audio book you would like to generate an immersive version of in the 'recordings' directory.
- Enter the command
python3.11 main.py
. This will begin running the generator. - Once running, select "Generate Immersive Audio" in the drop down menu, and press enter.
- Follow the prompts to select the audio book you would like to generate an immersive version of
- The output mp3 file will be found in
/media/out/{YOUR_AUDIO_BOOK_NAME}/{YOUR_AUDIO_BOOK_NAME}-immersive.mp3
- Show off your immersive audio book to your friends and family!
Atmosphere was inspired by the idea of enhancing the storytelling experience by providing listeners with a more immersive and sensory-rich experience. By overlaying ambient audio onto audio book segments with strong auditory and sensory scenes, Atmosphere is able to provide a rich and immersive experience for listeners. Atmosphere could be used to enhance the storytelling experience for a wide variety of audio books, including fiction, non-fiction, and educational audio books. An immersive audio book could be used to help listeners better understand and visualize the story, and could be used to help listeners better engage with the story.
Ultimately, a product like Atmosphere could be added onto audio book streaming services such as Google Play and Audible to offer customers an immersive option alongside standard audio books, this would be especially desireable for heavily descriptive books and really try to invest the reader in the environment/scene. Atmosphere could also help improve the accessibility of audio books for people with visual impairments or poor visualization skills, by providing additional sensory information to help listeners better understand and connect with the story.
Atmosphere excels at locating and assigning sounds for scenes that are general and require little to no contextualization or background information to properly classify. Examples of such scenes might be that of a river flowing in the distance, the hustle and bustle of a factory, or birds chirping. Scenes such as these can be simply taken at face value and directly interpreted without strict examination of tone, setting, and figurative language.
Admittedly, Atmosphere predictably struggles to convey adequate sensory enhancement in scenes that can not be snipped out of their place within the context of the story as a whole. Often Atmosphere takes scenes too literally and fails to understand the nuance of metaphors and may lose sight of the original scene. Similarly, Atmosphere has a tendency to place sound effects from far-removed and unfitting time frames into scenes; a habit that can be disorienting and distracting to listeners. Atmosphere regularly fails to recognize the nuanced differences between a busy medieval marketplace, and Times Square in the early 21st century. Again, this inadequacy is rooted in a lack of context for Gemini to draw from.
We believe that Atmosphere could be improved by providing Gemini with more context and background information about the audio book it is analyzing. By providing Gemini with more information about the setting, characters, and themes of the audio book, Gemini would be better equipped to identify and classify scenes that require more nuanced interpretation. Additionally, Atmosphere could be improved by providing Gemini with more information about the tone and style of the audio book, so that Gemini can better match the sound effects to the mood and atmosphere of the audio book.