Audio support in Openverse

[zackkrida]

A long awaited feature for Openverse is support for Audio files. We currently have some limited audio support through meta search, but would like to expand this to first-class support for audio files. This issue is meant to track progress on Audio across all three repositories; sub-issues within each repo can and should be made. Feel free to edit this post to link them.

Supporting documents

• Francisco's original audio mockups
• Provider API research (avalibility, fields, etc.)

Required work per repo

Frontend

Audio work is tracked in WordPress/openverse-frontend#118

• Search results page defaults to an 'all media types' view (see below example:)

• Audio results no longer shows meta search; instead shows audio results
• [Audio] Sidebar filters openverse-frontend#105 Conditional filters in the sidebar when audio results are present:
  • Genre
  • Duration
  • Format (filetype)
• New frontend components
  • Audio track list view
    • waveform
    • play/pause button
    • license icons
    • duration
    • album art
  • Single audio result view
    • 'similar audio' and 'more from the same creator' sections
    • download buttons
    • metadata tables

API

• Django models for audio
  • Reuse code from image models
  • Identify fields to store
• New endpoints for audio
  • Add DRF serializers for audio
  • Add DRF endpoints for audio
• Make the thumbnail server work with audio album art/images
• Add audio ingestion to the Ingestion server
  • Replace hard-coded media type (image) in the ingestion server with the media type based on db model that's being ingested
  • Add Elasticsearch audio models
  • Add audio ingestion to the ingestion server (copying the audio from Upstream DB, and creating the Elasticsearch audio models)
  • Add Audio sample data in .csv format for local testing
  • Update sample data for audio

Catalog

• New audio DB table
  • Add audio db table and Update loader task to save the audio data to the DB
  • Update popularity calculations in the materialized view to include audio
• New generic model for a catalog entity with ImageStore as a subclass
  • Add catalog entity subclasses for audio
• New DAGs for each audio provider (Provider API research)
  • Freesound.org
  • Jamendo
  • Wikimedia Commons

[obulat]

Here is a list of Audio-related API Ingestion issues in the Catalog repository:

• [API Integration - AUDIO] Wikimedia Commons [API Integration - AUDIO] Wikimedia Commons (original #316) #1750
• [API Integration - AUDIO] FreeSound.org [API Integration - AUDIO] FreeSound.org (original #282) #1798
• [API Integration - AUDIO] Free Music Archive / Tribe Of Noise Free Music Archive / Tribe Of Noise #1795
• [API Integration - AUDIO] Internet Archive Audio Internet Archive Audio #1777
• [API Integration - AUDIO] ccmixter ccMixter #1770
• [API Integration - AUDIO] IMSLP IMSLP #1768
• [API Integration - AUDIO] Jamendo [API Integration - AUDIO] Jamendo (original #345) #1763

[fcoveram]

The initial idea behind audio integration was to introduce a general view that gathers all media types supported. This proposal assumed that some users wanted to see what Search offers.

However, during user interviews, we identified that users looking for a relevant result were already sure about the media type they wanted before entering Search. For instance, users searching for audio results were expecting to listen on the page and navigate through tags related to audio content. I can't envision a case where users decided to pick an image instead of audio for whatever purpose.

That said, I suggest redesigning this first view to provide different experiences for different users.

[zackkrida]

@panchovm Yes, I think most users will choose a media type on the home page, when they make their search. If they do that, they will only see results for their chosen media type. We only need the general view if a user searches for "all" media types, which I do not think will be very common, but we should still have that view.