Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Audio PAIRS audio scenario #3149

Merged
merged 4 commits into from
Nov 10, 2024
Merged

Conversation

ImKeTT
Copy link
Collaborator

@ImKeTT ImKeTT commented Nov 10, 2024

This PR adds the Audio PAIRS audio scenario, which is an audio extension of the PAIRS (https://arxiv.org/abs/2402.05779) from VHELM. The dataset was created by:

  1. Collecting instructions that were used for PAIRS's image generation from Table C.1, C.2, C.3 in the PAIRS paper (https://arxiv.org/abs/2402.05779).
  2. Assigning four different characters for each instruction, i.e., black woman, black man, white woman, white man.
  3. Using OpenAI's TTS-001-HD API (tutorial) to generate the audio files. In detail, the voice of Echo for men and the voice of Nova for women (voice options).
  4. Using the original questions from PAIRS as the Audio PAIRS's questions.
  5. Adding the "unclear" option as the correct answer following VHELM.

Audio samples of the data can be found here.
I'm also attaching the run_spec.json and scenario_state.json files.

run_spec_audio_pairs.json
scenario_state_audio_pairs.json

@ImKeTT ImKeTT requested a review from teetone November 10, 2024 05:43
@ImKeTT ImKeTT merged commit 3d28de9 into stanford-crfm:main Nov 10, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants