Support multimodal models: GPT-4V, Gemini Vision, Luminous Multimodal

[teetone]

Changes

• Support multimodal models:
  • GPT-4V: openai/gpt-4-vision-preview (resolves Support GPT-4V #2070)
  • Gemini Vision: google/gemini-pro-vision (resolves Support Google's Gemini multimodal models #2103, resolves Add support for multimodal prompt for Gemini #2187)
  • Luminous Multimodal: AlephAlpha/luminous-base, AlephAlpha/luminous-extended (resolves Support Luminous-base and Luminous-extended (multimodal) #1929)
    • Switched to using the Aleph Alpha Python SDK to support sending multimodal prompts
• Handled IDEFICS instruct special prompting and cleaned up the adapter specs in vlm_run_specs.py from before when IDEFICS was the only supported VLM in HELM.
• Fixed VizWizScenario bug where annotations were not being downloaded

Example generations

Prompt

Where is the horse?

Outputs

• GPT-4V: The horse is on a city street, standing on the sidewalk near the curb. There is a person next to the horse, who seems to be a police officer based on the uniform. The street sign indicates this is the corner of "N CALVERT" and "W BIDDLE" streets.
• Gemini Vision: The horse is on the sidewalk, next to a woman.
• Luminous Extended: A horse is on the street.
• Luminous Base: A horse is on the street. A horse is walking down the street.
• IDEFICS 9B Instruct: The horse is on the street.

[JosselinSomervilleRoberts]

LGTM

[yifanmai]

LGTM