RFC: chat completions endpoint with support for receiving and generating audio

[fedirz]

This is what I'm referring to: Audio generation

OpenAI has recently added gpt-4o-audio-preview model, which supports the following input/output combinations:

• text in → text + audio out
• audio in → text + audio out
• audio in → text out
• text + audio in → text + audio out
• text + audio in → text out

I want to create a POST /v1/chat/completions endpoint emulating this functionality. This project will not turn into another LLM runtime like Ollama or VLLM. I'm thinking of speaches acting as a proxy. Here's the flow I have in mind for text + audio in → text + audio out:

• receive the audio inputs

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "What is in this recording?"
      },
      {
        "type": "input_audio",
        "input_audio": {
          "data": "...",
          "format": "wav"
        }
      }
    ]
  }
]

• transcribe the input audio and transform the messages into something a regular LLM can work with

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "What is in this recording?"
      },
      {
        "type": "text",
        "text": "A rainbow is an optical phenomenon caused by refraction, internal reflection and dispersion of light in water droplets resulting in a continuous spectrum of ..."
      }
    ]
  }
]

• Forward the transformed messages to an OpenAI-compatible endpoint the user specified in the config.
• Receive the response from an LLM, generate speech from it, and send it back to the user

transformed response with LLM text response + generated speech

Comments and suggestions are welcome!

[silvacarl2]

I think that one component as well to include in this picture is an LLM too. We are looking at Llama-3.2-3B-Instruct but fine tuning it for answering phone calls for a doctor's office.

[fedirz]

Request caching must be implemented to support multi-turn conversations.

[fedirz]

@silvacarl2 just wanted to let you know that a mix of audio, text, and image inputs is supported in v0.7.0

[jadams777]

Request caching must be implemented to support multi-turn conversations.

Is that part of the v0.7.0 release?

[fedirz]

No, I haven't added request caching for transcriptions yet. See https://speaches.ai/usage/voice-chat/#limitations

[jadams777]

Ok, that link is still down. Thanks.

…

On Fri, Jan 31, 2025 at 6:58 AM Fedir Zadniprovskyi < ***@***.***> wrote: No, I haven't added request caching for transcriptions yet. See https://speaches.ai/usage/voice-chat/#limitations — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARDSKD3KGT2SROEE6BEZOAD2NOFQ3AVCNFSM6AAAAABVATF6YOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRXGU2TAMJTGE> . You are receiving this because you commented.Message ID: ***@***.***>

[fedirz]

Ok, that link is still down. Thanks.
…

Fixed in #304. Sorry about that