JSON in streaming mode is not compatible with openAIs

[Andreybest]

LocalAI version: 1.9.1 (helm)

Environment, CPU architecture, OS, and Version: RKE2, Linux local-ai-********-**** 5.4.0-121-generic #137-Ubuntu SMP 2022 x86_64 GNU/Linux

Describe the bug
While in streaming mode, returned JSON is not compatible with openAI, making libs dependent on those values - break (like langchain.js)

To Reproduce
Make request to /chat/completions or to /completions with streaming mode.

Expected behavior
JSONs responses should be same as in openai

Real behaviour

Completion:
LocalAIs:

{
  "object": "text_completion",
  "model": "ggml-gpt4all-j-v1.3-groovy.bin",
  "choices": [
    {
      "text": " Hall"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}
{
  "model": "ggml-gpt4all-j-v1.3-groovy.bin",
  "choices": [
    {
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

OpenAIs:

{
  "id": "cmpl-7TfSkTFsM81498l9eVd8ZIo8FPiYT",
  "object": "text_completion",
  "created": 1687305250,
  "choices": [
    {
      "text": "!",
      "index": 0,
      "logprobs": null,
      "finish_reason": null
    }
  ],
  "model": "text-ada-001"
}
{
  "id": "cmpl-7TfSkTFsM81498l9eVd8ZIo8FPiYT",
  "object": "text_completion",
  "created": 1687305250,
  "choices": [
    {
      "text": "",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "model": "text-ada-001"
}

Chat completions:

LocalAIs:

{
  "object": "chat.completion.chunk",
  "model": "ggml-gpt4all-j-v1.3-groovy.bin",
  "choices": [
    {
      "delta": {
        "content": "!"
      }
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}
{
  "model": "ggml-gpt4all-j-v1.3-groovy.bin",
  "choices": [
    {
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

OpenAIs:

{
  "id": "chatcmpl-7TfpCEV5eaJ07Q3lNlR31ds2FqVAI",
  "object": "chat.completion.chunk",
  "created": 1687306642,
  "model": "gpt-3.5-turbo-0301",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "."
      },
      "finish_reason": null
    }
  ]
}
{
  "id": "chatcmpl-7TfpCEV5eaJ07Q3lNlR31ds2FqVAI",
  "object": "chat.completion.chunk",
  "created": 1687306642,
  "model": "gpt-3.5-turbo-0301",
  "choices": [
    {
      "index": 0,
      "delta": {},
      "finish_reason": "stop"
    }
  ]
}

If we consider current langchain.js (0.0.96)
I get errors in completion for missing "index" in "choices"
And for chat completion, it's missing "delta" in "choices"

Code for reproducing
I've switched between LocalAI and OpenAI with comments
Completions:

import openai

# openai.api_base = "http://localhost:80/v1"

openai.api_key = "your-key"

prompt = "Say the sentence: Test it's testing, hello!"

model = "text-ada-001"
# model = "ggml-gpt4all-j-v1.3-groovy.bin"

# Make the API request
response = openai.Completion.create(
    model=model,
    prompt=prompt,
    max_tokens=50,
    temperature=0.28,
    top_p=0.95,
    n=1,
    echo=True,
    stream=True
)

for chunk in response:
    print(chunk)

print(response)

Chat completions:

import openai

# openai.api_base = "http://localhost:80/v1"

openai.api_key = "your-key"

model = "gpt-3.5-turbo"
# model = "ggml-gpt4all-j-v1.3-groovy.bin"

response = openai.ChatCompletion.create(
    model=model,
    messages=[
        {'role': 'user', 'content': "Write: I love peanutseries"}
    ],
    temperature=0,
    stream=True
)

for chunk in response:
    print(chunk)

P.S.
Thanks for a great work guys! You are doing absolutely wonderful thing, keep it up!

[Andreybest]

Thank you @mudler !!