Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow token_logprobs and top_logprobs List to have Objects of None type #186

Merged
merged 2 commits into from
Jul 25, 2024

Conversation

narengogi
Copy link
Contributor

@narengogi narengogi commented Jul 23, 2024

Title:
Update types for token_logprobs and top_logprobs inside Completion Response

Description:

Motivation:
OpenAI SDK returns the response object as is, so there is no type checking by Pydantic, it's inferred on the client side and hence doesn't fail withe openAI SDK
Whereas portkey-sdk manipulates the object and Pydantic check fails

Example OpenAI Response

{
  "id": "cmpl-9nlcqIQdJrv3sqCf8L69Pn3LWrkMI",
  "object": "text_completion",
  "created": 1721648172,
  "model": "davinci-002",
  "choices": [
    {
      "text": "What is the current temperature in San Francisco? also set light brightness to 50 and color to warm white and these cool lights wont work in the car. It’s possible that your",
      "index": 0,
      "logprobs": {
        "tokens": [
          "What",
          " is",
          " the"
        ],
        "token_logprobs": [
          None,
          -1.8412793,
          -1.1794527,
        ],
        "top_logprobs": [
          None,
          {
            " is": -1.8412793,
            " are": -2.5893137
          },
          {
            " the": -1.1794527,
            " your": -2.4726083
          }], "
            text_offset
            ": [0, 4, 7, 11, 19, 31, 34, 38, 48, 49, 54, 58, 64, 75, 78, 79, 81, 85, 91, 94, 99, 105, 109, 115, 120, 127, 132, 137, 140, 144, 148, 149, 152, 154, 163, 168]}, "
            finish_reason
            ": "
            length
            "}], "
            usage
            ": {"
            prompt_tokens
            ": 20, "
            completion_tokens
            ": 16, "
            total_tokens
            ": 36}}

Related Issues:
Portkey-AI/gateway#463

Testing Done:

  • Tested by invoking chat.completions.create and completions.create on the client

@narengogi narengogi force-pushed the fix/logprobs-type-update branch from 7e39efd to 72a4f42 Compare July 23, 2024 06:26
@narengogi narengogi marked this pull request as ready for review July 23, 2024 06:42
@dosubot dosubot bot added size:XS This PR changes 0-9 lines, ignoring generated files. auto:enhancement labels Jul 23, 2024
@csgulati09 csgulati09 self-requested a review July 23, 2024 12:30
@dosubot dosubot bot added the lgtm label Jul 23, 2024
@VisargD VisargD merged commit 6f93abb into Portkey-AI:main Jul 25, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:enhancement lgtm size:XS This PR changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants