Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to LogProbs retrieval by implementing the native Structured Output Parser from OpenAI #1223

Open
WittmannF opened this issue Nov 27, 2024 · 0 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request

Comments

@WittmannF
Copy link

Is your feature request related to a problem? Please describe.
I've been reading the previous pull requests and found that #916 has been deprecated in favor of #938 which calls structured outputs using tool call by setting strict to True. In theory this is great for compatibility with previous models, however, as reported in #742 , tool/function calls doesn't output logprobs. Here's an example:

import openai
import json

openai_client = openai.OpenAI()

completion = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    tools=[
        {
            "type": "function",
            "function": {
                "name": "ExtractUser",
                "description": "Correctly extracted `ExtractUser` with all the required parameters with correct types",
                "parameters": {
                    "properties": {
                        "name": {"title": "Name", "type": "string"},
                        "age": {"title": "Age", "type": "integer"},
                    },
                    "required": ["age", "name"],
                    "type": "object",
                },
            },
        }
    ],
    tool_choice={"type": "function", "function": {"name": "ExtractUser"}},
    logprobs=True,
    messages=[
        {"role": "user", "content": "Extract Jason is 25 years old"},
    ],
)
print(completion.choices[0].logprobs)

# Output: ChoiceLogprobs(content=None, refusal=None)

While using the native structured output parser from OpenAI will provide logprobs:

import os
from pydantic import BaseModel, field_validator
import openai

client = openai.OpenAI()

class User(BaseModel):
    name: str
    age: int

response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Extract John is 18 years old.",
        }
    ],
    response_format=User,
    logprobs=True,
)


response_logprobs = response.choices[0].logprobs.content
print(response_logprobs)

# output: [ChatCompletionTokenLogprob(token='{"', bytes=[123, 34], logprob=-1.9361265e-07, top_logprobs=[]), ChatCompletionTokenLogprob(token='name', bytes=[110, 97, 109, 101], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='":"', bytes=[34, 58, 34], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='John', bytes=[74, 111, 104, 110], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='","', bytes=[34, 44, 34], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='age', bytes=[97, 103, 101], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='":', bytes=[34, 58], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='18', bytes=[49, 56], logprob=0.0, top_logprobs=[]), ChatCompletionTokenLogprob(token='}', bytes=[125], logprob=0.0, top_logprobs=[])]

Describe the solution you'd like
Continue #916

Describe alternatives you've considered
Using the native structured outputs from open ai client (client.beta.chat.completions.parse)

@github-actions github-actions bot added documentation Improvements or additions to documentation enhancement New feature or request labels Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant