Support returning logprobs in Predictor

[veronicalyu320]

This PR allows Predict to return logprobs of each token as part of Prediction.

Example usage

If you want logprobs:
Set logprobs=True and optionally top_logprobs (int between 0 and 20, see openai doc for details):

import dspy
from dspy import Predict

predict_instance = Predict("question -> answer")
lm = dspy.LM("gpt-4o-mini", logprobs=True)
dspy.configure(lm=lm)
result = predict_instance(question="Where is the Eiffel Tower located?")
print(result)

Output:

Prediction(
    answer='The Eiffel Tower is located in Paris, France, on the Champ de Mars near the Seine River.',
    logprobs={'content': [{'token': '[[', 'bytes': [91, 91], 'logprob': -2.1008714e-06, 'top_logprobs': []}, {'token': ' ##', 'bytes': [32, 35, 35], 'logprob': -1.9361265e-07, 'top_logprobs': []}, {'token': ' answer', 'bytes': [32, 97, 110, 115, 119, 101, 114], 'logprob': 0.0, 'top_logprobs': []}, {'token': ' ##', 'bytes': [32, 35, 35], 'logprob': -8.180258e-06, 'top_logprobs': []}, {'token': ' ]]\n', 'bytes': [32, 93, 93, 10], 'logprob': -0.00010926496, 'top_logprobs': []}, {'token': 'The', 'bytes': [84, 104, 101], 'logprob': 0.0, 'top_logprobs': []}, {'token': ' Eiffel', 'bytes': [32, 69, 105, 102, 102, 101, 108], 'logprob': 0.0, 'top_logprobs': []}, ...], 'refusal': None}
)

If you don't want logprobs:
The usage and behavior is the same as before:

... # everything else stays the same
lm = dspy.LM("gpt-4o-mini")
...

Output:

Prediction(
    answer='The Eiffel Tower is located in Paris, France, on the Champ de Mars near the Seine River.'
) 

Caveat

If an LM (e.g. o1-mini) doesn't take logprobs as argument and you still set dspy.LM(..., logprobs=True), you will get an error like openai does not support parameters: {'logprobs': True}, for model=o1-mini. Please check the corresponding LM documentation before setting this parameter.

[okhat]

This appears to change the return type of __call__ even if the user doesn't request logprobs? We normally return a list of strings. This seems to return a list of dicts even if logprobs=False.

[veronicalyu320]

SG, updated.