Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix openai image schema bug #1268

Merged
merged 1 commit into from
May 13, 2024
Merged

Conversation

usamimeri
Copy link
Contributor

@usamimeri usamimeri commented May 13, 2024

User description

I try to run android assistant with gpt-4-turbo, but there was an error indicating the messages schema was incorrect:
"Invalid type for 'messages[1].content[1].image_url': expected an object, but got a string instead
by checking the openai-sdk doc https://platform.openai.com/docs/guides/vision and https://platform.openai.com/docs/api-reference/chat/create
"I found that the format {"type":"image_url","image_url":YOUR_IMAGE_URL} can be used on gpt-4-vision-preview, but not on gpt-4-turbo, whereas the format {"type":"image_url","image_url":{"url":YOUR_IMAGE_URL}} can be used on both."

Consequently, I update the schema for more compability.

Copy link

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Description updated to latest commit (82b4be5)

Copy link

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Review 🔍

⏱️ Estimated effort to review [1-5]

2, because the PR involves a simple and straightforward change in the data structure of an image URL within a single method. The change is localized and does not appear to affect other parts of the codebase.

🧪 Relevant tests

No

⚡ Possible issues

Data Type Consistency: Ensure that all calling functions or methods that use _user_msg_with_imgs are updated to handle the new object structure for image_url instead of a string. If not, this could lead to runtime errors.

🔒 Security concerns

No

Copy link

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Code Suggestions ✨

CategorySuggestion                                                                                                                                    Score
Possible bug
Correct the structure of the dictionary to properly format the image URL

The dictionary key for "image_url" is incorrectly nested within another dictionary. This
might lead to unexpected behavior when accessing the "url". The "image_url" key should
directly contain the "url" value.

metagpt/provider/base_llm.py [68]

-content.append({"type": "image_url", "image_url": {"url": url}})
+content.append({"type": "image_url", "image_url": url})
 
Suggestion importance[1-10]: 10

Why: The suggestion correctly identifies a critical issue where the dictionary structure for "image_url" was changed in a way that could break functionality. The suggested fix restores the correct dictionary structure, ensuring the code functions as intended.

10

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 0% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 70.20%. Comparing base (f201b2f) to head (82b4be5).

Files Patch % Lines
metagpt/provider/base_llm.py 0.00% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1268      +/-   ##
==========================================
- Coverage   70.22%   70.20%   -0.03%     
==========================================
  Files         316      316              
  Lines       18860    18860              
==========================================
- Hits        13245    13240       -5     
- Misses       5615     5620       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@better629
Copy link
Collaborator

lgtm

@better629 better629 merged commit da0e47b into geekan:main May 13, 2024
0 of 2 checks passed
@usamimeri usamimeri deleted the image_url_debug branch May 17, 2024 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants