Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support binary image URLs without copying image data #853

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

lasse-it
Copy link

@lasse-it lasse-it commented Sep 18, 2024

Describe the change
This PR allows memory allocation to be optimised when providing a binary image in a ChatCompletionRequest.
It allows for providing the raw image bytes by reference, instead of having to copy the base64 encoded string.

Provide OpenAI documentation link
No API documentation is relevant.

Describe your solution
The solution makes it possible to provide any type of image URL and implements a BinaryImageURL struct that using MarshalJSON converts the raw image bytes to JSON in the most memory-effective way.

Tests

I've tested the changes using a memory profiler. I've profiled the memory allocation using a base64 encoded image URL and using the BinaryImageURL.

Using a 20mb picture, a 31,7% or 74 mb decrease in memory allocation can be seen for the provided tests.

Additional context
call tree
flamegraph

@lasse-it lasse-it marked this pull request as ready for review September 18, 2024 14:21
Copy link

codecov bot commented Sep 19, 2024

Codecov Report

Attention: Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 98.74%. Comparing base (774fc9d) to head (aa19584).
Report is 49 commits behind head on master.

Files with missing lines Patch % Lines
chat.go 73.33% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #853      +/-   ##
==========================================
+ Coverage   98.46%   98.74%   +0.28%     
==========================================
  Files          24       26       +2     
  Lines        1364     1437      +73     
==========================================
+ Hits         1343     1419      +76     
+ Misses         15       10       -5     
- Partials        6        8       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant