Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Micro batching for embedding clients #1122

Merged
merged 13 commits into from
May 29, 2024
Merged

Micro batching for embedding clients #1122

merged 13 commits into from
May 29, 2024

Conversation

maheshrajamani
Copy link
Contributor

@maheshrajamani maheshrajamani commented May 28, 2024 •
edited
Loading

What this PR does:
Micro batching for embedding clients

Which issue(s) this PR fixes:
Fixes #1078

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@maheshrajamani maheshrajamani self-assigned this May 28, 2024
@maheshrajamani maheshrajamani requested a review from a team as a code owner May 28, 2024 18:39
@maheshrajamani maheshrajamani changed the title [WIP]Micro batching for embedding clients Micro batching for embedding clients May 28, 2024
@maheshrajamani
Copy link
Contributor Author

Set the custom embedding provider with batch size 1. This should cover the testing of micro batching

tatu-at-datastax
tatu-at-datastax approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@tatu-at-datastax tatu-at-datastax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionally makes sense, looks good. I would prefer naming changes (more importantly actual Record type over Pair; less important "maxBatchSize" over "batchSize"), but let you decide if it's worth it.

@maheshrajamani maheshrajamani merged commit 2b1d285 into main May 29, 2024
3 checks passed
@maheshrajamani maheshrajamani deleted the micro-batching-embedding branch May 29, 2024 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tatu-at-datastax tatu-at-datastax tatu-at-datastax approved these changes

Assignees

@maheshrajamani maheshrajamani

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add (re-)batching for calls to external embedding providers
2 participants
@maheshrajamani @tatu-at-datastax