-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add processors for generating embeddings, adding context to docs, submitting prompts #2008
base: main
Are you sure you want to change the base?
Conversation
type embeddingProcConfig struct { | ||
APIKey string `json:"apiKey" validate:"required"` | ||
Endpoint string `json:"endpoint" default:"https://api.openai.com/v1"` | ||
Model string `json:"model" validate:"required,inclusion=gpt-4|gpt-4-turbo|gpt-3.5-turbo|text-davinci-003|text-davinci-002|text-curie-001|text-babbage-001|text-ada-001"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so these are not embeddings models?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, I was too quick to copy-paste all the models.:)
Msg("got embeddings") | ||
|
||
for i, record := range records { | ||
record.Metadata[EmbeddingMetadataBase64] = embeddings.Data[i].EmbeddingBase64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These can be horribly large, my suggestion here is to:
- add the embedding to
.Payload.After
- include the model name used for the embedding.
Particularly 2, since there are others openai compat systems which may allow for different models to be used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, these can be very large (depending on the model) and I find that you can reduce the size of them with compression. All floats too. Base64 will not be very compressible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, especially about no. 2. As for 1, that would also mean that raw data records get transformed into structured data records. That might be unexpected for some destinations.
Description
Depends on: https://github.com/conduitio-labs/conduit-connector-weaviate/tree/haris/different-vectors.
Quick checks