-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add embedding support api #528
Conversation
Fixed the $vector update field
Fixed the $vector update field
...argate/sgv2/jsonapi/service/embedding/configuration/PropertyBasedEmbeddingServiceConfig.java
Outdated
Show resolved
Hide resolved
...argate/sgv2/jsonapi/service/embedding/configuration/PropertyBasedEmbeddingServiceConfig.java
Outdated
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/service/embedding/operation/EmbeddingServiceCache.java
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/service/embedding/operation/EmbeddingServiceCache.java
Outdated
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/service/embedding/operation/EmbeddingServiceCache.java
Outdated
Show resolved
Hide resolved
...in/java/io/stargate/sgv2/jsonapi/service/embedding/operation/HuggingFaceEmbeddingClient.java
Outdated
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/service/embedding/operation/OpenAiEmbeddingClient.java
Outdated
Show resolved
Hide resolved
...ain/java/io/stargate/sgv2/jsonapi/service/embedding/operation/test/TestEmbeddingService.java
Outdated
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/api/model/command/impl/CreateCollectionCommand.java
Show resolved
Hide resolved
src/main/java/io/stargate/sgv2/jsonapi/service/bridge/executor/NamespaceCache.java
Outdated
Show resolved
Hide resolved
@@ -15,17 +15,23 @@ public record CreateCollectionOperation( | |||
String name, | |||
boolean vectorSearch, | |||
int vectorSize, | |||
String vectorFunction) | |||
String vectorFunction, | |||
String vectorize) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a JSON serialization of Vectorize service configuration?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the value is stored part of table's comment.
@@ -66,6 +72,9 @@ protected QueryOuterClass.Query getCreateTable(String keyspace, String table) { | |||
+ vectorSize | |||
+ ">, " | |||
+ " PRIMARY KEY (key))"; | |||
if (vectorize != null) { | |||
createTableWithVector = createTableWithVector + " WITH comment = '" + vectorize + "'"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if it matters, but if vectorize
is serialized JSON it could still have characters that require escaping (esp. apostrophes themselves). Although I guess those would lead to CQL query failure, not corrupted data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what kind of characters to handle since this is just a text value, If needed I will add permitted character for the vectorize options.
src/main/java/io/stargate/sgv2/jsonapi/service/resolver/CommandResolverService.java
Outdated
Show resolved
Hide resolved
...va/io/stargate/sgv2/jsonapi/service/resolver/model/impl/CreateCollectionCommandResolver.java
Show resolved
Hide resolved
...stargate/sgv2/jsonapi/service/resolver/model/impl/CreateEmbeddingServiceCommandResolver.java
Show resolved
Hide resolved
...main/java/io/stargate/sgv2/jsonapi/service/resolver/model/impl/DeleteOneCommandResolver.java
Outdated
Show resolved
Hide resolved
...ain/java/io/stargate/sgv2/jsonapi/service/resolver/model/impl/InsertManyCommandResolver.java
Outdated
Show resolved
Hide resolved
...ava/io/stargate/sgv2/jsonapi/api/model/command/deserializers/SortClauseDeserializerTest.java
Show resolved
Hide resolved
src/test/java/io/stargate/sgv2/jsonapi/api/v1/VectorizeSearchIntegrationTest.java
Show resolved
Hide resolved
src/test/java/io/stargate/sgv2/jsonapi/api/v1/VectorizeSearchIntegrationTest.java
Show resolved
Hide resolved
src/test/java/io/stargate/sgv2/jsonapi/service/embedding/operation/TestEmbeddingService.java
Outdated
Show resolved
Hide resolved
...est/java/io/stargate/sgv2/jsonapi/service/embedding/operation/EmbeddingServiceCacheTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, looks good overall -- I added (quite) a few suggestions; feel free to use or ignore whatever makes (or doesn't make) sense.
{ | ||
"findOneAndUpdate": { | ||
"filter" : {"_id" : "id"}, | ||
"update" : {"$setOnInsert" : {"$vectorize" : "New York"}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the difference between this $setOnInsert and upper $set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When upsert option is set to true during update operation, we will create new document if no document found based on condition. setOnInsert will work when this a new document.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$setOnInsert
is only applied in case a new document is inserted, but not if one exists.
Used for things like adding id field(s), or creation-time.
What this PR does:
Support for vectorize method
Fixed the update operation for vector data
Which issue(s) this PR fixes:
Fixes #524
Checklist