Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for default exclusion of $vector and $vectorize #1016

Merged
merged 20 commits into from
Apr 9, 2024

Conversation

tatu-at-datastax
Copy link
Contributor

What this PR does:

Will change default projection inclusion to exclude $vector and $vectorize, needing explicit inclusion.

Which issue(s) this PR fixes:
Fixes #1005

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@tatu-at-datastax tatu-at-datastax self-assigned this Apr 2, 2024
@@ -191,36 +228,32 @@ static PathCollector collectPaths(JsonNode def, boolean includeSimilarity) {
}

public DocumentProjector buildProjector() {
if (isDefaultProjection()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caller will use the short-cut, no need to check whether explicit definition happens to work like default projection.

@@ -126,8 +126,7 @@ public void byIdAfterUpdate() {
"name": "Joe",
"age": 42,
"enabled": true,
"value": -1,
"$vector" : [ 0.5, -0.25 ]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplifying the test since $vector not really needed for test (but new default projection would drop)

Copy link
Collaborator

@vkarpov15 vkarpov15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far

@tatu-at-datastax tatu-at-datastax changed the title (WIP) Add support for default exclusion of $vector (WIP) Add support for default exclusion of $vector and $vectorize Apr 5, 2024
@tatu-at-datastax tatu-at-datastax changed the title (WIP) Add support for default exclusion of $vector and $vectorize Add support for default exclusion of $vector and $vectorize Apr 5, 2024
@tatu-at-datastax tatu-at-datastax marked this pull request as ready for review April 5, 2024 18:19
@tatu-at-datastax tatu-at-datastax requested a review from a team as a code owner April 5, 2024 18:19
paths,
slices,
// doc-id included unless explicitly excluded
!Boolean.FALSE.equals(idInclusion),
Copy link
Contributor

@Yuqi-Du Yuqi-Du Apr 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting.
So by default, idInclusion will be null, which will give addDocId as

  • true for inclusion-based projection
  • false for exclusion-based projection

So essentially the meaning of addDocId will be flipped, in terms of what projection is used inclusion-based or exclusion-based. The variable name addDocId (similarly add$vector, add$vectorize) gives some confusion to me initially, but it makes sense afterwards.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, naming is hard. Instead of "addDocId" could use "includeDocId" but that would be even more confusing.
If you (or anyone else) has better name suggestion would be happy to rename.

@Yuqi-Du
Copy link
Contributor

Yuqi-Du commented Apr 9, 2024

if we understand correctly,
"projection": {"name" : true, "age":false} will error out, since projection logic thinks this is inclusion-based
but since $vector and $vectorize are special cases and handled differently(not by List paths), so we still allow something like this ?
"projection": { "$vector": 1, "$vectorize" : 0}

@tatu-at-datastax
Copy link
Contributor Author

if we understand correctly, "projection": {"name" : true, "age":false} will error out, since projection logic thinks this is inclusion-based but since $vector and $vectorize are special cases and handled differently(not by List paths), so we still allow something like this ? "projection": { "$vector": 1, "$vectorize" : 0}

Correct: _id, $vector, $vectorize (and any future $-prefixed special fields) are special in that they can be used for any inclusion/exclusion case.

@tatu-at-datastax tatu-at-datastax merged commit 4ac5ffc into main Apr 9, 2024
2 checks passed
@tatu-at-datastax tatu-at-datastax deleted the tatu/1005-default-exclude-vector branch April 9, 2024 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Exclude $vector and $vectorize from the default Projection
3 participants