Fixes #767: Second part of "no-index" changes: allow large non-indexed String values, implicit allow for "$vector"

[tatu-at-datastax]

What this PR does:

Completes changes needed by #767 by:

1. Adding more testing for Shredder
2. Fix the issue wrt dropping indexing of "$vector" (needs handling similar to _id for query projections, i.e. always index)
3. Changing validation to ignore length of non-Indexed Strings (only limited by max doc size)

These are resolved by refactoring validation logic into smaller parts, run at different times during shredding process.

Which issue(s) this PR fixes:
Fixes #767

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CLA Signed: DataStax CLA

[vkarpov15]

Looks good so far 👍

[tatu-at-datastax]

Remaining issue: when returning "after" Document from findOneAndXxx() need to retain it without pruning for "no-index", may need to create a deep copy.

[maheshrajamani]

Overall looks good except for the clarification on $vector field

[tatu-at-datastax]

@maheshrajamani On $vector: I was thinking about first automatically adding it to allow list (sort of similar to _id) like you suggested, but since vector handling is different in other parts this seemed like slightly better way. I tried removing it automatically, but then noticed the problem with "after" document returning where it usually needs to be retained. But I think either way there is need for some special casing -- and if we need to support multiple vectors, this probably needs to be rewritten anyway.

But I wonder... if $vector is NOT automatically added, would it prevent use of $vector in filter clause?

I'll have to think about this bit more, thanks!

[tatu-at-datastax]

Sounds like implicit addition of $vector for non-empty "allow" -- similar to _id for filter Projection -- is what we'll be going with. Users can still add it to "deny" list if they really want to exclude it from being indexed; but that seems like an odd use case (why add $vector in Collection at all then?).

[tatu-at-datastax]

Need to make change for auto-inclusion of $vector, moving back to Draft.

[tatu-at-datastax]

Ready for re-review.

[tatu-at-datastax]

@maheshrajamani "$vector" handling changed as discussed: any other concerns? (I know you have approved but wanted to double-check before merging)

[maheshrajamani]

@maheshrajamani "$vector" handling changed as discussed: any other concerns? (I know you have approved but wanted to double-check before merging)

@tatu-at-datastax Changes LGTM

[vkarpov15]

LGTM