[Design Revisit] Projection handling of $vector

[maheshrajamani]

$vector field is designed to come in the json document. Since this data is stored as separate field in the physical table, do this need to be stored as part of doc_json field? If it's not to be stored in doc_json, need to handle this for projection.

[sync-by-unito]

➤ Mahesh Rajamani commented:

Aaron Morton Can you opine on this?

[sync-by-unito]

➤ Mahesh Rajamani commented:

Aaron Morton Do we need to revisit this?

[sync-by-unito]

➤ Aaron Morton commented:

I think we should keep it in the doc_json even though it is a duplication.

The reason is that if we drive a CDC process off the table, and the entire document is in the doc_json field, we can easily just grab that one field. There may be other situations where we want to be able rebuild the document, and having the doc_json field only partially store the document would make that more error prone.

With my long term thinking hat on, we have a general rule that we only SELECT the doc_json field the other fields that store data extracted from the doc_json field are only used in the WHERE (and ORDER) fields - we only use the SAI indexes of those fields. If we keep this rule, we can make storage optimizations to avoid storing the column value and only store it in the SAI index. We BREAK this rule with the current implementation of sorting, but it would be a great way to reduce the on disk size.