Rename numBytesAdded/Removed metrics and add deletion vector metrics in Databricks 12.2 shims [databricks] #8624
Conversation
Signed-off-by: Andy Grove <andygrove@nvidia.com>
| metrics_to_remove = ["executionTimeMs", "numOutputBytes", "rewriteTimeMs", "scanTimeMs", | ||
| "numRemovedBytes", "numAddedBytes", "numTargetBytesAdded", "numTargetBytesInserted", | ||
| "numTargetBytesUpdated", "numTargetBytesRemoved", | ||
| "numDeletionVectorsAdded", "numDeletionVectorsRemoved"] |
There was a problem hiding this comment.
It is not ideal to just ignore the deletion vector metrics. I am still trying to find a better solution.
There was a problem hiding this comment.
as @jlowe noted in #8628 (comment):
"Seems like there's two phases to supporting deletion vectors. There's full support which is tracked by #8554, but before that is completed we should be reporting the metrics (always 0) in the delta log stats just like the CPU does when not using deletion vectors. I could see shipping without full deletion support, but we should be adding these deletion metrics even when we don't support them being non-zero. I'm OK with that being a followup, but IMO having the metric is a P0 for the release while the full support issue is not."
andygrove
changed the title
|
build |
|
build |
|
build |
sameerz
added
the
task
Closes #8423 (this PR fixes the final issue and removes the last references to this issue)
Databricks 12.2 renames the
numBytes[Added|Removed]metrics in the Delete command to num[Added|Removed]Bytes for consistency with OSS Delta Lake. It also adds newnumDeletionVectors[Added|Removed]metrics.Changes in this PR: