Add RoaringBitmapArray to store indices of rows deleted in a data file #1486
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a new bitmap implementation called `RoaringBitmapArray`, which replaces usage of `Roaring64Bitmap` in deletion vectors. This implementation is optimized for use case of handling row indices, which are always clustered between 0 and the index of the last row number of a file, as opposed to being arbitrarily sparse over the whole `Long` space. The implementation is much simpler than `Roaring64Bitmap` and less error prone, as well as faster. GitOrigin-RevId: 862989348614520672065786c5607f0ade7a93e7
3 tasks
|
Can you crosslink to the issue in the description as well? |
Linked to the main issue which also contains the small project plan |
tdas
reviewed
Nov 16, 2022
| } | ||
|
|
||
| /** Add all values in `range` to the container. */ | ||
| def addRange(range: NumericRange[Long]): Unit = { |
There was a problem hiding this comment.
This is a lot of methods... do you think you will need all of these methods for the full DV implementation?
There was a problem hiding this comment.
These are going to be mostly used in tests when you can generate a custom DV for a given set of indices. Let me mark it as for tests only.
tdas
reviewed
Nov 16, 2022
core/src/main/scala/org/apache/spark/sql/delta/deletionvectors/RoaringBitmapArray.scala
Show resolved
Hide resolved
tdas
reviewed
Nov 16, 2022
core/src/main/scala/org/apache/spark/sql/delta/deletionvectors/RoaringBitmapArray.scala
Show resolved
Hide resolved
vkorukanti
changed the title
tdas
approved these changes
Nov 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is part of the feature: Support reading Delta tables with deletion vectors (more at #1485)
Adds a new bitmap implementation called
RoaringBitmapArray. This will be used to encode the deleted row indices. There already exists aRoaring64Bitmapprovided by theorg.roaringbitmaplibrary , but this implementation is optimized for use case of handling row indices, which are always clustered between 0 and the index of the last row number of a file, as opposed to being arbitrarily sparse over the wholeLongspace.How was this patch tested?
Unit tests
Does this PR introduce any user-facing changes?
No