Adds an ArrayDirectory class to manage all URIs within the array directory #2909
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stavrospapadopoulos
requested review from
Shelnutt2,
KiterLuc and
eric-hughes-tiledb
stavrospapadopoulos
force-pushed
the
sp/array_directory
branch
from
5415fdc
to
d10bd3c
Compare
…tory. This introduces several performance improvements, especially around redundant URI listings, parallelizing URI listings, etc. Also makes VFS::ls a noop for POSIX and HDFS when the listed directory does not exist instead of throwing an error, matching the functionality of the object stores. Finally, it removes partial vacuuming, as that leads to incorrect behavior with time traveling.
stavrospapadopoulos
force-pushed
the
sp/array_directory
branch
from
d10bd3c
to
df33d7b
Compare
KiterLuc
approved these changes
Feb 25, 2022
| /* API */ | ||
| /* ********************************* */ | ||
|
|
||
| const URI& ArrayDirectory::array_uri() const { |
There was a problem hiding this comment.
Potentially rename to uri()
Shelnutt2
reviewed
Feb 25, 2022
Shelnutt2
approved these changes
Feb 25, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an
ArrayDirectoryclass to manage all URIs within the array directory. This introduces several performance improvements, especially around removing redundant URI listings, parallelizing URI listings, etc. It also paves the way for better format versioning, especially when we need to shuffle files around in the array directory for better performance in the future (there is an upcoming PR for that).Notes:
VFS::lsa noop for POSIX and HDFS when the listed directory does not exist instead of throwing an error, matching the functionality of the object stores.ArrayDirectory, but the unit tests are missing. This is because at the moment the class just incorporated practically existing code (moved fromStorageManagerand optimized). If there is anything wrong with the class at the moment, all the tests will break (as it affects loading fragments, schemas, metadata, etc). Moreover, this class will be enhanced in an upcoming PR that will move all URI creations from the writer and array schema classes inArrayDirectory. Therefore, we will add proper unit tests in that PR.TYPE: IMPROVEMENT
DESC: Adds an ArrayDirectory class to manage all URIs within the array directory.