refactor(rust): add LazyFileListReader trait #6937
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The new trait
LazyFileListReader
deduplicates glob logic in the polars-lazy crate.It makes much easier to add glob support to other formats, see ndjson #6638 issue.
Breaking changes
NO breaking changes are intended in this PR.
I've assumed that
_impl
methods are not part of the public API (if that's not the case, it's trivial to fix).Some glob error messages are streamlined among file formats.
API philosophical discussion
There is a small inconsistency in polars-lazy, i.e. there are 2 reader conventions:
Scan
Reader
I've chosen (arbitrarily) the second one for the design of the trait... 🤔
The first one might be potentially more elegant with factory metaphor: