Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace inputFiles with location.rootPaths.toString #11323

Merged

Conversation

zpuller
Copy link
Collaborator

@zpuller zpuller commented Aug 13, 2024

Fixes #9604

When evaluating a FileSourceScanExec within the isDeltaLakeMetadataQuery check, use .relation.rootPaths instead of .inputFiles to get the list of filenames being scanned.

Tested locally and verified the relevant paths still show up for matching delta queries.

Signed-off-by: Zach Puller <zpuller@nvidia.com>
@zpuller
Copy link
Collaborator Author

zpuller commented Aug 13, 2024

build

@zpuller zpuller merged commit 5302651 into NVIDIA:branch-24.10 Aug 16, 2024
43 of 44 checks passed
@zpuller zpuller deleted the delta_lake_meta_query_root_paths branch August 19, 2024 16:31
@sameerz sameerz added the performance A performance related task/issue label Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance A performance related task/issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Delta Lake metadata query detection can trigger extra file listing jobs
3 participants