[BUG] Delta Lake metadata query detection can trigger extra file listing jobs

[jlowe]

isDeltaLakeMetadataQuery can invoke inputFiles on a FileSourceScanExec's relation, and on highly partitioned data sources this will often trigger a Spark job to do the listing of files in the table. Users have seen extra stages to do file listings appear that have been triggered by isDeltaLakeMetadataQuery. Setting spark.rapids.sql.detectDeltaLogQueries to false causes these extra stages to disappear.

[jlowe]

We may be able to do the metadata detection much cheaper by checking rootPaths on the FileIndex rather than inputFiles which probably would avoid doing anything really expensive. I suspect we'll see the special metadata directories in the rootPaths results on metadata queries without needing a full file listing, but this needs to be verified.

[zpuller]

I was able to try swapping out inputFiles for rootPaths , and it seems to work in that i can see the set of paths still looks the same in my queries, and the isDeltaLakeMetadataQuery check returns the same result. However, I don't see any evidence in the spark UI or otherwise that the new approach is performing less work. I tried running it against a 1000 partition delta table (with not much data).

Do we have a more direct or exact scenario or repro available to test against?

[jlowe]

The savings is tied to the relative costs of inputFiles vs. rootPaths. For many relations, it's much more expensive to produce the former rather than the latter. The former requires a recursive listing of directories, and on some platforms, that's pretty expensive (e.g.: cloud stores which typically do not like to do these things). Think of a partitioned table in the cloud with many thousands of partition directories and millions of files -- it's going to be expensive to gather all these input files.

If you're on a platform where listing directories is very cheap and there aren't many files to list, you'll probably not see much difference.