files_by_partitions() changed the return type from absolute to relative paths in 0.6.2

[jeisinge]

Environment

Delta-rs version: 0.6.2

Binding: python

Environment:

• Cloud provider: Databricks
• OS: Ubuntu
• Other:

---

Bug

What happened:
In previous versions <= 0.6.1, files_by_partitions returned the absolute path of the files. However, in 0.6.2, the relative path is returned.

What you expected to happen:
Minor version upgrades will not break API.

How to reproduce it:

  delta_lake_path = "/tmp/foo"
  delta_table = DeltaTable(delta_lake_path)
  parquet_files = delta_table.files_by_partitions(partition_filters)

In <= 0.61, parquet_files is an absolute path. In 0.6.2, it is relative to delta_lake_path.

More details:
Unclear if this has been resolved per #880 .

It sounds like https://delta-io.github.io/delta-rs/python/usage.html#custom-storage-backends could be used. However, this Delta Lake is on a mount, so it is unclear if this is needed as it appears as a local file to the system. More importantly, it was unexpected for the API to change between minor versions.

It is also worth noting that https://delta-io.github.io/delta-rs/python/usage.html#custom-storage-backends does not specify that the path will be absolute or relative.

[wjones127]

Thank you for pointing this out; we should not have changed that API.

[vsytchenko-exos]

Is there any news regarding the issue? Any plans to fix it?

[wjones127]

Sorry haven't gotten around to this, but I will look at this soon. IMO we have too many methods for getting files right now, and I'd like to par them down. Here's what I'm thinking:

• DeltaTable.files() -> returns relative paths. Add optional partition_filters argument.
• DeltaTable.file_uris() -> returns absolute URI. Add optional partition_filters argument.
• DeltaTable.files_by_partition() -> bring back to absolute paths, but also deprecate in favor of file_uris().
• DeltaTable.file_paths() -> delete. Has already been deprecated for a while.