Delta-RS Saved Delta Table not properly ingested into Databricks

[niltecedu]

Environment

Delta-rs version:
0.18.2

Binding:
Not sure

Environment:

• Cloud provider:
  Azure Data Lake Gen2

• OS:
  Windows 11

• Other:

---

Bug

What happened:
Ingesting the delta table stored on the azure data lake, made by the deltalake pakage on azure databricks leads to databricks not picking up the delta tables properly; it loads up all of the parquet files instead of reading it as delta table.

What you expected to happen:
Only the latest version of the delta table is ingested

How to reproduce it:
Created a pandas dataframe, write to azure multiple times using overwrrite,

More details:
Creating a ticket with databricks as well to resolve the issue.

[ion-elgreco]

We need more info because this sounds like you are not reading the table correctly with spark

[niltecedu]

Heya, reading the table via databricks sql, loading it via deltatable package locally with pandas

Loading it in databricks sql

Loading it in databricks sql with distinct

The databricks sql also keeps the older version of the file which different rows

[niltecedu]

Nevermind, me being a knobhead. I saved to azure data lake with .parquet in the blobname

Fixing it gives me this for ingestion (I am quite new to databricks)

Closing this now thanks!