Python: modification time in add actions table is wrong

[wjones127]

Environment

Delta-rs version: 0.7.0 Python

Binding: Python

Environment:

• Cloud provider:
• OS: MacOS M1
• Other:

---

Bug

What happened:

What you expected to happen:

How to reproduce it:

import pandas as pd
from deltalake import DeltaTable, write_deltalake

example_df = pd.DataFrame({
    "part": ["a", "a", "b", "b"],
    "value": [1, 2, 3, 4]
})

write_deltalake(
    "example_table",
    example_df,
    partition_by=["part"],
    mode="overwrite"
)

table = DeltaTable("example_table")
table.get_add_actions(flatten=True).column("modification_time")

Outputs:

<pyarrow.lib.TimestampArray object at 0x137e994e0>
[
  1970-01-20 09:27:03.636,
  1970-01-20 09:27:03.636
]

More details:

We are likely interpreting the time resolution wrong somewhere along the way.

[guyrt]

Can I take this? Probably get a PR in by end of weekend.

Issue is in the data write. Note the precision of add timestamps vs remove and commitinfo timestamps.

{"add":{"modificationTime":1675743351}}
{"add":{"modificationTime":1675743351,}
{"remove":"deletionTimestamp":1675743351371, ... 
{"remove":{"deletionTimestamp":1675743351371, ...
{"commitInfo":{"operationParameters":{"mode":"Overwrite","partitionBy":"[\"part\"]"},"timestamp":1675743351373}}

[wjones127]

Please do!