feat(python): allow python objects to be passed as new values in .update()

[ion-elgreco]

Description

A user can now add a new_values dictionary that contains python objects as a value.

Some weird behavior's I noticed, probably related to datafusion, updating a timestamp column has to be done by providing a unix timestamp in microseconds. I personally find this very confusing, I was expecting to be able to pass "2012-10-01" for example in the updates.

Another weird behaviour is with list of string columns. I can pass {"list_of_string_col":"[1,2,3]"} or {"list_of_string_col":"['1','2','3']"} and both will work. I expect the first one to raise an exception on invalid datatypes. Combined datatypes "[1,2,'3']" luckily do raise an error by datafusion.

Related Issue(s)

• closes update don't seems to be working #1740

[wjones127]

Looks like a nice interface. I have some suggestions to make some of the parameter handling clearer.

[ion-elgreco]

@wjones127 I've integrated all your feedback in the code

[wjones127]

One small nit on the docs, otherwise looks good :)

[ion-elgreco]

One small nit on the docs, otherwise looks good :)

Your approval got dismissed with the commit 😛

[franz101]

Am I correct this does not work with dictionaries?

data = pa.table({"x": [1, 2, 3], "y": [{"a":"a"}, {"a":"b"}, {"a":"c"}]},schema=dt.schema().to_pyarrow(False))
write_deltalake("tmp", data, mode="overwrite", overwrite_schema=True)
dt.update(predicate="id = 2",updates={"y": {"a":"banana"}})

Throws:
DeltaError: Generic DeltaTable error: Schema error: No field named id. Valid fields are x, y.

[ion-elgreco]

@franz101 if you want to use python types you need to use the new_values parameter however it doesn't supports dictionaries.