Python API - delta.appendOnly enforcement #590
Conversation
WarSame
changed the title
There was a problem hiding this comment.
Thanks for making this PR!
It looks like you've tested for the case where the configuration is passed while creating the table, but what about if the table already exists? That configuration should be saved with the table. To accomplish this, we should be able to read the existing table configuration from the metadata property:
delta-rs/python/deltalake/table.py
Line 54 in 4aad6aa
Dict[str, str], not List[str]. If you could correct that in this PR, it would be appreciated 👍 .)
|
Thank you @WarSame 👍 LGTM, ready to merge after fixing the formatting issue in the CI. |
|
Sorry for the delay. I've had Covid for the last week so this has taken the backburner. I believe I've fixed the formatting issues, though it appears the workflows need approval to run. I did add code to account for an existing table's metadata, and updated the tests to match that. Finally, it was getting to be a few lines of code to check these conditions so I pulled it out into its own functions. Please let me know if you prefer it inline. |
Oh no! Well I hope you are feeling better now.
Yeah that looks good to me. |
|
@WarSame Looks like you just need to sort the imports, and then this PR is good-to-go. 😄 |
|
@wjones127 I've done so and it should be good to go! |
* Add appendOnly check and test * Add .coverage to .gitignore * Readd metadata test, fix test * Whitespace * Remove unneeded concat * Shorten condition - is this less clear? * Fix configuration type annotation * Move code to func, test existing config * Fix formatting * Add error string match, remove config to rely on existing config * Fix linting for function * Rename function * Add version, add mode to error * Add mode/data_store_type itertools product * Sort imports
Description
Adding enforcement of
delta.appendOnlyto the Python API. Ifdelta.appendOnlyis passed in as a key to the configuration and the writingmodeis notappendthen the write will fail with a ValueError.Related Issue(s)
This deals with the first bullet point in #575 . The second bullet point about column invariants is not handled here, so this PR does not close the issue on its own.
Documentation
This configuration field is already covered in the
usage.rst-> "Examining a Table" -> "Metadata" section.