feat: adopt kernel schemas and improve protocol support #1756
Conversation
github-actions
bot
added
binding/python
rust/src/kernel/actions/arrow.rs
Outdated
| } | ||
| PrimitiveType::Timestamp => { | ||
| // Issue: https://github.com/delta-io/delta/issues/643 | ||
| Ok(ArrowDataType::Timestamp(TimeUnit::Microsecond, None)) |
There was a problem hiding this comment.
What about timestamps that were written with the UTC timezone? Are we doing anything with the IsAdjustedToUtc flag in the parquet timestamp column to return the arrow datatype as Timestamp(tz='UTC')
There was a problem hiding this comment.
right now this is not handled. Within delta this is also gated behind the timestampNtz feature. As this PR is quite hard to review due to its size, I focussed on not adjusting any business logic but rather only update the schema / action definitions and getting things to build. So support for that feature should be in a follow-up PR.
|
@wjones127 @rtyler - There are some failing python tests related to how we handle schema metadata, where we need to make a decision on the kernel side. Other then that, I think it's ready for review. Since this PR is again quite sizable, I focussed on not making any changes to the logic and limit impact on the APIs. As such no new feature support in this PR :). |
rust/src/kernel/schema.rs
Outdated
| #[derive(Debug, Serialize, Deserialize, PartialEq, Clone)] | ||
| #[serde(rename_all = "camelCase")] | ||
| /// Primitive types supported by Delta | ||
| pub enum PrimitiveType { |
|
@wjones127 - made display for primitive types consistent with protocol...
This is an open question to me and mainly motivated by not wanting Raised this in the kernel sync - i.e. does delta metadata support nested structures? - but jury is still out in that :). As for going forward here I would appreciate your opinion. Continuing to use |
Absent any evidence, I don't know either. My slight inclination is to move toward strings. |
I share that inclination, so just went ahead and made the change :). |
Description
this PR depends on #1741.Migrating the implementation of actions and schema over from kernel. The schema is much more complete in terms of the more recent delta features and more rigorously leverages the rust type system.
Related Issue(s)
Documentation