Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamps having different timezone cannot be represented in the same field of a schema #5823

Open
berkaysynnada opened this issue Apr 1, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@berkaysynnada
Copy link
Contributor

Is your feature request related to a problem or challenge?

We should be able to handle the cases where timestamps with different timezone data are stored in the same field of a schema. However, Field struct holds a DataType info, and if we want to place a different timezone:
ArrowError(InvalidArgumentError("column types must match schema types, expected Timestamp(Millisecond, Some(\"+03:00\")) but found Timestamp(Millisecond, None) at column index 0"))

Describe the solution you'd like

Timezone check for timestamps can be ignored while matching columns with the schema

Describe alternatives you've considered

Additional context

@berkaysynnada berkaysynnada added the enhancement New feature or request label Apr 1, 2023
@tustvold
Copy link
Contributor

tustvold commented Apr 1, 2023

I don't believe this is possible with the arrow data model, a column must only contain data relative to a single timezone epoch (or none). That being said timezone aware parsing was added to arrow-rs in apache/arrow-rs#3795 and allows parsing a timestamp with a given timezone, into the timezone of the column, e.g. UTC, preserving the same instant in time

apache/arrow-rs#1936 is the related arrow-rs ticket to handle this in the cast kernel (in addition to the existing CSV and JSON support)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants