Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Support for Timestamp datatype. #1032

Merged
merged 24 commits into from
Jun 14, 2023
Merged

[FEAT] Support for Timestamp datatype. #1032

merged 24 commits into from
Jun 14, 2023

Conversation

xcharleslin
Copy link
Contributor

@xcharleslin xcharleslin commented Jun 10, 2023

Adds support for an Arrow-like Timestamp datatype in Daft. Timestamp type is parameterized by (TimeUnit, TimeZone).

Storage format

  • TimeUnit and TimeZone are stored in the field only. They are stored in the Arrow format (TimeUnit enum, TimeZone Option).
  • Timestamps are stored in the Arrow format: i64 representing number of TimeUnits from the UNIX epoch.

String representation

  • All timestamps are printed in ISO 8601.
  • The number of subsecond digits printed reflects the resolution of the timestamp type.
  • All timezones are printed in offset format (+-XX:XX) regardless of the timezone.

Ingestion

  • Lossless ingestion from PyArrow tables.
  • Ingestion from Python datetime.datetime to Timestamp(us, tz).

Casting

  • Cast to and from integer types.
  • Cast to and from String type (ISO 8601 only)

Expressions

  • Blackbox

Drivebys

Updated arrow_cast for Daft Physical -> Daft Logical.

  • Previously, casted directly to Arrow Physical.
  • Updated, casts to Arrow Logical and then to Arrow Physical.

Separate PR:

  • Numeric expressions (requires Duration type)

@xcharleslin xcharleslin changed the title Charles/timestamp Support for Timestamp datatype. Jun 13, 2023
@xcharleslin xcharleslin marked this pull request as ready for review June 14, 2023 00:48
@jaychia jaychia changed the title Support for Timestamp datatype. [FEAT] Support for Timestamp datatype. Jun 14, 2023
Copy link
Member

@samster25 samster25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

src/array/ops/as_arrow.rs Show resolved Hide resolved
tests/dataframe/test_temporals.py Outdated Show resolved Hide resolved
@github-actions github-actions bot added the enhancement New feature or request label Jun 14, 2023
@xcharleslin xcharleslin enabled auto-merge (squash) June 14, 2023 19:45
@xcharleslin xcharleslin merged commit 6206c92 into main Jun 14, 2023
@xcharleslin xcharleslin deleted the charles/timestamp branch June 14, 2023 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants