Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Date typing information from arrow lost after sample. #99

Closed
bmschmidt opened this issue Feb 11, 2021 · 2 comments · Fixed by #101
Closed

Date typing information from arrow lost after sample. #99

bmschmidt opened this issue Feb 11, 2021 · 2 comments · Fixed by #101

Comments

@bmschmidt
Copy link

bmschmidt commented Feb 11, 2021

Thanks again for this great library.

If you load a table using aq.fromArrow, date type information is preserved;
but after some operations, the dates revert to being represented as some kind of integers, not as datetime.

Apologies for not providing an example inline, because I'm not confident about generating
an Arrow date field in Javascript. In the notebook at https://observablehq.com/@bmschmidt/a-files,
you can see that date information is preserved after a filter

table.filter(d => d.country=="Mexico").select("birth date").view()

gives

birth date
--
1915-06-21T00:00:00.000Z
1900-04-14T00:00:00.000Z
1899-04-15T00:00:00.000Z
1886-04-12T00:00:00.000Z

But after a sample or slice, you get ints:

table.sample(10).select("birth date").view()

gives

birth date
--
-1841614848
1234978816
-275112960
689860608


@bmschmidt
Copy link
Author

bmschmidt commented Feb 11, 2021

I should clarify that this situation seems to hold with {unpack: true} and {unpack: false}.

I'm also confused about what these numbers mean -- I would have thought milliseconds since Unix epoch which arrow uses as backing, but they are not. (All of the people in that particular collection were born before 1920, but many of the integers are positive.)

@jheer
Copy link
Member

jheer commented Feb 11, 2021

Thanks! I think this is due to a reify bug I just fixed today over in #97. I will add some additional tests to confirm.

This was referenced Feb 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants