Arrow support #167
Arrow support #167
Conversation
Co-authored-by: Curtis Vogt <curtis.vogt@gmail.com>
|
I'm expecting Julia 1.0 to fail as ArrowTypes doesn't support that version of Julia currently |
| t = Arrow.Table(Arrow.tobuffer(table)) | ||
|
|
||
| # Arrow.jl converts all Period types into Second | ||
| @test_broken eltype(t.col) == HourEnding{ZonedDateTime, Open, Closed} |
There was a problem hiding this comment.
I'll probably end up disabling AnchoredInterval support to start with because of this
There was a problem hiding this comment.
We can't explicitly override this behaviour? Given how much data we have saved in HE format needing to save hour data as Intervals would be a significant performance hit and likely wouldn't make this PR worth it for us. Also, doesn't Arrow have it's own interval type which does the same thing?
There was a problem hiding this comment.
I've forgotten a bunch about this PR but it should be possible to encode the period span in the serialized Arrow form. I do remember I was mainly focused on Interval support so I think this comment was more about that the implementation currently in place is incomplete.
Codecov Report
@@ Coverage Diff @@
## master #167 +/- ##
==========================================
+ Coverage 81.09% 82.18% +1.09%
==========================================
Files 11 12 +1
Lines 603 640 +37
==========================================
+ Hits 489 526 +37
Misses 114 114
Continue to review full report at Codecov.
|
Fixing this in: apache/arrow-julia#223 |
|
Should probably wait for apache/arrow-julia#229 which allows us some additional flexibility with how we encode |
Replaces #156. Changes here include the commits from that PR and uses the updated serialization framework introduces in apache/arrow-julia#156