Upgrading to Arrow 5 & Arrow Flight

[nevi-me]

Perspective uses arrow 1.0.1, but arrow is now on 5.0.0. Trying to use perspective with other projects that support Arrow (or Arrow itself) is quite hard because the IPC format has evolved since 1.0.1.

Are there plans to upgrade to a more recent version of Arrow?

[timkpaine]

This is in draft #1545

[texodus]

Trying to use perspective with other projects that support Arrow (or Arrow itself) is quite hard because the IPC format has evolved since 1.0.1.

I would like some more details on precisely what issues you've experienced. I am not certain (and I am having trouble finding a clear answer in the Arrow docs), but I suspect 1.0.1 and 5.0.0 use the same "format version" as they call it. The IPC format is magic-header encoded and should fail with a clear versioning error had the format changed (and it does not when we e.g. run Arrow 5.0.0 perspective test suite which uses 0.17.0 era Arrow files as test data).

#1545 is on a slow train to nowhere, as it currently segfaults in our Python test suite and I'm unaware of any performance, stability or feature improvements w.r.t. how it is used by Perspective. If there was a clear bug-or-feature driver for this change (e.g. if I am wrong about the format version), I can look at it again.