Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The idea here is to minimise our (de)serialisation overhead. We are traversing a list multiple times right now to go from an Elixir list to a Polars series.
This PR aims to:
The latter requires a little explanation: we currently traverse the list when decoding, then again when we call
from_vec
. This moves it into a single traversal overTerm
s, and accumulates aSeries
directly from the iterator over theTerm
, decoding as we go. This allows us to pass an{:error, reason}
tuple from thefrom_list
function if there is an invalid type in the list, which gives us the nice error message we want and for which we're currently making a pass in Elixir.Left to do:
from_list
functions inseries.rs
from_list
with type inference inseries.rs
Blockers:
This might have to wait on Bump Polars 0.37 #861 as Polars 0.36.2 can't build a string series fromPolars 0.37 can't either. I've raised this PR fix: ImplementOption<&str>
orOption<String>
. We can also just leave strings as inefficient and come back and fix that after.FromIterator
forSeries
for more types pola-rs/polars#14625.