Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite from_list #863

Closed
wants to merge 2 commits into from
Closed

Rewrite from_list #863

wants to merge 2 commits into from

Conversation

cigrainger
Copy link
Member

@cigrainger cigrainger commented Feb 19, 2024

The idea here is to minimise our (de)serialisation overhead. We are traversing a list multiple times right now to go from an Elixir list to a Polars series.

This PR aims to:

  • Remove the list traversal on the Elixir side entirely
  • Minimise traversals on the Rust side

The latter requires a little explanation: we currently traverse the list when decoding, then again when we call from_vec. This moves it into a single traversal over Terms, and accumulates a Series directly from the iterator over the Term, decoding as we go. This allows us to pass an {:error, reason} tuple from the from_list function if there is an invalid type in the list, which gives us the nice error message we want and for which we're currently making a pass in Elixir.

Left to do:

  • Finish implementing this pattern for all from_list functions in series.rs
  • Implement a generic from_list with type inference in series.rs
  • Remove the type checking Elixir-side

Blockers:

@cigrainger cigrainger force-pushed the cg/from-list branch 5 times, most recently from 43fcbaf to 7a7bdbc Compare February 21, 2024 11:56
@josevalim
Copy link
Member

Closing this because #928 tackles most of it. We tried skipping vectors and building Series directly, but we didn't measure a meaningful gain for now.

@josevalim josevalim closed this Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants