Pairwise correlation

[cristineguadelupe]

No description provided.

[josevalim]

Do we need it in the backend? I think the implementation within Explorer.DataFrame was better, no? Or is this faster in any way?

[philss]

@josevalim we did using the backend because I wanted to raise for the lazy version, since we cannot implement for that right now. Do you think it worth to revert it anyway?

[josevalim]

@philss maybe we can implement it with lazy by implementing it with a mutate + select/discard?

[josevalim]

This way it works with lazy and within Explorer.DataFrame as well! Something like:

mutate_with(df, fn ldf ->
  Enum.map(columns, ...)
end)
|> select(existing_columns -- new_columns, :discard)

WDYT?

[philss]

Oh, I'm going to try.

[philss]

@josevalim I think this won't work the way we want, because mutate_with expects only lazy_series/expressions as values, and we are trying to create it with a list of lazy series. On the other hand, we could try create a column for each pair and try to pivot the results later. But again, pivoting does not work with lazy frames.

Maybe there is another way to reshape this DF, but I don't know yet.
I'm going to investigate more tomorrow :)

[josevalim]

Hrm, now that you mention it, I think you are right. Feel free to ship it. :)

[josevalim]

Actually, don't ship it yet. I would like Chris' approval on the API before. :D

[cigrainger]

I like the API. Matches R's cor() pretty cleanly. One thing we should be noting in the docs is that this is the pearson correlation. We should (in a future PR, I'll raise a ticket) add a method opt and enable spearman https://pola-rs.github.io/polars/docs/rust/dev/polars_lazy/dsl/fn.spearman_rank_corr.html. I can't find kendall in the docs, but spearman would at least open things up for rank correlation.

[philss]

@cigrainger makes sense! Thanks! We added a note in the function description, just like we have with the Series counter part. Please let us know if this is enough.

[cigrainger]

Looks great!