feat(python): automagically upconvert with_columns kwarg expressions with multiple output names to struct; extend **named_kwargs support to select

[alexander-beedie]

Closes #6486, removes the "experimental" status from with_columns kwargs, and adds the same capability to select.

Example:

df = pl.DataFrame(
    data = {
        "x1": [1,2,6],
        "x2": [1,2,3],
    }
).with_columns(
    pct_change = pl.col(["x1","x2"]).pct_change(),
)
# ┌─────┬─────┬────────────────┐
# │ x1  ┆ x2  ┆ pct_change     │
# │ --- ┆ --- ┆ ---            │
# │ i64 ┆ i64 ┆ struct[2]      │
# ╞═════╪═════╪════════════════╡
# │ 1   ┆ 1   ┆ {null,null}    │
# │ 2   ┆ 2   ┆ {1.0,1.0}      │
# │ 6   ┆ 3   ┆ {2.0,0.5}      │
# └─────┴─────┴────────────────┘

[alexander-beedie]

@ritchie46: this is actually really nice behaviour... :)
Time to bring with_columns kwargs into the light for 0.16 and remove the experimental flag?

[ritchie46]

Does this work with pl.col(DATA_TYPE) and pl.col(REGEX)? How does this workout with pl.some_expanding_expr().suffix() and keep_name.

I think we need to add a function on the rust side of py-polars that will be called if **kwargs are used and checks if this expression is eligible for this, I must agree, nice feature.

We should also add some extra tests here that use lazy + projection / struct field expansion. Lazy must know the schema at all times.

[alexander-beedie]

Does this work with pl.col(DATA_TYPE) and pl.col(REGEX)? How does this workout with pl.some_expanding_expr().suffix() and keep_name.

Works like a charm for DATA_TYPE ...

df = pl.DataFrame( {"x1":[1,2,6], "x2":[1,2,3]} ).with_columns( ints=pl.col(pl.Int64) )
# ┌─────┬─────┬───────────┐
# │ x1  ┆ x2  ┆ ints      │
# │ --- ┆ --- ┆ ---       │
# │ i64 ┆ i64 ┆ struct[2] │
# ╞═════╪═════╪═══════════╡
# │ 1   ┆ 1   ┆ {1,1}     │
# │ 2   ┆ 2   ┆ {2,2}     │
# │ 6   ┆ 3   ┆ {6,3}     │
# └─────┴─────┴───────────┘

... and for "expanding" calls (with suffix or keep_name) ...

df = pl.DataFrame( {"x1":[1,2,6], "x2":[1,2,3]} ).with_columns( mins=pl.all().min().suffix("_min") )
# ┌─────┬─────┬───────────┐
# │ x1  ┆ x2  ┆ mins      │
# │ --- ┆ --- ┆ ---       │
# │ i64 ┆ i64 ┆ struct[2] │
# ╞═════╪═════╪═══════════╡
# │ 1   ┆ 1   ┆ {1,1}     │
# │ 2   ┆ 2   ┆ {1,1}     │
# │ 6   ┆ 3   ┆ {1,1}     │
# └─────┴─────┴───────────┘
df.unnest('mins')
# ┌─────┬─────┬────────┬────────┐
# │ x1  ┆ x2  ┆ x1_min ┆ x2_min │
# │ --- ┆ --- ┆ ---    ┆ ---    │
# │ i64 ┆ i64 ┆ i64    ┆ i64    │
# ╞═════╪═════╪════════╪════════╡
# │ 1   ┆ 1   ┆ 1      ┆ 1      │
# │ 2   ┆ 2   ┆ 1      ┆ 1      │
# │ 6   ┆ 3   ┆ 1      ┆ 1      │
# └─────┴─────┴────────┴────────┘

... but doesn't for REGEX (as the expr doesn't know if it will match one or more cols until evaluated).

[alexander-beedie]

I think we need to add a function on the rust side of py-polars

You read my mind ;) I don't like relying on the ComputeError to detect multiple output names - being able to use something like expr.meta.has_multiple_outputs() would feel much more solid.

[ritchie46]

... but doesn't for REGEX (as the expr doesn't know if it will match one or more cols until evaluated).

Then I think we must follow up with that checking function and then we can make it default, I think.

[alexander-beedie]

Side-note: how does one select column names that looks like regexes? 🤣

df = pl.DataFrame( {"^x.z$": [1,2,3]} )
# shape: (3, 1)
# ┌───────┐
# │ ^x.z$ │
# │ ---   │
# │ i64   │
# ╞═══════╡
# │ 1     │
# │ 2     │
# │ 3     │
# └───────┘

df.select("^x.z$")
# shape: (0, 0)
# ┌┐
# ╞╡
# └┘

[ritchie46]

Side-note: how does one select column names that looks like regexes? rofl

df = pl.DataFrame( {"^x.z$":[1, 2, 3]} )
# shape: (3, 1)
# ┌───────┐
# │ ^x.z$ │
# │ ---   │
# │ i64   │
# ╞═══════╡
# │ 1     │
# │ 2     │
# │ 3     │
# └───────┘

df.select("^x.z$")
# shape: (0, 0)
# ┌┐
# ╞╡
# └┘

Those column names are not allowed in polars. ;) Or you must use a regex escape as fallback.

The meta utilities are coming up btw.

[alexander-beedie]

Those column names are not allowed in polars. ;) Or you must use a regex escape as fallback.

Shall we actively prevent them? Could easily detect/raise inside col.
Probably a good idea, given the above, heh.

The meta utilities are coming up btw.

Fantastic :)

[ritchie46]

Those column names are not allowed in polars. ;) Or you must use a regex escape as fallback.

Shall we actively prevent them? Could easily detect/raise inside col. Probably a good idea, given the above, heh.

The meta utilities are coming up btw.

Fantastic :)

Yeah, I could add an assert there.

[gab23r]

Why do not automatically convert expression with multiple names into a struct when we used alias so that :

df = pl.DataFrame( {"x1":[1,2,6], "x2":[1,2,3]} ).with_columns(pl.col(pl.Int64).alias('ints'))

would work as well ?

I think this is a nice feature, It could be extend to other functions like select for example

[alexander-beedie]

Why do not automatically convert expression with multiple names into a struct when we used alias so that :

Hmm... It doesn't feel as intentional?
(Though perhaps my feeling there will change as more people discover this method ;)

I think this is a nice feature, It could be extend to other functions like select for example

Update: Done

[ritchie46]

One small remark.

Can we also add a structifying example to with_columns. This behavior deserves some documentation. :)

[alexander-beedie]

Can we also add a structifying example to with_columns. This behavior deserves some documentation. :)

A very good point :)
Will take care of that in a few hours when I'm back.

[alexander-beedie]

Done; added an extra docstring example/explanation, squashed, rebased...👌

[ritchie46]

Almost there @alexander-beedie. Maybe I missed it, but I believe we don't yet have added this functionality to select?

[alexander-beedie]

Ahh, you didn't miss it - apparently I missed that we were extending it there (though it makes perfect sense to do so!).

I'll be at the airport shortly - will see if there's time to do it before boarding - and, if not, I'm still hoping it might be possible to do a commit from 10km up 😂

[ritchie46]

Ahh, you didn't miss it - apparently I missed that we were extending it there (though it makes perfect sense to do so!).

Yeah, I want to keep those two consistent.

There is no hurry. The 0.16 release will still take few days. Have a good flight!

[ritchie46]

Alright. Here goes! 💯