feat: Improve the syntax for conditions with multiple predicates

[dangotbanned]

Fixes #2759

Throughout this PR I frequently reference polars.when, and I want to make sure any reviewers have the origin handy.

Tasks

Review

• Investigate strange SchemaValidationError feat: Improve the syntax for conditions with multiple predicates #3427 (comment)

  • Redesigned _Conditions -> _Conditional
  • See commit 1292962
  • Generic w/ constraint on the type of condition allowed, aligning with vega-lite

  In addition, there are two ways to encode the data that satisfy the specified condition:

  1. Combining one conditional field definition with a single base value or datum definition.
  2. Combining one or more conditional datum or value definitions with a field, datum, or value definition.

• Investigate @mattijn proposed str_as: Literal["value", "datum", "field", ...] as an alternative to str_as_lit feat: Improve the syntax for conditions with multiple predicates #3427 (comment)

  • str_as_lit=False -> str_as="shorthand"
  • See commit d0784f8
  • See thread feat: Improve the syntax for conditions with multiple predicates #3427 (comment)

Core

• Add more clearly defined types
  • Update alt.condition signature with overloads
• Extract existing multi-stage parsing logic into new functions
• Define utility classes for when-then-otherwise syntax
• Reuse/adapt parsing functions for utility classes
• Investigate existing function/method(s) that would accept the new conditions
  • Return of otherwise
    • Valid anywhere alt.condition is used
  • Return of then
    • Valid as encoding input dfc14de (#3427)
    • Identify other cases feat: Improve the syntax for conditions with multiple predicates #3427 (comment)
• Add top-level when function as entry-point
  • Rename _when -> when
• Refactor to reduce code duplication 38d425f (#3427)
• Add docs for when-then-otherwise
  • Initial docs, for: classes, most new types and functions
  • when
  • When.then
  • Then.when
  • Then.otherwise
  • ChainedWhen.then
• Add tests for when-then-otherwise
  • for passing case identified in Adapting Vega-Lite example "Bar Chart with Highlighting on Hover and Selection on Click" #3301
  • for _Then ending conditions
  • for mixed predicate input

Stretch

These would be nice to have and were demonstrated in #2759 (comment). Each idea comes from polars.when.

However, they need further discussion due to deviating from alt.condition's original behaviour.

• Wrap literals passed to then, otherwise in alt.value. 38d425f (#3427)
• Wrap kwargs passed to when as alt.datum.key.__eq__(value). 38d425f (#3427)
  • I found only 2 examples that used alt.condition(**kwargs), for something other than empty:
    • Selection Detail passes legend=None.
    • Interactive Scatter Plot and Linked Layered Histogram passes scale=alt.Scale(...).
  • Solved by moving **kwargs from when -> then, otherwise.
• Modify when's predicate argument to be variadic, reducing all predicates as an AND reduction. 38d425f (#3427)
  • Probably the least contentious
    • alt.condition already accepts the result of this.
    • when(predicate, **kwargs) -> when(*predicate, **kwargs) doesn't change the meaning of arguments other than predicate
      • A concern that I raised when proposing making this change within alt.condition

[dangotbanned]

👍😊

@mattijn All of #3427 (comment) is finished and ready for review.
Did a little bit of tidying up as well.

If you'd like to defer the docs part to another PR, I'd be happy with merging as-is.

[mattijn]

Thanks! Looks great! A few question for clarifications.

I was confused about these two new tests:

when_constraint = alt.when(Origin="Europe")

when_constraints = alt.when(
    Name="Name_1", 
    Color="Green", 
    Age=25, 
    StartDate="2000-10-01"
)

expected_constraint = alt.datum.Origin == "Europe"

expected_constraints = (
    (alt.datum.Name == "Name_1")
    & (alt.datum.Color == "Green")
    & (alt.datum.Age == 25)
    & (alt.datum.StartDate == "2000-10-01")
)

This is syntax I have not seen before. I noticed this is mentioned in the polars docs, that mentions this:

constraints
    Apply conditions as `col_name = value` keyword arguments that are
    treated as equality matches, such as `x = 123`. As with the predicates
    parameter, multiple conditions are implicitly combined using `&`.

And

Pass conditions as keyword arguments:

>>> df.with_columns(val=pl.when(foo=4, bar=0).then(99).otherwise(-1))
shape: (3, 3)
┌─────┬─────┬─────┐
│ foo ┆ bar ┆ val │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i32 │
╞═════╪═════╪═════╡
│ 1   ┆ 3   ┆ -1  │
│ 3   ┆ 4   ┆ -1  │
│ 4   ┆ 0   ┆ 99  │
└─────┴─────┴─────┘

Do you know where this type of syntax comes from? Is it unique for polars or also used elsewhere?

This only functions with a single = operator? Ie. a predicate like foo<4 is not supported? And what about alt.when('my column has spaces'=4)

[dangotbanned]

Thanks! Looks great! A few question for clarifications.

I was confused about these two new tests:

    when_constraint = alt.when(Origin="Europe")
    when_constraints = alt.when(
        Name="Name_1", Color="Green", Age=25, StartDate="2000-10-01"
    )
    expected_constraint = alt.datum.Origin == "Europe"
    expected_constraints = (
        (alt.datum.Name == "Name_1")
        & (alt.datum.Color == "Green")
        & (alt.datum.Age == 25)
        & (alt.datum.StartDate == "2000-10-01")
    )

This is syntax I have not seen before. I noticed this is mentioned in the polars docs, that mentions this:

    constraints
        Apply conditions as `col_name = value` keyword arguments that are treated as
        equality matches, such as `x = 123`. As with the predicates parameter, multiple
        conditions are implicitly combined using `&`.

And

    Pass conditions as keyword arguments:

    >>> df.with_columns(val=pl.when(foo=4, bar=0).then(99).otherwise(-1))
    shape: (3, 3)
    ┌─────┬─────┬─────┐
    │ foo ┆ bar ┆ val │
    │ --- ┆ --- ┆ --- │
    │ i64 ┆ i64 ┆ i32 │
    ╞═════╪═════╪═════╡
    │ 1   ┆ 3   ┆ -1  │
    │ 3   ┆ 4   ┆ -1  │
    │ 4   ┆ 0   ┆ 99  │
    └─────┴─────┴─────┘

Do you know where this type of syntax comes from? Is it unique for polars or also used elsewhere?

This only functions with a single = operator? Ie. a predicate like foo<4 is not supported? And what about alt.when('my column has spaces'=4)

@mattijn I'm confused also but for a different reason.
This has been in here for a while and it is described with an example in the doc for alt.when

altair/altair/vegalite/v5/api.py

Line 1035 in 87cd9bd

Examples

It was also part of the original proposal #2759 (comment)

I also listed it under the Stretch Tasks #3427 (comment)

It utilises **kwargs, so you can technically do it for columns with spaces like this:

alt.when(**{'my column has spaces':4})

I would maybe note this in the User Guide as an option, but I'd probably lean on a different feature for those cases.

Apologies, I'm doing this from my phone - but I'm pretty sure one of the later tests has this syntax of unpacking.

foo<4 will work if foo defines __lt__. It would be collected in *more_predicates, instead of **constraints.

I'll update this tomorrow with a more complete response, but appreciate you checking this out so quickly

[mattijn]

Great, this docstring you refer to is very clear. Thanks for linking.

It would be great if we can extend the documentation as well in a follow up PR on this new feature's scope to manage user expectations.

Maybe without realizing, but there have been many new things in this PR! Thanks for all these efforts.

I'm approving, but please comment when you think this PR can be merged!

[dangotbanned]

Great, this docstring you refer to is very clear. Thanks for linking.

It would be great if we can extend the documentation as well in a follow up PR on this new feature's scope to manage user expectations.

Maybe without realizing, but there have been many new things in this PR! Thanks for all these efforts.

I'm approving, but please comment when you think this PR can be merged!

@mattijn thank you, I'm ready for merging

Will absolutely be following up with another PR w/ more docs, especially examples on the methods (not just alt.when)

[mattijn]

Thank you @dangotbanned!