when/then/otherwise silently converts values to nulls for Enum series

[mcrumiller]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Issue description

When an Enum is used in a when/then chain, and the result is not in the Enum's categories, the result is silently converted to null, when it should raise an Exception.

import polars as pl
from polars import col, when

df = pl.DataFrame({
    "a": pl.Series(["a", "b"], dtype=pl.Enum(["a", "b"]))
})
df.with_columns(
    when(col("a") == "a").then(col("a"))
    .otherwise(pl.lit("c"))
)

shape: (2, 1)
┌──────┐
│ a    │
│ ---  │
│ enum │
╞══════╡
│ a    │
│ null │
└──────┘

Installed versions

--------Version info---------
Polars:               0.20.14
Index type:           UInt32
Platform:             Windows-10-10.0.19045-SP0
Python:               3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           0.3.2
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.2
numpy:                1.26.2
openpyxl:             3.1.2
pandas:               2.1.4
pyarrow:              14.0.1
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.23
xlsx2csv:             0.8.2
xlsxwriter:           3.1.9

[mcrumiller]

Here is an open question: should the supertype of an enum be a categorical?

[c-peters]

No, I would not be in favor of casting Enum's to categoricals that would lose the purpose of the validation.

The goal of the Enum datatype is to provide validation and allow for fast operations on other Enum columns with the same encoding. This should raise an error since c is not a member of categories. I haven't had a time to look, but I think our recent change in non-strict casting might have caused this.

I will have a look later this week.

[mcrumiller]

No, I would not be in favor of casting Enum's to categoricals that would lose the purpose of the validation.

I agree, was just raising the question.

I haven't had a time to look, but I think our recent change in non-strict casting might have caused this.

Are you referring to #14910? That went into 20.15, this issue existed in 20.14 prior to that PR.

[c-peters]

I'm referring to #14728. I'm assuming there is a non strict cast somewhere that leads to the null