Minimum of ordered categorical data in Panda DataFrames

[Guillaume1801]

I have a Pandas DataFrame with one Serie containing ordered Categorical data. Some value of this Serie may be missing (NaN). I want to get the minimum without taking into account NaNs but I obtained strange results ...

Code:

raw_cat = pd.Categorical(["a", "b", "c", "a"],
                         categories=["b", "c", "d"],
                         ordered=True)
s = pd.Series(raw_cat)
raw_cat.min(numeric_only=True), s.min(numeric_only=True)

Output:

('b', nan)

Expected utput:

('b', 'b')

I am getting the desired output when running this code with pandas 0.23.4 but not with pandas 0.24.0 and above.

Is this an issue or a misunderstanding? Thank you for your help.

[jorisvandenbossche]

Thanks for the report! I can confirm this regression.

[jorisvandenbossche]

So it seems that the numeric_only keyword is no longer properly passed through to the Categorical.min implementation. Investigation welcome!

[arnov]

I dove into this a bit, but shouldn't the argument be skipna? Because I am unsure what numeric_only would mean for a categorical series.

[Guillaume1801]

I agree with you ! It makes no sense to use this argument while the argument used to removed NaNs in all other Pandas' methods is skipna ...

[arnov]

To add to the confusion, Categorical supports the dropna argument in the mode method, while it seems to be skipna in a lot of other places.

[jorisvandenbossche]

@arnov I actually thought exactly the same when answering on this issue, so I opened #25303 (but forgot to link to it here).

So I agree that skipna is more logical, but I don't think we can't simply change it as you did in #25304, we will have to deprecate the keyword and behaviour first.

Short term, I think it would be good to "just" fix it using numeric_only (so we can include this for 0.24.2), and then for 0.25.0 we could think about deprecating it. But let's first discuss that in #25303

[jreback]

closed by #25304