Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: maximum of pd.Series([np.nan],dtype=ordered_category) raise #33450

Closed
2 of 3 tasks
mizuy opened this issue Apr 10, 2020 · 7 comments · Fixed by #33513
Closed
2 of 3 tasks

BUG: maximum of pd.Series([np.nan],dtype=ordered_category) raise #33450

mizuy opened this issue Apr 10, 2020 · 7 comments · Fixed by #33513
Labels
Categorical Categorical Data Type Regression Functionality that used to work in a prior pandas version
Milestone

Comments

@mizuy
Copy link

mizuy commented Apr 10, 2020

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

In [1]: pd.__version__
Out[1]: '1.0.3'

In [2]: pd.Series([np.nan],dtype=pd.CategoricalDtype([0,1],ordered=True)).max()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-5a47d189b696> in <module>
----> 1 pd.Series([np.nan],dtype=pd.CategoricalDtype([0,1],ordered=True)).max()

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pandas/core/generic.py in stat_func(self, axis, skipna, level, numeric_only, **kwargs)
  11213             return self._agg_by_level(name, axis=axis, level=level, skipna=skipna)
  11214         return self._reduce(
> 11215             f, name, axis=axis, skipna=skipna, numeric_only=numeric_only
  11216         )
  11217

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pandas/core/series.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   3870
   3871         if isinstance(delegate, Categorical):
-> 3872             return delegate._reduce(name, skipna=skipna, **kwds)
   3873         elif isinstance(delegate, ExtensionArray):
   3874             # dispatch to ExtensionArray interface

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pandas/core/arrays/categorical.py in _reduce(self, name, axis, **kwargs)
   2123         if func is None:
   2124             raise TypeError(f"Categorical cannot perform the operation {name}")
-> 2125         return func(**kwargs)
   2126
   2127     @deprecate_kwarg(old_arg_name="numeric_only", new_arg_name="skipna")

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)
    212                 else:
    213                     kwargs[new_arg_name] = new_arg_value
--> 214             return func(*args, **kwargs)
    215
    216         return cast(F, wrapper)

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pandas/core/arrays/categorical.py in max(self, skipna)
   2188         if not good.all():
   2189             if skipna:
-> 2190                 pointer = self._codes[good].max()
   2191             else:
   2192                 return np.nan

~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/numpy/core/_methods.py in _amax(a, axis, out, keepdims, initial, where)
     28 def _amax(a, axis=None, out=None, keepdims=False,
     29           initial=_NoValue, where=True):
---> 30     return umr_maximum(a, axis, None, out, keepdims, initial, where)
     31
     32 def _amin(a, axis=None, out=None, keepdims=False,

ValueError: zero-size array to reduction operation maximum which has no identity

In the older version, the same code didn't raise an error.

In [10]: pd.__version__
Out[10]: '0.25.3'

In [11]: pd.Series([np.nan],dtype=pd.CategoricalDtype([0,1],ordered=True)).max()
    ...:
Out[11]: nan

Problem description

Because of this behavior, I failed to df.groupby().max() for ordered categories.

Expected Output

Expected output should be np.nan

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.0.3
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.0
pip : 20.0.2
setuptools : 40.8.0
Cython : None
pytest : 5.2.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.2.2
lxml.etree : 4.4.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.2
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : 0.15.0
pytables : None
pytest : 5.2.1
pyxlsb : None
s3fs : 0.2.2
scipy : 1.3.1
sqlalchemy : None
tables : None
tabulate : 0.8.5
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.2.2
numba : None

@mizuy mizuy added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 10, 2020
@dsaxton dsaxton added Regression Functionality that used to work in a prior pandas version Categorical Categorical Data Type and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 10, 2020
@dsaxton
Copy link
Member

dsaxton commented Apr 10, 2020

Thanks, looks like the bug is on master as well

@jorisvandenbossche
Copy link
Member

Adding this to the 1.0.4 milestone, as this is a candidate for backporting in case we start doing that

@jreback
Copy link
Contributor

jreback commented Apr 13, 2020

let’s not add to the 1.04

there are a number of candidates but it’s going to take dedicated work to patch that branch

@jreback jreback modified the milestones: 1.0.4, 1.1 Apr 13, 2020
@jreback
Copy link
Contributor

jreback commented Apr 13, 2020

just add the regression tags

@jorisvandenbossche jorisvandenbossche modified the milestones: 1.1, 1.0.4 Apr 13, 2020
@jorisvandenbossche
Copy link
Member

Milestone is IMO much easier to keep track with. When we decide to not do a 1.0.4 release, we can simply move those to 1.1

@jreback
Copy link
Contributor

jreback commented Apr 13, 2020

that boat has sailed

and it’s plain confusing

milestone reflects what it is merged into not what you want

@simonjayhawkins
Copy link
Member

regression in #27929 (i.e. 1.0.0)

37526c1 is the first bad commit
commit 37526c1
Author: Mak Sze Chun makszechun@gmail.com
Date: Mon Dec 2 18:37:11 2019 +0800

API/DEPR: Change default skipna behaviour + deprecate numeric_only in Categorical.min and max (#27929)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Categorical Categorical Data Type Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants