BUG: reduction operations failing if min_count is larger

[galipremsagar]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

>>> import pandas as pd
>>> df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
>>> df
   x  y
0  1  4
1  2  5
2  3  6
>>> df.sum(min_count=10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/generic.py", line 11067, in sum
    self, axis, skipna, level, numeric_only, min_count, **kwargs
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/generic.py", line 10787, in sum
    "sum", nanops.nansum, axis, skipna, level, numeric_only, min_count, **kwargs
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/generic.py", line 10774, in _min_count_stat_function
    min_count=min_count,
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/frame.py", line 8847, in _reduce
    res, indexer = df._mgr.reduce(blk_func, ignore_failures=ignore_failures)
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 354, in reduce
    nbs = blk.reduce(func, ignore_failures)
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 400, in reduce
    nb = self.make_block(res_values)
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 286, in make_block
    return make_block(values, placement=placement, ndim=self.ndim)
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 2732, in make_block
    return klass(values, ndim=ndim, placement=placement)
  File "/nvme/0/pgali/envs/cudfdev/lib/python3.7/site-packages/pandas/core/internals/blocks.py", line 143, in __init__
    f"Wrong number of items passed {len(self.values)}, "
ValueError: Wrong number of items passed 1, placement implies 2

This actually works in 1.1.5:

>>> df
   x  y
0  1  4
1  2  5
2  3  6
>>> df.sum(min_count=10)
x   NaN
y   NaN
dtype: float64

Problem description

Reduction operations like sum, prod, etc.. are failing when min_count is larger in latest pandas, whereas this used to work in 1.1.5.

Expected Output

>>> df
   x  y
0  1  4
1  2  5
2  3  6
>>> df.sum(min_count=10)
x   NaN
y   NaN
dtype: float64

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.7.9.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-76-generic
Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.1
numpy : 1.19.5
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 49.6.0.post20210108
Cython : 0.29.21
pytest : 6.2.2
hypothesis : 6.1.1
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.20.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 0.8.5
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 1.0.1
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.52.0

[lithomas1]

@galipremsagar I can reproduce it on master, and can confirm that it is a regression. Thanks for the bug report(PRs and investigations very welcome).

[simonjayhawkins]

This actually works in 1.1.5:

first bad commit: [075ed8b] REF: handle axis=None case inside DataFrame.any/all to simplify _reduce (#35899) cc @jbrockmendel

The traceback after this commit was

2021-02-12T10:38:04.4627068Z Traceback (most recent call last):
2021-02-12T10:38:04.4628059Z   File "../39738.py", line 7, in <module>
2021-02-12T10:38:04.4628678Z     result = df.sum(min_count=10)
2021-02-12T10:38:04.4629486Z   File "/home/runner/work/pandas/pandas/pandas/core/generic.py", line 11350, in stat_func
2021-02-12T10:38:04.4630231Z     return self._reduce(
2021-02-12T10:38:04.4631038Z   File "/home/runner/work/pandas/pandas/pandas/core/frame.py", line 8721, in _reduce
2021-02-12T10:38:04.4631778Z     if is_object_dtype(result.dtype):
2021-02-12T10:38:04.4633085Z AttributeError: 'float' object has no attribute 'dtype'

https://github.com/simonjayhawkins/pandas/runs/1886279618?check_suite_focus=true

[simonjayhawkins]

The current path which gives ValueError: Wrong number of items passed 1, placement implies 2

first bad commit: [7d257c6] REF: ignore_failures in BlockManager.reduce (#35881)

[simonjayhawkins]

different regression, but also resulting from using different path

>>> import pandas as pd
>>>
>>> pd.__version__
'1.3.0.dev0+884.g4c5e6fac26'
>>>
>>> df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
>>> df
   x  y
0  1  4
1  2  5
2  3  6
>>>
>>> df.sum(min_count=5)
x     6
y    15
dtype: int64
>>>

The min count is now established from the 2d shape

[simonjayhawkins]

different regression, but also resulting from using different path

this was also occurring in 1.1.5, so not caused by same commit