REGR: reduction operations failing if min_count is larger

[simonjayhawkins]

• closes BUG: reduction operations failing if min_count is larger #39738
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[simonjayhawkins]

i'll add a test and whatsnew shortly, if we want to go this route for backport purposes. This fix does not have much value in isolation as the regression #39738 (comment) is not yet addressed here.

[simonjayhawkins]

The regression this fixes is caused by

        if np.ndim(result) == 0:
            # TODO(EA2D): special case not needed with 2D EAs
            res_values = np.array([[result]])
        else:
            res_values = result.reshape(-1, 1)

in pandas/core/internals/blocks.py

this does not account for the fact that _maybe_null_out (and therefore nansum) may return a scalar.

nansum, _maybe_null_out and check_below_min_count need some work if they are to be called with 2d arrays, but this is outside the scope of a backport PR.

I still need to track down the earlier regression, where min_count above the row/column length but below the 2d shape does not work as intended, i.e. does nothing

[jreback]

i don't think we should block on this for 1.2.3. This patch may involve some non-trivial work.

[simonjayhawkins]

i don't think we should block on this for 1.2.3. This patch may involve some non-trivial work.

sure. a 1.2.4 milestone has been created. easily moved.

[jreback]

i don't think we should block on this for 1.2.3. This patch may involve some non-trivial work.

sure. a 1.2.4 milestone has been created. easily moved.

great just don't want to block (and i don't see anything urgent to prevent release).

[simonjayhawkins]

and i don't see anything urgent to prevent release

I was thinking maybe wait till #40144 is fixed. also affects backport. but we will likely have another release before the np-dev issues cause mainstream issues.

[jreback]

and i don't see anything urgent to prevent release

I was thinking maybe wait till #40144 is fixed. also affects backport. but we will likely have another release before the np-dev issues cause mainstream issues.

cool

[simonjayhawkins]

This fix does not have much value in isolation as the regression #39738 (comment) is not yet addressed here.

this seems to be a long standing behavior. This seems non-intuitive to me since I expected min_count to apply column-wise/row-wise if axis is specified.

[jreback]

thanks @simonjayhawkins

its possible we want to refactor this for 1.3, but for later, cc @jbrockmendel

[jreback]

@meeseeksdev backport 1.2.x