CI: NumPy logical reductions (any, all) fail on DataFrame with NumPy master

[TomAugspurger]

https://travis-ci.org/pandas-dev/pandas/jobs/348215410#L2717

[gw1] linux -- Python 3.6.4 /home/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.frame.test_analytics.TestDataFrameAnalytics object at 0x7f86fa5d54a8>
axis = 0
    @pytest.mark.parametrize("axis", [0, 1, None])
    def test_clip_against_frame(self, axis):
        df = DataFrame(np.random.randn(1000, 2))
        lb = DataFrame(np.random.randn(1000, 2))
        ub = lb + 1
    
>       clipped_df = df.clip(lb, ub, axis=axis)
pandas/tests/frame/test_analytics.py:2066: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/generic.py:5605: in clip
    if np.any(pd.isnull(lower)):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = 0    False
1    False
dtype: bool
    def __nonzero__(self):
        raise ValueError("The truth value of a {0} is ambiguous. "
                         "Use a.empty, a.bool(), a.item(), a.any() or a.all()."
>                        .format(self.__class__.__name__))
E       ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[TomAugspurger]

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: np.any(pd.DataFrame({"A": [False], "B": [False]}))
Out[3]:
A    False
B    False
dtype: bool

In [4]: np.__version__
Out[4]: '1.15.0.dev0+400607b'

Out[3] was previously False

Looking through NumPy commits, this was possibly numpy/numpy@b344da9

If I pass keepdims, pandas raises an exception

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-94458d898bc2> in <module>()
----> 1 np.any(pd.DataFrame({"A": [False], "B": [False]}), keepdims=False)

~/Envs/pandas-dev/lib/python3.6/site-packages/numpy/numpy/core/fromnumeric.py in any(a, axis, out, keepdims)
   1989
   1990     """
-> 1991     return _wrapreduction(a, np.logical_or, 'any', axis, None, out, keepdims=keepdims)
   1992
   1993

~/Envs/pandas-dev/lib/python3.6/site-packages/numpy/numpy/core/fromnumeric.py in _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs)
     79                 return reduction(axis=axis, dtype=dtype, out=out, **passkwargs)
     80             else:
---> 81                 return reduction(axis=axis, out=out, **passkwargs)
     82
     83     return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

~/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/core/generic.py in logical_func(self, axis, bool_only, skipna, level, **kwargs)
   7984     def logical_func(self, axis=None, bool_only=None, skipna=None, level=None,
   7985                      **kwargs):
-> 7986         nv.validate_logical_func(tuple(), kwargs, fname=name)
   7987         if skipna is None:
   7988             skipna = True

~/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/compat/numpy/function.py in __call__(self, args, kwargs, fname, max_fname_arg_count, method)
     48                 validate_args(fname, args, max_fname_arg_count, self.defaults)
     49             elif method == 'kwargs':
---> 50                 validate_kwargs(fname, kwargs, self.defaults)
     51             elif method == 'both':
     52                 validate_args_and_kwargs(fname, args, kwargs,

~/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/util/_validators.py in validate_kwargs(fname, kwargs, compat_args)
    154     """
    155     kwds = kwargs.copy()
--> 156     _check_for_invalid_keys(fname, kwargs, compat_args)
    157     _check_for_default_values(fname, kwds, compat_args)
    158

~/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/util/_validators.py in _check_for_invalid_keys(fname, kwargs, compat_args)
    125         raise TypeError(("{fname}() got an unexpected "
    126                          "keyword argument '{arg}'".
--> 127                          format(fname=fname, arg=bad_arg)))
    128
    129

TypeError: any() got an unexpected keyword argument 'keepdims'

So we're being too strict here.

(cc @hameerabbasi just FYI)

[hameerabbasi]

From what I can tell, the failure is because Pandas doesn't realize that ufunc.reduce(...) accepts keepdims (via **kwargs). I may be wrong.

Ideally, something like this should not have occurred, since the function signature for the methods themselves (any, all, etc.) didn't change. However, we did base it on the np.logical_or ufunc instead of the cached methods in numpy/core/_methods.py. This can be reversed if it is super-important, or made to depend on the methods in numpy/core/_methods.py methods again.

[TomAugspurger]

From what I can tell, the failure is because Pandas doesn't realize that ufunc.reduce(...) accepts keepdims (via **kwargs). I may be wrong.

Yes I think you're right. We do some keyword argument filtering within pandas itself (@gfyoung I think you know the most about this?)

This can be reversed if it is super-important

I'm not sure. @shoyer do you have thoughts here? I'm not sure what the relative usage of np.any(frame) vs. frame.any is.

Either way, pandas will fix things on our end.

[gfyoung]

@TomAugspurger : Given my previous runarounds with numpy and pandas compatibility, I agree that frame.any would be preferable.

As for keyword argument filtering, we try to keep them as strict as possible so as to prevent extraneous keywords from filtering and discouraging the mixing of the numpy API with pandas objects. So in this case, we were disallowing the keepdims argument, but we could update it to accept when validating.

[shoyer]

The problem seems to be that NumPy now checks for any/all methods to defer to, like it's long done for sum, mean, min, max, etc.

On the whole, I think this is more consistent for pandas: np.any and np.all now work like other NumPy aggregation functions when called on pandas objects..

[TomAugspurger]

I'm not sure how to proceed here. I would like to ensure that np.all(dataframe) continues to work like before. It'd be a shame to break that, as that was a reasonable way to get a scalar boolean from a dataframe.

However, this conflicts with NumPy's (reasonable) expectation that np.all(thing, **kwargs) matches thing.all(**kwargs). In the past that's differed since np.all used to reduce a DataFrame to a scalar, while DataFrame.all reduced to a Series.

AFAICT, even implementing keepdims for DataFrame.all/any won't solve the issue, since our default is different.

[hameerabbasi]

Looping in @eric-wieser here. Eric, would it be reasonable to just make it so that (for any and all), getattr is not called like it is now (as it wasn't being called before)? Or possibly give priority to the ufunc over the attribute call (I'm not sure if that will fix this).

Edit: Semi-relevant xkcd: https://xkcd.com/1172/

[jorisvandenbossche]

A possible option would be to implement df.any(axis=None) to actually reduce over all dimensions? Or does numpy not pass axis=None to df.any(..)? But, that would of course also make it inconsistent with the rest of pandas.
(currently I think the main reason the default is axis=None but interpreted as axis=0 for both series and dataframe, is only Panel. But since that is deprecated, I think we can ignore consistency with Panel)

[hameerabbasi]

Yes, that would be best. df.any() has a default axis of maybe 0/1 (I'm not sure what's appropriate here), but np.any usually passes axis=None by default, thus achieving the desired behavior. Alternatively, you can create a sentinel PdAxis (which would be invalid with np.any) to signify that this is coming from Pandas.

In fact, this is how ufunc.reduce works, default is axis=0 but sum/prod/any/all override it to None.

In fact, if DataFrame already implements __array_ufunc__ or similar, df.all etc can just defer to that rather than duplicating code.

[jreback]

A possible option would be to implement df.any(axis=None) to actually reduce over all dimensions?

-1 on this. We don't use None as a valid parameter anywhere else and would be inconsistent. ok with using a signaling object (rather than None) so that np.any works (as current) though.

[hameerabbasi]

@jreback The issue here is that Numpy itself passes axis=None. It definitely does this for sum/prod. The thing that changed is that now it does this for any and all too.

[jorisvandenbossche]

Accepting None temporarily as a valid value (and which is not overridden by the implicit default of 0) makes it possible to actually deprecate this, without having the hard breakage as we see now in this issue.

[TomAugspurger]

I don't think I'll get to this for 0.23.0

[jorisvandenbossche]

Personally, for now accepting axis=None as backwards compatibility, should be a rather easy short-term solution. That way we can discuss later the long term solution that we want (a long term deprecation would probably have the same short-term solution as temporary backwards compatibility)

[hameerabbasi]

Wouldn't having the same solution as sum/prod be acceptable? I'm not too familiar with Pandas but sounds like it should be handled the same way now, as it is in Numpy?

[TomAugspurger]

Personally, for now accepting axis=None as backwards compatibility, should be a rather easy short-term solution.

I'd like to do this for 0.23.1. I have a PR mostly ready.

[TomAugspurger]

Quick status update: I have this mostly working so that np.all(df) has the previous behavior (reducing to a scalar). It allows DataFrame.all(axis=None) to mean reduce all axes. I can push these changes up now, but would it be strange for only the logical reductions to support axis=None, while for the other reductions axis=None means reduce the stat axis?

Is there a planned release date for NumPy 1.15?

[jorisvandenbossche]

can push these changes up now, but would it be strange for only the logical reductions to support axis=None, while for the other reductions axis=None means reduce the stat axis?

That is strange, but maybe something we can leave for after 0.23.1 to discuss/change ?

[gfyoung]

Is there a planned release date for NumPy 1.15?

@TomAugspurger : FWIW, probably not for some time. 1.14.4 just came out 5 days ago, and they also do candidate releases before an official major release.

I agree with @jorisvandenbossche to leave for after 0.23.1.

[TomAugspurger]

OK there's talk on the NumPy ML of 1.15 rcs coming up soon so I'm going to finish this today.

---

One question: DataFrame works around NumPy's object behavior:

In [4]: pd.DataFrame({"A": ['a', 'b'], 'B': ['c', 'd']}).all()
Out[4]:
A    True
B    True
dtype: bool

But we don't do that for Series.

In [7]: pd.Series(['a', 'b']).all()
Out[7]: 'b'

Do we want to clean this all up at once?

[hameerabbasi]

would it be strange for only the logical reductions to support axis=None, while for the other reductions axis=None means reduce the stat axis?

The interpretation of axis=None is up to the implementer. What's weird us, ufunc.reduce(...) defaults to the axis=0 behavior ("across the first axis"), and NumPy usually interprets axis=None to mean "across all axes".

Though, if you so wish, you can make it mean something else. I'd say you should set axis=None to mean whatever you wish for the default to be.

[TomAugspurger]

Just to be clear, we'll follow NumPy's behavior of axis=None meaning reduce across all axes, 0 for first axis, etc. The difference is the defaults. The pandas methods will default to axis=0, NumPy will default to axis=None.

[jorisvandenbossche]

Do we want to clean this all up at once?

Let's maybe keep that for 0.24.0 ?

Do you have a PR to review for this?

[TomAugspurger]

Later today hopefully.

[TomAugspurger]

PR at #21486