Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Series bool ops testing needs fleshing out #19792

Closed
jbrockmendel opened this issue Feb 20, 2018 · 3 comments
Closed

Series bool ops testing needs fleshing out #19792

jbrockmendel opened this issue Feb 20, 2018 · 3 comments
Labels
Dtype Conversions Unexpected or buggy dtype conversions Enhancement Needs Tests Unit test(s) needed to prevent regressions Numeric Operations Arithmetic, Comparison, and Logical operations

Comments

@jbrockmendel
Copy link
Member

Looks like it's almost all in tests.series.test_operators.test_operators_bitwise

In particular, op(series, index) will always raise for op in [&, |, ^]. (Presumably also op(series, categorical), but I haven't looked at that closely.) I can fix this and add some basic tests for it, but the dtype logic in ops._bool_method_SERIES doesn't make entirely clear what the desired behavior is, so some input will be needed to flesh them out more thoroughly.

@jreback
Copy link
Contributor

jreback commented Feb 21, 2018

not real clear what u r asking here - can u show an example

@jbrockmendel
Copy link
Member Author

TL;DR: it is not clear what the pandas convention is for the desired behavior of these operations with dtypes including timedelta64/datetime64/datetime64[tz]/categorical/[extension?]

In some cases these ops are bool-like and in others they are bit-like, see how filler is defined https://github.com/pandas-dev/pandas/blob/master/pandas/core/ops.py#L1100, roughly:

        elif isinstance(other, ABCSeries):
            name = com._maybe_match_name(self, other)
            is_other_int_dtype = is_integer_dtype(other.dtype)
            other = fill_int(other) if is_other_int_dtype else fill_bool(other)
            filler = (fill_int if is_self_int_dtype and is_other_int_dtype
                      else fill_bool)
        else:
            # scalars, list, tuple, np.array
            filler = (fill_int if is_self_int_dtype and
                      is_integer_dtype(np.asarray(other)) else fill_bool)

The error-catching in the nested function (https://github.com/pandas-dev/pandas/blob/master/pandas/core/ops.py#L1063) is about patching numpy's conventions with pandas conventions:

arr = np.random.randn(5) * 1000
i = arr.astype('i8')
b = arr.astype(bool)
f = arr.astype('f8')

i & b --> returns 0/1-values in  a ndarray[int64] 
i & f --> raises TypeError
b & f --> raises TypeError

pd.Series(i) & pd.Series(b) --> returns Series[bool]
pd.Series(i) & pd.Series(f) --> returns Series[bool] 
pd.Series(b) & pd.Series(f) --> returns Series[bool]

AFAICT this was written without datetime64/datetime64[tz]/timedelta64/categorical being taken into account, so it is not obvious what the desired behavior is for these other dtypes.

@gfyoung gfyoung added Testing pandas testing functions or related to the test suite Datetime Datetime data dtype Categorical Categorical Data Type labels Feb 21, 2018
@mroeschke mroeschke added Dtype Conversions Unexpected or buggy dtype conversions Numeric Operations Arithmetic, Comparison, and Logical operations Enhancement and removed Categorical Categorical Data Type Datetime Datetime data dtype labels May 22, 2020
@mroeschke mroeschke added Needs Tests Unit test(s) needed to prevent regressions and removed Testing pandas testing functions or related to the test suite labels Jul 1, 2022
@jbrockmendel
Copy link
Member Author

The main important piece here is the dt64/td64, so closing in favor of #19759

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dtype Conversions Unexpected or buggy dtype conversions Enhancement Needs Tests Unit test(s) needed to prevent regressions Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

No branches or pull requests

4 participants