Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Series[datetime64/timedelta64] bool ops broken #19759

Closed
jbrockmendel opened this issue Feb 19, 2018 · 7 comments
Closed

Series[datetime64/timedelta64] bool ops broken #19759

jbrockmendel opened this issue Feb 19, 2018 · 7 comments
Labels
Bug Datetime Datetime data dtype Numeric Operations Arithmetic, Comparison, and Logical operations Timedelta Timedelta data type

Comments

@jbrockmendel
Copy link
Member

dti = pd.date_range('2016-01-01', periods=3)
ser = pd.Series(dti)
tr = np.array([True, True, True])

>>> ser & tr
0    False
1    False
2    False
dtype: bool

>>> ser | ser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/ops.py", line 1047, in wrapper
    other = fill_int(other) if is_other_int_dtype else fill_bool(other)
  File "pandas/core/ops.py", line 1036, in <lambda>
    fill_bool = lambda x: x.fillna(False).astype(bool)
  File "pandas/util/_decorators.py", line 138, in wrapper
    return func(*args, **kwargs)
  File "pandas/core/generic.py", line 4443, in astype
    **kwargs)
  File "pandas/core/internals.py", line 3606, in astype
    return self.apply('astype', dtype=dtype, **kwargs)
  File "pandas/core/internals.py", line 3473, in apply
    applied = getattr(b, f)(**kwargs)
  File "pandas/core/internals.py", line 569, in astype
    **kwargs)
  File "pandas/core/internals.py", line 2603, in _astype
    return super(DatetimeBlock, self)._astype(dtype=dtype, **kwargs)
  File "pandas/core/internals.py", line 657, in _astype
    values = astype_nansafe(values.ravel(), dtype, copy=True)
  File "pandas/core/dtypes/cast.py", line 668, in astype_nansafe
    to_dtype=dtype))
TypeError: cannot astype a datetimelike from [datetime64[ns]] to [bool]

It isn't clear to me what the desired behavior is here.

Also operating with an Index falls through to a case that I think is intended to only catch scalars:

tdi = pd.TimedeltaIndex(['1 Day', '2 Days', 'NaT'])
ser = pd.Series(tdi)
>>> ser & tdi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/ops.py", line 1062, in wrapper
    res_values = na_op(self.values, other)
  File "pandas/core/ops.py", line 1020, in na_op
    if not isna(y):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
@jreback
Copy link
Contributor

jreback commented Feb 19, 2018

there are some issues about this already
see if u can find them

@jbrockmendel
Copy link
Member Author

There are a few that are broadly related but I didn't see any about dt64/td64 dtypes. AFAICT the issue that needs to be nailed down before I can fix is: what are these operations supposed to represent for these dtypes? Since bitwise ops don't make a lot of sense for these dtypes it may just be a matter of making Series.astype(bool) work for datetime64 and timedelta64 dtypes (or ideally an implementation with less far-reaching effects)

@mroeschke mroeschke added Datetime Datetime data dtype Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff labels Jan 13, 2019
@mroeschke
Copy link
Member

FWIW the TDI/Series case returns a value now:

In [73]: tdi = pd.TimedeltaIndex(['1 Day', '2 Days', 'NaT'])
    ...: ser = pd.Series(tdi)

In [74]: ser & tdi
Out[74]:
0    False
1    False
2    False
dtype: bool```

@jbrockmendel
Copy link
Member Author

I would have expected [True, True, False]; wouldn't you?

@mroeschke
Copy link
Member

Agreed. Just noting that this doesn't raise a ValueError anymore

@mroeschke mroeschke added Bug Numeric Operations Arithmetic, Comparison, and Logical operations Timedelta Timedelta data type and removed Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff labels Apr 1, 2020
@jorisvandenbossche
Copy link
Member

An option is also to simply raise an error stating that logical ops are not supported for timedelta/datetime dtypes ?

@jbrockmendel
Copy link
Member Author

These all now raise in ways that make sense to me. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype Numeric Operations Arithmetic, Comparison, and Logical operations Timedelta Timedelta data type
Projects
None yet
Development

No branches or pull requests

4 participants