BUG: DataFrame+DataFrame ops

[jbrockmendel]

There are code paths for arithmetic methods (in particular DataFrame._combine_frame, _combine_match_index) that operate with self.values, other.values, and as a result behave differently from their Series/Index analogues. Example:

df = pd.DataFrame([pd.Timedelta(seconds=1), pd.Timedelta(seconds=2)])
df2 = pd.DataFrame([1*10**9, 2*10**9])

>>> df + df2  # <-- should raise TypeError
>>> df + df2
         0
0 00:00:02
1 00:00:04

>>> df[0] + df2[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/ops.py", line 1274, in wrapper
    result = dispatch_to_index_op(op, left, right, pd.TimedeltaIndex)
  File "pandas/core/ops.py", line 1331, in dispatch_to_index_op
    'operation [{name}]'.format(name=op.__name__))
TypeError: incompatible type for a datetime/timedelta operation [add]

AFAICT there are two options for how to fix these:

1. Have all DataFrame arith/comparison ops operate column-wise. Downside is performance hit on currently-OK operations; AFAIK this is part of the motivation behind using Blocks in the first place.

2. Make .values point at EAs, implement the arith/comparison methods on EAs.

• This would require implementing a bunch of not-currently-existing EAs for pretty much all the standard np dtypes (Make EAs for all Block subclasses #22388)
• This would require supporting 2D EAs.

[jorisvandenbossche]

@jbrockmendel the other option is to do it block wise? (which basically means column-wise for EAs, without the performance hit for consolidated numpy dtypes)

[jbrockmendel]

the other option is to do it block wise? (which basically means column-wise for EAs, without the performance hit for consolidated numpy dtypes)

That is similar to what I had in mind with option 2 above. Just instead of dispatching to the consolidated numpy array ops, it would be dispatching to the consolidated EA ops.

There are a bunch of differences between pandas and numpy arith/comparison ops, most of which are handled in core.ops. In the status quo we still don't have full internal-consistency for how these operations behave. I think the best way to achieve internal consistency is to have One True Implementation for each of these ops. This can either be on the Series (option 1 above) or in EA (option 2). The former is definitely easier to implement short-term, but I think less elegant long-term.

[jorisvandenbossche]

That is similar to what I had in mind with option 2 above. Just instead of dispatching to the consolidated numpy array ops, it would be dispatching to the consolidated EA ops.

Yes, but implementation wise that is still a big difference no?

[jbrockmendel]

Yes, but implementation wise that is still a big difference no?

On the EA side, yes, since it would mean implementing a bunch of new EA subclasses (and moving a bunch of logic from core.ops). On the DataFrame/BlockManager/Block side, not so much.

[jbrockmendel]

Closed by #22696.