ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations #4051

jtratner · 2013-06-27T02:45:45Z

Resolves #3765.

This is going to be split into two parts and then rebased on top of #3482.

Tasks:

Move arithmetic operations to separate file (e.g., ops.py).
Explore combining series/panel/frame operation methods.
Add rand, rxor, ror, etc...

This adds:

numexpr acceleration to Series, Panel, Panel4D, and DataFrame (SparsePanel has to opt-out b/c doesn't respond to shape)
Adds a test mechanism to numexpr to check that it actually is accelerated.
Moves all arithmetic operators to core/generic and binds them at import to reduce code duplication.
Makes all of Series, Panel, Panel4D, DataFrame, SparsePanel support the entire range of arithmetic operators [previously some were missing].
Cleans up core/expressions slightly
Test cases for all operations.
Improves a few test utilities to be better (error message for isnull and better error for isinstance checks in util/testing)

@cpcloud - I think this would be very easy to add into your current changes with eval. (mostly just changes if hasattr --> getattr to be cleaner).

jtratner · 2013-06-27T02:46:28Z

I can combine a few of the commits if necessary ... just wanted to make the progression clearer.

cpcloud · 2013-06-27T03:05:03Z

This is great. Numexpr ftw.

jtratner · 2013-06-27T04:03:36Z

yay ... another network error :-/

cpcloud · 2013-06-27T04:20:45Z

fixes r on the way

cpcloud · 2013-06-27T04:21:02Z

moving at the speed of travis :|

jtratner · 2013-06-27T04:22:52Z

well, I realized I missed testing the entire set of arithmetic methods in the individual cases (outside of test_expressions), so I guess it's okay. Btw - is it worth it to mark the panel tests as slow in expressions if they add 1.5s to the total suite?

jtratner · 2013-06-27T04:23:02Z

(1.5s total)

cpcloud · 2013-06-27T04:25:28Z

Is it a single test? If so yes else no

jtratner · 2013-06-27T04:50:46Z

no, it's not.

Now evaluate takes eval_kwargs (generally just truediv) to numexpr.evaluate plus adds a set of testing methods to check that numexpr was actually used successfully. Also changes the if hasattr idiom --> getattr.

Adds a better error for ``isinstance`` checks as well as a better error message for the ``isnull`` case.

Makes the entire arithmetic test suite explicit as well as sets up test cases to make sure the default_axis responses do not change.

BUG: Fix ``_fill_zeros`` call to work even if TypeError (previously was inconsistent).

SparsePanel has to opt-out because it doesn't respond to ``shape``. ENH: Add flex comparison methods to Series and Panel

…tic ops

jreback · 2013-06-27T11:31:46Z

@jtratner this real nice..

not that I think ops actually make much sense on a Panel4D, but maybe throw a couple of tests, e.g. maybe adding them or something...?

jtratner · 2013-06-27T11:40:49Z

@jreback okay, will do.

jreback · 2013-06-27T11:55:15Z

pandas/core/series.py

@@ -326,18 +329,35 @@ def _flex_method(op, name):
    -------
    result : Series
    """ % name
+    # copied directly from _arith_method above...we'll see whether this works
+    def na_op(x, y):


completely random

do you think its possible to consolidate these arithmetic ops out of series/frame/panel and move to say ops.py
(they still prob need to be separate but maybe can somewhat be combined)....

I am refereing to the actual methods, e.g. _flex_method and the sub-functions na_op

just a thought

I was definitely thinking of this. It's possible that much of series and
frame could be combined. I'll explore.

gr8......I would STILL be inclined to keep them separate as it reduces code complexity but there may be some easily fixed stuff (though +1 to move to say ops.py)

jreback · 2013-07-20T01:08:12Z

@jtratner

@cpcloud and I created a new master branch to handle the new eval stuff (which is actually ready to merge, just after 0.13), branch is eval-3393 (and links to the eval PR); feel free to add any of this type of stuff to itt

jtratner · 2013-07-20T01:13:40Z

oh awesome. I'll definitely do that.

On Fri, Jul 19, 2013 at 9:08 PM, jreback notifications@github.com wrote:

@jtratner https://github.com/jtratner

@cpcloud https://github.com/cpcloud and I created a new master branch
to handle the new eval stuff (which is actually ready to merge, just after
0.13), branch is eval-3393 (and links to the eval PR); feel free to add any
of this type of stuff to itt

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/4051#issuecomment-21285701
.

jreback · 2013-07-20T01:16:47Z

you can push, just DONT rebase
and make sure you pull before you push, if you can't push let me know

jtratner · 2013-07-20T01:34:31Z

@jreback good to know.

jreback · 2013-07-23T22:46:47Z

@jtratner after I update #3482 / #4324

I think it would be appropriate to use part of this PR, namely which moved the arithmetic/comp ops to generic.py to cover all of NDFrame (as Series will then inherit too), e.g. your _add_flex.... type methods...(which I guess will live in ops.py?
(this is essentially the 3rd todo in #4324)

the numexpr stuff can be done separately (after)

jtratner · 2013-07-24T00:46:48Z

Yeah, I agree with you on that. Adding numexpr speedup is trivial compared
to normalizing ops between all the parts.

To be clear: do you want me to make a PR and/or push some of these changes
to eval-3393 branch that you have already or do you want me to wait until
after its merged?
On Jul 23, 2013 6:47 PM, "jreback" notifications@github.com wrote:

@jtratner https://github.com/jtratner after I update #3482 #3482
#4324 #4324

I think it would be appropriate to use part of this PR, namely which moved
the arithmetic/comp ops to generic.py to cover all of NDFrame (as Series
will then inherit too), e.g. your _add_flex.... type methods...(which I
guess will live in ops.py?
(this is essentially the 3rd todo in #4324 #4324
)

the numexpr stuff can be done separately (after)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/4051#issuecomment-21451945
.

jreback · 2013-07-24T00:54:47Z

I think that eval-3393 is pretty independent of the series refactor. what you can do if you want is make a new PR on top of series #3482 (or do after I have merged). The numexpr stuff let's do after both of these PR's are in in your artihmetic/comparison changes.

jtratner · 2013-07-24T00:58:54Z

Makes sense

cpcloud · 2013-07-24T00:58:57Z

@jreback should probably merge series first then eval...i'll rebase after that merge and then test again to make sure things are working correctly....but shouldn't really matter that much

jtratner · 2013-07-24T01:00:17Z

Now that I think about it, separating out the numexpr part will make this way easier to apply on top of that PR.

jtratner · 2013-07-24T03:38:32Z

Closing as per @jreback 's suggestion and will split into two separate PRs.

jtratner added 10 commits June 27, 2013 00:54

CLN: Allow evaluate to take eval_kwargs + testing

1bc7f15

Now evaluate takes eval_kwargs (generally just truediv) to numexpr.evaluate plus adds a set of testing methods to check that numexpr was actually used successfully. Also changes the if hasattr idiom --> getattr.

TST: Make test utilities more expressive.

dad26b7

Adds a better error for ``isinstance`` checks as well as a better error message for the ``isnull`` case.

TST: Test all arithmetic ops + frame default_axis

d4167d8

Makes the entire arithmetic test suite explicit as well as sets up test cases to make sure the default_axis responses do not change.

ENH: Add bind_method to core/common

b15c225

ENH: Add classmethods to generate arithmetic operators in core/generic

6e9eef5

ENH: Use numexpr + normalize signature throughout.

738089e

BUG: Fix ``_fill_zeros`` call to work even if TypeError (previously was inconsistent).

ENH: Abstract arithmetic into core/generic

5793a04

SparsePanel has to opt-out because it doesn't respond to ``shape``. ENH: Add flex comparison methods to Series and Panel

CLN: PEP8 the tests/test_expressions

7572230

DOC: Document entire range of arithmetic methods in api.rst

0c455ba

TST: Flesh out test cases for Series and Panel to include all arithme…

66259e7

…tic ops

jreback reviewed Jun 27, 2013
View reviewed changes

jtratner mentioned this pull request Jul 24, 2013

add rand, ror, rxor methods to frame #4331

Closed

cpcloud mentioned this pull request Jul 24, 2013

ENH/API: add right-hand-side bool methods to DataFrame #4334

Closed

jtratner closed this Jul 24, 2013

This was referenced Aug 10, 2013

CLN: series to now inherit from NDFrame #3482

Merged

CLN/ENH: Provide full suite of arithmetic (and flex) methods to all NDFrame objects. #4560

Closed

jtratner mentioned this pull request Sep 29, 2013

CLN/ENH: Provide full suite of arithmetic (and flex) methods to all NDFrame objects. #5022

Merged

jreback mentioned this pull request Sep 29, 2013

CLN: Post Series subclass from NDFrame #4324

Closed

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations #4051

ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations #4051

jtratner commented Jun 27, 2013

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

jreback commented Jun 27, 2013

jtratner commented Jun 27, 2013

jreback Jun 27, 2013

jtratner Jun 27, 2013

jreback Jun 27, 2013

jreback commented Jul 20, 2013

jtratner commented Jul 20, 2013

jreback commented Jul 20, 2013

jtratner commented Jul 20, 2013

jreback commented Jul 23, 2013

jtratner commented Jul 24, 2013

jreback commented Jul 24, 2013

jtratner commented Jul 24, 2013

cpcloud commented Jul 24, 2013

jtratner commented Jul 24, 2013

jtratner commented Jul 24, 2013

ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations #4051

ENH: Use numexpr for all arithmetic operators + refactor arithmetic operations #4051

Conversation

jtratner commented Jun 27, 2013

This is going to be split into two parts and then rebased on top of #3482.

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

jtratner commented Jun 27, 2013

cpcloud commented Jun 27, 2013

jtratner commented Jun 27, 2013

jreback commented Jun 27, 2013

jtratner commented Jun 27, 2013

jreback Jun 27, 2013

Choose a reason for hiding this comment

jtratner Jun 27, 2013

Choose a reason for hiding this comment

jreback Jun 27, 2013

Choose a reason for hiding this comment

jreback commented Jul 20, 2013

jtratner commented Jul 20, 2013

jreback commented Jul 20, 2013

jtratner commented Jul 20, 2013

jreback commented Jul 23, 2013

jtratner commented Jul 24, 2013

jreback commented Jul 24, 2013

jtratner commented Jul 24, 2013

cpcloud commented Jul 24, 2013

jtratner commented Jul 24, 2013

jtratner commented Jul 24, 2013