API/DEPR: Remove +/- as setops for Index (GH8227) #14127

jorisvandenbossche · 2016-08-31T10:38:15Z

xref #13777, deprecations put in place in #8227

tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

jreback · 2016-08-31T13:38:53Z

pandas/indexes/base.py

        return Index(other + np.array(self))

    __iadd__ = __add__

-    def __sub__(self, other):


I think this is not defined in super? so this should raise NotImplementedError I think?

yep, this is something that I still have to handle.

The problem is that sub/add are added for subclasses in either _add_numericlike_set_methods_disabled or _add_numeric_methods or both ovenwritten (always handled both), but in the base class we only want __add__ and not __sub__

If I leave it in here, I think it should rather raise a TypeError? (for usage of base Index class)

yeah TypeError is prob ok too

the only reason to raise NotImplemmeted is that then subclasses could override - but we don't want that

codecov-io · 2016-08-31T14:54:22Z

Current coverage is 85.25% (diff: 100%)

Merging #14127 into master will decrease coverage by <.01%

@@             master     #14127   diff @@
==========================================
  Files           139        139          
  Lines         50501      50495     -6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43058      43051     -7   
- Misses         7443       7444     +1   
  Partials          0          0

Powered by Codecov. Last update 4488f18...dbcc655

jreback · 2016-08-31T16:41:13Z

@jorisvandenbossche we should do this for the RC as well.

jorisvandenbossche · 2016-08-31T18:44:58Z

@jreback yep, as it is quite a substantial change (for people that ignored the warnings)

Is there a reason that we do not allow numeric ops for the base Index class? (eg pd.Index(['a', 'b'] could have a clear meaning, just letting the objects determine whether an operation can be handled)

Because eg the __add__ in the Index class could benefit from the machinery in _add_numeric_methods_binary (eg to ensure attributes like name are passed correctly. Something that is also not correct on current master)

jreback · 2016-08-31T19:42:50Z

I think we don't allow numeric operations on Index as only possible one is add for string concat
nothing else makes sense

in general these r not well defined so no reason to allow

shoyer · 2016-08-31T20:56:57Z

I think we should let the objects in the Index decide whether the numeric operation is valid or not. This is the approach NumPy uses for object dtype. For example, one might put decimal objects in a pandas.Index.

jreback · 2016-08-31T21:01:20Z

the numpy approach leads to lots of bugs and edge cases - constraining things somewhat to not allowing everything on everything is much more user friendly

decimal objects in an Index are a disaster - should not be encouraged

jreback · 2016-08-31T23:16:31Z

doc/source/whatsnew/v0.19.0.txt

+Addition and subtraction of certain Index types performed set operations
+(set union and difference). This behaviour was already deprecated since 0.15.0,
+and is now disabled and, when possible, ``+``
+and ``-`` are used for numeric operations (:issue:`8227`, :issue:`14127`).


I would be more specific that 'when possible' (maybe say for numeric & datetimelike)

datetimelike not yet (DatetimeIndex - DatetimeIndex does still set op, but this one is a bit more work to remove).

And + also works for strings, so not only numeric. Further, for numeric indices, + and - did already do numeric ops and not setops .. (to make it a bit more complex :-))

So I did not directly come up with a more accurate but simple wording, but if you have a suggestion all ears!

jreback · 2016-09-02T11:36:54Z

doc/source/whatsnew/v0.19.0.txt

+    In [1]: pd.Index(['a', 'b']) + pd.Index(['a', 'c'])
+    FutureWarning: using '+' to provide set union with Indexes is deprecated, use '|' or .union()
+    Out[1]: Index(['a', 'b', 'c'], dtype='object')
+


make this clear that this ONLY applies to Index itself and not-subclasses (maybe show that numeric/datetimelike ops are unchanged)

yes, will try to clarify

jreback · 2016-09-03T15:32:04Z

@jorisvandenbossche some small comments, but otherwise lgtm. merge when ready.

…eter * github.com:pydata/pandas: (554 commits) BUG: compat with Stata ver 111 Fix: F999 dictionary key '2000q4' repeated with different values (pandas-dev#14198) BLD: Test for Python 3.5 with C locale BUG: DatetimeTZBlock can't assign values near dst boundary BUG: union_categorical with Series and cat idx BUG: fix str.contains for series containing only nan values BUG: Categorical constructor not idempotent with ext dtype TST: Make encoded sep check more locale sensitive (pandas-dev#14161) DOC: minor typo in 0.19.0 whatsnew file (pandas-dev#14185) BUG: fix tz-aware datetime convert to DatetimeIndex (GH 14088) BUG : bug in setting a slice of a Series with a np.timedelta64 RLS: v0.19.0rc1 DOC: clean-up 0.19.0 whatsnew file (pandas-dev#14176) DOC: cleanup build warnings (pandas-dev#14172) Add steps to run gbq integration testing to the contributing docs (pandas-dev#14144) ENH: concat and append now can handle unordered categories (pandas-dev#13767) DEPR: Deprecate pandas.core.datetools (pandas-dev#14105) API/DEPR: Remove +/- as setops for DatetimeIndex/PeriodIndex (GH9630) (pandas-dev#14164) Fix trivial typo in comment (pandas-dev#14174) API/DEPR: Remove +/- as setops for Index (GH8227) (pandas-dev#14127) ...

* github.com:pydata/pandas: (554 commits) BUG: compat with Stata ver 111 Fix: F999 dictionary key '2000q4' repeated with different values (pandas-dev#14198) BLD: Test for Python 3.5 with C locale BUG: DatetimeTZBlock can't assign values near dst boundary BUG: union_categorical with Series and cat idx BUG: fix str.contains for series containing only nan values BUG: Categorical constructor not idempotent with ext dtype TST: Make encoded sep check more locale sensitive (pandas-dev#14161) DOC: minor typo in 0.19.0 whatsnew file (pandas-dev#14185) BUG: fix tz-aware datetime convert to DatetimeIndex (GH 14088) BUG : bug in setting a slice of a Series with a np.timedelta64 RLS: v0.19.0rc1 DOC: clean-up 0.19.0 whatsnew file (pandas-dev#14176) DOC: cleanup build warnings (pandas-dev#14172) Add steps to run gbq integration testing to the contributing docs (pandas-dev#14144) ENH: concat and append now can handle unordered categories (pandas-dev#13767) DEPR: Deprecate pandas.core.datetools (pandas-dev#14105) API/DEPR: Remove +/- as setops for DatetimeIndex/PeriodIndex (GH9630) (pandas-dev#14164) Fix trivial typo in comment (pandas-dev#14174) API/DEPR: Remove +/- as setops for Index (GH8227) (pandas-dev#14127) ...

jorisvandenbossche added API Design Clean labels Aug 31, 2016

jorisvandenbossche added this to the 0.19.0 milestone Aug 31, 2016

jreback reviewed Aug 31, 2016
View reviewed changes

jorisvandenbossche force-pushed the index-setops branch from bc44066 to a7dfd73 Compare August 31, 2016 14:54

jorisvandenbossche force-pushed the index-setops branch from a7dfd73 to 96b827d Compare August 31, 2016 15:41

jreback reviewed Aug 31, 2016
View reviewed changes

jorisvandenbossche changed the title ~~[WIP] API/DEPR: Remove +/- as setops for Index (GH8227)~~ API/DEPR: Remove +/- as setops for Index (GH8227) Sep 1, 2016

jorisvandenbossche modified the milestones: 0.19.0, 0.19.0rc Sep 1, 2016

jorisvandenbossche mentioned this pull request Sep 1, 2016

RLS: 0.19.0 #13991

Closed

jreback reviewed Sep 2, 2016
View reviewed changes

API/DEPR: Remove +/- as setops for Index (GH8227)

dbcc655

jorisvandenbossche force-pushed the index-setops branch from 96b827d to dbcc655 Compare September 4, 2016 10:05

jreback mentioned this pull request Sep 4, 2016

DEPR: deprecations log for removed issues #13777

Closed

jorisvandenbossche merged commit 8023029 into pandas-dev:master Sep 6, 2016

jorisvandenbossche modified the milestones: 0.19.0rc, 0.19.0 Sep 7, 2016

jorisvandenbossche modified the milestones: 0.19.0, 0.19.0rc Sep 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API/DEPR: Remove +/- as setops for Index (GH8227) #14127

API/DEPR: Remove +/- as setops for Index (GH8227) #14127

jorisvandenbossche commented Aug 31, 2016 •

edited

Loading

jreback Aug 31, 2016

jorisvandenbossche Aug 31, 2016 •

edited

Loading

jreback Aug 31, 2016

codecov-io commented Aug 31, 2016 •

edited

Loading

jreback commented Aug 31, 2016

jorisvandenbossche commented Aug 31, 2016

jreback commented Aug 31, 2016

shoyer commented Aug 31, 2016

jreback commented Aug 31, 2016

jreback Aug 31, 2016

jorisvandenbossche Sep 1, 2016

jreback Sep 2, 2016

jorisvandenbossche Sep 4, 2016

jreback commented Sep 3, 2016

API/DEPR: Remove +/- as setops for Index (GH8227) #14127

API/DEPR: Remove +/- as setops for Index (GH8227) #14127

Conversation

jorisvandenbossche commented Aug 31, 2016 • edited Loading

jreback Aug 31, 2016

Choose a reason for hiding this comment

jorisvandenbossche Aug 31, 2016 • edited Loading

Choose a reason for hiding this comment

jreback Aug 31, 2016

Choose a reason for hiding this comment

codecov-io commented Aug 31, 2016 • edited Loading

Current coverage is 85.25% (diff: 100%)

jreback commented Aug 31, 2016

jorisvandenbossche commented Aug 31, 2016

jreback commented Aug 31, 2016

shoyer commented Aug 31, 2016

jreback commented Aug 31, 2016

jreback Aug 31, 2016

Choose a reason for hiding this comment

jorisvandenbossche Sep 1, 2016

Choose a reason for hiding this comment

jreback Sep 2, 2016

Choose a reason for hiding this comment

jorisvandenbossche Sep 4, 2016

Choose a reason for hiding this comment

jreback commented Sep 3, 2016

jorisvandenbossche commented Aug 31, 2016 •

edited

Loading

jorisvandenbossche Aug 31, 2016 •

edited

Loading

codecov-io commented Aug 31, 2016 •

edited

Loading