Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DEPR: df.iloc[:, foo] = bar attempt to set inplace #45333

Merged
merged 28 commits into from
May 23, 2022
Merged

DEPR: df.iloc[:, foo] = bar attempt to set inplace #45333

merged 28 commits into from
May 23, 2022

Conversation

jbrockmendel
Copy link
Member

@jbrockmendel jbrockmendel commented Jan 12, 2022 •
edited
Loading

@jbrockmendel jbrockmendel added API - Consistency Internal Consistency of API/Behavior Deprecate Functionality to remove in pandas Indexing Related to indexing on series/frames, not to indexes themselves labels Jan 15, 2022
@jreback
Copy link
Contributor

jreback commented Jan 16, 2022

can you rebase

@jbrockmendel
Copy link
Member Author

Made a couple of efforts recently to simplify the iloc.__setitem__ and they all crash on the rocks of this inconsistency.

@jbrockmendel
Copy link
Member Author

There's a problem here in that the use .iloc(axis=1)[foo]=bar won't actually work, so still need a better alternative.

But getting this done should be a very high priority for 1.5 as its a blocker for all kinds of inconsistency-fixing.

@jbrockmendel jbrockmendel mentioned this pull request Jan 31, 2022
Closed
3 tasks
@jbrockmendel jbrockmendel mentioned this pull request Mar 11, 2022
Closed
3 tasks
@jbrockmendel
Copy link
Member Author

@jreback in the Feb call i think we agreed to move forward with this but there was some bikeshedding over the name of the replacement method. Do you remember what we landed on?

@jbrockmendel jbrockmendel mentioned this pull request Apr 15, 2022
Closed
1 task
@jbrockmendel jbrockmendel mentioned this pull request May 22, 2022
Merged
@jbrockmendel
Copy link
Member Author

@jreback gentle ping, finally green

@jreback jreback added this to the 1.5 milestone May 23, 2022
@jreback jreback merged commit 46bcf37 into pandas-dev:main May 23, 2022
@jreback
Copy link
Contributor

jreback commented May 23, 2022

thanks @jbrockmendel yeah has been out here for a while :->

followup, i think we need .isetitem in the api docs (and maybe a mention in the indexing.rst as well)

@jbrockmendel jbrockmendel deleted the depr-isetitem branch May 24, 2022 01:20
@lesteve lesteve mentioned this pull request Jun 15, 2022
Merged
@phofl phofl mentioned this pull request Jun 16, 2022
Closed
yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request Jul 13, 2022
@jbrockmendel @yehoshuadimarsky
DEPR: df.iloc[:, foo] = bar attempt to set inplace (pandas-dev#45333)
af81898
jorisvandenbossche added a commit to jorisvandenbossche/pandas that referenced this pull request Jul 16, 2022
@jorisvandenbossche
fixup after updata main and column_setitem + iloc inplace setitem cha…
9339f15 
…nges (pandas-devgh-45333)
mroeschke added a commit that referenced this pull request Aug 20, 2022
@jorisvandenbossche @mroeschke
API: New copy / view semantics using Copy-on-Write (#46958)
221f636 
* API: New copy / view semantics using Copy-on-Write

* fix more tests

* Handle CoW in BM.iset

* Handle CoW in xs

* add bunch of todo comments and usage warnings

* Insert None ref in BM.insert

* Ensure to not assume copy_on_write is set in case of ArrayManager

* Handle refs in BM._combine / test CoW in select_dtypes

* handle get_numeric_data for single block manager

* fix test_internals (get_numeric_data now uses CoW)

* handle refs in consolidation

* fix deep=None for ArrayManager

* update copy/view tests from other PR

* clean-up fast_xs workarounds now it returns a SingleBlockManager

* tracks refs in to_frame

* fixup after updata main and column_setitem + iloc inplace setitem changes (gh-45333)

* fix inplace fillna + fixup new tests

* address comments + update some todo comments

* Update pandas/core/internals/managers.py

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>

* fixup linting

* update new copy_view tests to use get_array helper

* add comment to setitem

* switch default to False, ensure CoW copies only happen when enabled + add additional test build with CoW

* update type annotations

* Fix stata issue to avoid SettingWithCopyWarning in read_stata

* update type + option comment

* fixup new rename test

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>
CloseChoice pushed a commit to CloseChoice/pandas that referenced this pull request Aug 21, 2022
@jorisvandenbossche @mroeschke @CloseChoice
API: New copy / view semantics using Copy-on-Write (pandas-dev#46958)
0157e90 
* API: New copy / view semantics using Copy-on-Write

* fix more tests

* Handle CoW in BM.iset

* Handle CoW in xs

* add bunch of todo comments and usage warnings

* Insert None ref in BM.insert

* Ensure to not assume copy_on_write is set in case of ArrayManager

* Handle refs in BM._combine / test CoW in select_dtypes

* handle get_numeric_data for single block manager

* fix test_internals (get_numeric_data now uses CoW)

* handle refs in consolidation

* fix deep=None for ArrayManager

* update copy/view tests from other PR

* clean-up fast_xs workarounds now it returns a SingleBlockManager

* tracks refs in to_frame

* fixup after updata main and column_setitem + iloc inplace setitem changes (pandas-devgh-45333)

* fix inplace fillna + fixup new tests

* address comments + update some todo comments

* Update pandas/core/internals/managers.py

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>

* fixup linting

* update new copy_view tests to use get_array helper

* add comment to setitem

* switch default to False, ensure CoW copies only happen when enabled + add additional test build with CoW

* update type annotations

* Fix stata issue to avoid SettingWithCopyWarning in read_stata

* update type + option comment

* fixup new rename test

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>
@anthonyaag anthonyaag mentioned this pull request Sep 21, 2022
Closed
3 tasks
@mroeschke mroeschke mentioned this pull request Sep 29, 2022
Merged
5 tasks
noatamir pushed a commit to noatamir/pandas that referenced this pull request Nov 9, 2022
@jorisvandenbossche @mroeschke @noatamir
API: New copy / view semantics using Copy-on-Write (pandas-dev#46958)
1397fd0 
* API: New copy / view semantics using Copy-on-Write

* fix more tests

* Handle CoW in BM.iset

* Handle CoW in xs

* add bunch of todo comments and usage warnings

* Insert None ref in BM.insert

* Ensure to not assume copy_on_write is set in case of ArrayManager

* Handle refs in BM._combine / test CoW in select_dtypes

* handle get_numeric_data for single block manager

* fix test_internals (get_numeric_data now uses CoW)

* handle refs in consolidation

* fix deep=None for ArrayManager

* update copy/view tests from other PR

* clean-up fast_xs workarounds now it returns a SingleBlockManager

* tracks refs in to_frame

* fixup after updata main and column_setitem + iloc inplace setitem changes (pandas-devgh-45333)

* fix inplace fillna + fixup new tests

* address comments + update some todo comments

* Update pandas/core/internals/managers.py

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>

* fixup linting

* update new copy_view tests to use get_array helper

* add comment to setitem

* switch default to False, ensure CoW copies only happen when enabled + add additional test build with CoW

* update type annotations

* Fix stata issue to avoid SettingWithCopyWarning in read_stata

* update type + option comment

* fixup new rename test

Co-authored-by: Matthew Roeschke <emailformattr@gmail.com>
@rhshadrach rhshadrach mentioned this pull request Dec 18, 2022
Closed
@jorisvandenbossche jorisvandenbossche mentioned this pull request Jan 2, 2023
Merged
@SorooshMani-NOAA SorooshMani-NOAA mentioned this pull request Apr 24, 2023
Closed
@michaelmckinsey1 michaelmckinsey1 mentioned this pull request Apr 25, 2023
Closed
@dannypeterson dannypeterson mentioned this pull request Nov 30, 2023
Merged
@omatthew98 omatthew98 mentioned this pull request Jul 16, 2024
Merged
8 tasks
omatthew98 added a commit to ray-project/ray that referenced this pull request Jul 17, 2024
@omatthew98
[py312] Fix test_tensor.py (#46669)
aefdf5f 
## Why are these changes needed?
To upgrade to py312 we need to upgrade to `pandas>=2.0.0`. This upgrade
introduced a breaking change in the syntax of some of our code / tests:
> Changed behavior in setting values with df.loc[:, foo] = bar or
df.iloc[:, foo] = bar, these now always attempt to set values inplace
before falling back to casting ([GH
45333](pandas-dev/pandas#45333))

As a result we needed to update these uses of loc to do direct
assignment.

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Matthew Owen <mowen@anyscale.com>
can-anyscale pushed a commit to ray-project/ray that referenced this pull request Jul 18, 2024
@omatthew98 @can-anyscale
[py312] Fix test_tensor.py (#46669)
77d02bd 
To upgrade to py312 we need to upgrade to `pandas>=2.0.0`. This upgrade
introduced a breaking change in the syntax of some of our code / tests:
> Changed behavior in setting values with df.loc[:, foo] = bar or
df.iloc[:, foo] = bar, these now always attempt to set values inplace
before falling back to casting ([GH
45333](pandas-dev/pandas#45333))

As a result we needed to update these uses of loc to do direct
assignment.

<!-- For example: "Closes #1234" -->

- [ ] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

Signed-off-by: Matthew Owen <mowen@anyscale.com>
Signed-off-by: can <can@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
API - Consistency Internal Consistency of API/Behavior Deprecate Functionality to remove in pandas Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Milestone
1.5
Development

Successfully merging this pull request may close these issues.

BUG/API: DataFrame.iloc[:, foo] = bar inplaceness?
2 participants
@jbrockmendel @jreback