use boolean indexing via getitem to trigger masking; add inplace keyword to where

[jreback]

in core/frame.py

changed method getitem to use mask directly (e.g. df.mask(df > 0) is equivalent semantically to df[df>0])
this would be a small API change as before df[df >0] returned a boolean np array

added inplace keyword to where method (to update the dataframe in place, default is NOT to use inplace, and return a new dataframe)

changed method boolean_set to use where and inplace=True (this allows alignment of the passed values and is slightly less strict than the current method)

all tests pass (as well as an added test in boolean frame indexing)

if included in 0.9.1 would be great (sorry for the late addition)

[wesm]

The files here were made executable again (notably, this causes nose to exclude them from runs). Something must be wrong with your env-- as a band-aid you can globally configure git so it doesn't change file permissions

[jreback]

git config core.filemode false
seems to fix this....pushed a change to update modes on these 2 files

[jreback]

I also think something like this should be added to the docs:

selecting with a single column of a df (eg df[df['A']>0])
will potentially change the shape of the frame (as entire rows can be dropped)

while masking with the entire frame (eg df[df>0])
returns an equal sized frame
(even in the case of partial masking on the rows)
eg df[df[:-1]>0]

[changhiskhan]

@jseabold :)

[changhiskhan]

@wesm @jreback, what's more intuitive to you:

1. mask where cond is True or False? Guess depends on whether you think of it as putmask-like or boolean selection.
2. should where take values from self or other when cond is True?

[wesm]

I'm unconvinced about the df[bool_dataframe] yielding a masked DataFrame behavior (instead of asking that people explicitly use mask). I guess the use case you're thinking of is to be able to do stuff like:

df[df > 0].mean(1)

vs

df.mask(df > 0).mean(1)

?

[jreback]

actually no; I use mask like this:

df = start_frame
for m in masks:
    df = df.mask(m)

so I am 'applying' a series of 'filters' to df, but I want to avoid reindexing at each step (as I like to preserve the shape);
(these operations are not necessarily simple either - so they are not all 'ands' - this is also a simplification - I don't do it in a loop like this but pass it around - and other functions need to know about the whole frame)

I am fine with making users do df.mask explicity; I was actually thinking that since df[df > 0 ] = 2 is supported, then the converse operation should directly yield a frame (rather than an ndarray)

df[df > 0] (in the new version)

is equivalent to the more verbose (assume that df[df >0] returns a frame rather than an ndarray)

df[df > 0].reindex_like(df)

[jreback]

in response to @changhiskhan

1. I agree 'True' is more natural (as that is what masking implies).....maybe it shouldn't be called mask, but 'select' (but that's taken!)..
2. i don't see a problem with how 'where' is done now....?

[changhiskhan]

@jreback, on #2, I was just thinking whether it's more intuitive that self.where(cond, other) is like np.where(cond, self, other) or np.where(cond, other, self). In your pull request, the cond is used in putmask where you need to invert the cond to match non-inplace DataFrame.where.

[jreback]

@changhiskhan
I had justed copied what was in setitem; here other is really the value that gets set, so it is what you want if cond is true

inplace = true is setitem, while inplace = false (e.g. use np.where) is getitem
and wanted to use the same alignment code (in theory these could actually/maybe should be separate methods)...inplace=True is kind of 'internal'; I dont' think it would ever would be called directly (instead you would call indirectly via setitem)

here's what I would do:

1. eliminate mask (the user can always recreate by inverting the condition / or make it call where with an inverted condition)
2. change the where signalture to: where(self, cond, other = NA, inplace = False)
3. change getitem to use where (rather than mask)

so then syntax is pretty clean: df.where(cond) is equiv to df[cond] when cond is a Frame

I have tested and am ready to push if this looks ok

[wesm]

I don't think the semantics for inplace=True are right-- it's doing the opposite of what you would want (leaving values in df where the condition is True otherwise making them values from the other value). Making the necessary changes...

[jreback]

ok...make sense.....only (minor) issue....in numpy pretty sure that negation (-) is completely equivalent of invert (~) when applied to boolean?

in mask should change ~cond to -cond ?

[wesm]

Yeah, no biggie either way

[jreback]

these might be helpful examples for whatsnew and/or docs.....

import pandas
import numpy as np

df = pandas.DataFrame(np.random.randn(5, 3),columns=['A','B','C'])
df

standard frame selection (with a boolean series as the condition),
get a frame that is not necessarily the same shape as the input frame

df[df.A>0]

standard frame selection (but with boolean frame as the condition,
get a masked frame as the result, the same size as the original frame)

df[df>0]

where is the underlying mechanism

df.where(df>0)

substitue values that meet the condition

df.where(df>0,-df)

setting values

df2 = df.copy()
df2[df2>0] = -df2
df2

masking is the inverse boolean operation of where

df.mask(df>0)

df.where(df<=0)

advanced: partial selection and setting

df2 = df.copy()
crit = df2[1:2]>0
crit
df2[crit] = 3
df2

[wesm]

if someone wants to do a PR to add these to the docs/what's new that'd be great. i am doing too many other things to do it myself

[jreback]

done....added a small pull-request with the changes for whatsnew for 0.9.1...let me know if you need anything further!