ENH: duplicated and drop_duplicates now accept keep kw

[sinhrks]

Closes #6511, Closes #8505.

Introduce keep kw to handle first / last / all duplicates based on the discussion above. If there is better API/kw, please lmk.

duplicated

• keep='first' (default): mark duplicates as True except for the first occurrence.
• keep='last': mark duplicates as True except for the last occurrence.
• keep=False: mark all duplicates as True.

drop_duplicates

• keep='first' (default): drop duplicates except for the first occurrence.
• keep='last': drop duplicates except for the last occurrence.
• keep=False: drop all duplicates.

import pandas as pd
s = pd.Series(['A', 'B', 'C', 'A', 'B', 'D'])

# mark duplicates except for the first occurrence
s.duplicated()
#0    False
#1    False
#2    False
#3     True
#4     True
#5    False
# dtype: bool

# mark duplicates except for the last occurrence
s.duplicated(keep='last')
#0     True
#1     True
#2    False
#3    False
#4    False
#5    False
# dtype: bool

# mark all duplicates
s.duplicated(keep=False)
#0     True
#1     True
#2    False
#3     True
#4     True
#5    False
# dtype: bool

# drop duplicates except for the first occurrence
s.drop_duplicates()
#0    A
#1    B
#2    C
#5    D
# dtype: object

# drop duplicates except for the last occurrence
s.drop_duplicates(keep='last')
#2    C
#3    A
#4    B
#5    D
# dtype: object

# drop all duplicates
s.drop_duplicates(keep=False)
#2    C
#5    D
# dtype: object

CC @shoyer, @wikiped, @socheon

[shoyer]

I like this functionality... but I'm not sure about the name/keywords.

take='first' and take='last' indicate that the first/last item should be kept. Thus, if no items are kept, it would make more sense to use the value take='none' rather than take='all'.

Also, I'm not sure about the name take. I know this is similar to the existing keyword argument take_last, but I don't find it very suggestive. I don't know if I have a better alternative, though... maybe keep?

[wikiped]

Much appreciated enhancement, but take does sound ambiguous to me as well (as in 'take-to-drop' or 'take-to-keep'). keep sounds like a clearer option.

[sinhrks]

Thanks, agreed that keep is preferable to take. Changing the PR if there is no other suggestions within few days.

duplicated

• keep='first' (default): mark duplicates as True except for the first occurrence.
• keep='last': mark duplicates as True except for the last occurrence.
• keep='none': mark all duplicates as True.

drop_duplicates

• keep='first' (default): drop duplicates except for the first occurrence.
• keep='last': drop duplicates except for the last occurrence.
• keep='none': drop all duplicates.

[jreback]

keep lgtm. I only -0 on the keep='none' though as its not very pythonic.

[sinhrks]

Alternative could be keep=False or keep=None?

[jreback]

I like, but I don't think will work keep=None (as it will then be the default), but keep=False looks ok to me

[shoyer]

keep=False works for me, though there's something to be said for all arguments having the same type.

[sinhrks]

Thanks, updated PR and description on top. Should this wait for 0.17?

[jreback]

yep

[jreback]

prints here

[jreback]

couple of minor comments

looks good

merge when u r ready

[jreback]

same that the take_last is deprecated (and put a pointer to it below)

[sinhrks]

Thanks. Fixed and added to master tracker.

[jreback]

@sinhrks I think this was left in here?

[sinhrks]

Ah, this is for testing tm.assert_produces_warning in a loop. Should remove?

[jreback]

u will have lots of failures

need to either remove some of the tests that have take_last or convert them to keep (prob best) or use an assert produces around them

[jreback]

not lots of failures but the same one