Add new function to remove duplicate rows from a DataFrame

[wesm]

Should be reasonably performant, probably just use sets + np.apply_along_axis

[seanjtaylor]

Not sure if this should be a 1) dedicated DataFrame method, 2) a method on DataFrameGroupBy, or 3) call to aggregate with a specific function.

# 1
df = DataFrame()
new_df = df.remove_duplicates(('key1', 'key2'))

#2
new_df = df.groupby(('key1', 'key2')).first(check_identical=True) # take only the first row for each group, make sure all other rows with this key have the same values

#3
new_df = df.groupby(('key1', 'key2')).aggregate(first_row) # some function that takes the first row for each group

[wesm]

Here's an easy and fast implementation using existing tools:

grouped = df.groupby(keys)
index = [gp_keys[0] for gp_keys in grouped.groups.values()]
new_df = df.reindex(index)

maybe possible to go even faster, I'll play around with it

[seanjtaylor]

This is exactly what I was looking for. And I learned something useful about pandas internals today.