ENH: consistent types in output of df.groupby

[ikamensh]

Is your feature request related to a problem?

The problem that occurred to me is due to inconsistent types outputted by for values, match_df in df.groupby(names) - if names is a list of length >1, then values can be a tuple. Otherwise it's an element type, which can be e.g. string.

I had a bug where I iterating over values resulted in going over characters of a string, together with zip it just discarded most of the field.

Describe the solution you'd like

Whenever df.groupby is called with a list, output a tuple of values found. Specifically when called with a list of length 1, output a tuple of length 1 instead of unwrapped element.

API breaking implications

This would change how existing code behaves.

Describe alternatives you've considered

Leave things as they are. Unfortunately it means people need to write special case handling ifs around groupby to get consistent types outputted.

Additional context

import pandas as pd


df = pd.DataFrame(columns=['a','b','c'], index=['x','y'])
df.loc['y'] = pd.Series({'a':1, 'b':5, 'c':2})
print(df)

values, _ = next(iter(df.groupby(['a', 'b'])))
print(type(values))  # <class 'tuple'>

values, _ = next(iter(df.groupby(['a'])))
print(type(values))  # <class 'int'>  # IMHO should be 'tuple'!

values, _ = next(iter(df.groupby('a')))
print(type(values))  # <class 'int'>

[rhshadrach]

I haven't yet considered the implementation or all the ramifications of this, but we do similar things in other places, e.g.

df['a']  # Series
df[['a']]  # DataFrame

df.quantile(0.1)  # Series, index is the columns of df
df.quantile([0.1])  # DataFrame, index consists of the value 0.1

My guess is that a tuple may be appropriate in the case of passing a list to groupby, but that the result should still have an Index (not a MultiIndex).

cc @jbrockmendel for any thoughts here.

[jbrockmendel]

a length-1 tuple seems reasonable to me

[rhshadrach]

Will need to deprecate current behavior for this.

[ahmedibrhm]

@rhshadrach should I add a warning or just fix it?

[rhshadrach]

I think this needs to be deprecated before changing.

[ahmedibrhm]

@rhshadrach does that mean raising a deprecation warning for the case where there is only one group name?

[rhshadrach]

Yes, that's correct - something along the lines of "In a future version of pandas, a length 1 tuple will be returned when grouping by a list of length 1. Don't supply a list with a single grouper to avoid this warning."

[ahmedibrhm]

@rhshadrach
I did two PRs

The deprecation PR #47761
The actual removal PR #47719
The actual removal PR will need to be rebased after the deprecation ig.

[GilWu]

Hi, I got the warning message "In a future version of pandas, a length 1 tuple will be returned when grouping by a list of length 1. Don't supply a list with a single grouper to avoid this warning."
—> How can I avoid this warning when I just need to group by one element?
df.groupby([test])

[rhshadrach]

@GilWu - It sounds like you're getting an improper warning. Can you open a new issue with a reproducible example demonstrating the warning message you're seeing?

[agu3rra]

Hi, I got the warning message "In a future version of pandas, a length 1 tuple will be returned when grouping by a list of length 1. Don't supply a list with a single grouper to avoid this warning." —> How can I avoid this warning when I just need to group by one element? df.groupby([test])

@GilWu
I managed to supress the warning by passing in a single element to group by instead of a list. So in your example, it'd become df.groupby(test).

It's still unclear to me if I should go ahead and make my code test the returned group name with isinstance(name, tuple) in a single element group by to avoid breaking it once this always returns a tuple as today it returns a single element.

[rhshadrach]

If you are passing a single element, the behavior will be the same in 2.0. If you are passing a length-1 list, the behavior will change (scalar in 1.5.x, tuple in 2.0).

In general, if you are not getting a warning then you should experience no change.

[Azureuse]

Apologies if I've missed something, but should this change not also be applied to df.groupby(...).groups?

import pandas as pd

df = pd.DataFrame({'x': [10, 20, 30], 'y': ['a', 'b', 'c']})

# group by atomic - names and keys are atomic
name, _ = next(iter(df.groupby('x')))
name    # `10` as expected
df.groupby('x').groups  # `{10: [0], 20: [1], 30: [2]}` as expected

# group by list of 2 - names and keys are 2-tuples
name, _ = next(iter(df.groupby(['x', 'y'])))
name  # `(10, 'a')` as expected
df.groupby(['x', 'y']).groups  # `{(10, 'a'): [0], (20, 'b'): [1], (30, 'c'): [2]}` as expected

# group by list of 1 - names are 1-tuples but keys are atomic!?
name, _ = next(iter(df.groupby(['x'])))
name  # `(10,)` as expected
df.groupby(['x']).groups  # `{10: [0], 20: [1], 30: [2]}` UNEXPECTED

[rhshadrach]

@Azureuse - can you open a new issue.

[Azureuse]

@Azureuse - can you open a new issue.

Done :) #58858