Move defaulting to pandas at groupby aggregation from API level to backend

[pastranaluis]

Hi! I have been working on some other backend for modin, and I have seen that modin defaults multiple column groupby to pandas. I have two questions/concerns:

1. Why if I pass a one element list ['column_title'], modin classifies this as a _is_multi_by? In particular, I am wondering what is the logic behind these lines of code in groupby.py

self._is_multi_by = (
                isinstance(by, type(self._query_compiler)) and len(by.columns) > 1
            ) or (
                not isinstance(by, type(self._query_compiler))
                and axis == 0
                and all(obj in self._query_compiler.columns for obj in self._by)
            )

the first condition makes sense to me, but the second one is throwing me off.

2. Why is the reason behind defaulting multi_col index to pandas? I would feel that the engine should take care of this.

Thanks!
LP

[dchigarev]

Hi @pastranakike, thanks for your question! Currently, Modin Groupby defaulting to pandas if we're applying aggregation function on multiple columns. Defaulting to pandas at the API level in this case is a kind of deprecated behavior, and definitely should be changed, I have classified this issue as a bug and I think that we'll fix that in this release.

About lists of one element as a by. I've found this if statement, that should filter these cases, do you have a reproducer where it doesn't work and groupby considered to be multi-column if by is a list of single element?

modin/modin/pandas/dataframe.py

Lines 358 to 363 in 3fd16a9

	if (
	not isinstance(by, (pandas.Series, Series))
	and is_list_like(by)
	and len(by) == 1
	):
	by = by[0]

[YarShev]

Hi @pastranakike , Answering to the first one question I don't see a case when by is one element list and Modin treats it as is_multi_by=True. If we look at simple example from pandas documentation, we can see that is_multi_by is treated as False by Modin.

import modin.pandas as pd
df = pd.DataFrame({'Animal': ['Falcon', 'Falcon',
                              'Parrot', 'Parrot'],
                   'Max Speed': [380., 370., 24., 26.]})
df
   Animal  Max Speed
0  Falcon      380.0
1  Falcon      370.0
2  Parrot       24.0
3  Parrot       26.0
df.groupby(['Animal']).mean() # is_multi_by=False while performing the operation
        Max Speed
Animal
Falcon      375.0
Parrot       25.0

If you have a reproducer where is_multi_by is treated as True for one element list by Modin, feel free to open new issue.
Regarding to the second one question, we are not currently processing the situation is_multi_col=True, that's why we are defaulting to pandas. For now we could move the logic regarding is_multi_col=True from API layer to the query compiler layer.