groupby() with concat() result columns can't be added

[ecoughlan]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.2
• Modin installed from (source or binary): pip
• Modin version: 0.6.3
• Python version: 3.7.3
• Exact command to reproduce: Snippet below

Describe the problem

Cannot add a combination of results to a DF composed of grouped results.

Source code / logs

import modin.pandas as pd #import pandas as pd
df = pd.DataFrame(data={"i":[1,1 ], "a":[0, 0], "b":[0, 0]})
g = df.groupby(by="i")
cat = pd.concat([g["a"].first(), g["b"].first()], keys=["x", "y"], axis=1)
cat["z"] = cat.x + cat.y

doesn't work because the subtraction results in two columns being returned with modin vs one in pandas, probably something is mixed up with a multi-index on the columns.

[devin-petersohn]

Thanks @ecoughlan, you are right that it is setting a multi-index on the columns. When keys is set for concat, Modin seems to be adding a level to the columns instead of replacing the label names with the values. It should be a straightforward fix.

As a side note, cat = pd.concat([g["a"].first(), g["b"].first()], keys=["x", "y"], axis=1) could be rewritten to cat = g.first().set_axis(["x", "y"], axis=1, inplace=False). I realize that isn't the point of the issue, but I want to add this as it is generally less efficient to do the way written.

Thanks again for posting!

[devin-petersohn]

@ecoughlan a quick update: This issue happened because the g["a"].first() object was a pandas.Series and we were internally checking for modin.pandas.Series. I fixed the case where it was creating a pandas.Series and tested that the Series behavior is still correct for modin.pandas.Series. The pandas.Series behavior in Modin was falling back to the DataFrame behavior, which does create a new level.