groupby.quantile(<arraylike>) fails with AssertionError

[ketzu]

Code Sample, a copy-pastable example if possible

# Your code here
df = pd.DataFrame(np.array([10*[_%4] for _ in range(100)]))            

df.groupby(0).quantile(0.5)                                            
# Out[19]: 
#     1    2    3    4    5    6    7    8    9
# 0                                             
# 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
# 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
# 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
# 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0

df.groupby(0).quantile([0.5,0.99])                                     
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-20-21c92d2481c9> in <module>
----> 1 df.groupby(0).quantile([0.5,0.99])

~/PycharmProjects/netsim_stats/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py in quantile(self, q, interpolation)
   1950 
   1951             indices = np.concatenate(arrays)
-> 1952             assert len(indices) == len(result)
   1953             return result.take(indices)
   1954 

AssertionError: 

df.quantile([0.5,0.99])                                                
#        0    1    2    3    4    5    6    7    8    9
# 0.50  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5
# 0.99  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
                                                              
df.groupby(0)[1].quantile(0.5) 
# 0
# 0    0.0
# 1    1.0
# 2    2.0
# 3    3.0
# Name: 1, dtype: float64

df.groupby(0)[1].quantile([0.5,0.99])

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-24-ebf6ade716ff> in <module>
----> 1 df.groupby(0)[1].quantile([0.5,0.99])

~/PycharmProjects/netsim_stats/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py in quantile(self, q, interpolation)
   1950 
   1951             indices = np.concatenate(arrays)
-> 1952             assert len(indices) == len(result)
   1953             return result.take(indices)
   1954 

AssertionError: 

Problem description

The above is a constructed minimal example.
I am not sure how much I should elaborate on the "why this is a problem".

groupby.quantile() fails with an assertion error for "larger" dataframes, smaller dataframes seem to work fine.

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.3.15-200.fc30.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : de_DE.UTF-8
LOCALE : de_DE.UTF-8

pandas : 0.25.3
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.1.1
setuptools : 40.8.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.2.5
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.2.0
pandas_datareader: None
bs4 : 4.6.3
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.2.5
matplotlib : 3.1.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.1.0
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[TomAugspurger]

The problem is in the call to reorder_levels

In [11]: df.groupby(0).quantile([0.5,0.99])
> /Users/taugspurger/sandbox/pandas/pandas/core/groupby/groupby.py(1941)quantile()
-> order = np.roll(list(range(result.index.nlevels)), -1)
(Pdb) n
> /Users/taugspurger/sandbox/pandas/pandas/core/groupby/groupby.py(1942)quantile()
-> result = result.reorder_levels(order)
(Pdb) result
          1    2    3    4    5    6    7    8    9
     0
0.50 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
     1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
     2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
     3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
0.99 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
     1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
     2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
     3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
(Pdb) order
array([1, 0])
(Pdb) result.reorder_levels(order)
       1    2    3    4    5    6    7    8    9
0 0
0 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
2 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
3 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
0 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
2 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
3 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0

We're expecting to use level positions there, but one of the index levels is named 0, the value 0 is interpreted as the label rather than the position.

[TomAugspurger]

I don't see a clean way to say that we really mean index levels when the index level names are integers.

In [19]: idx = pd.MultiIndex.from_product([['a', 'b'], [0, 1]], names=['a', 0])

In [20]: idx.reorder_levels([0, 1])
Out[20]:
MultiIndex([(0, 0),
            (1, 1),
            (0, 0),
            (1, 1)],
           )

Should we add a keyword to reorder_levels to control how that's interpreted?

def reorder_levels(self, order, positional=None):
    """
    positional : bool, optional
        How to interpret integer values in `order`.

          * None (default): prefer treating the values as labels,
            but fall back to positional if no label with that
            value is value.
          * True : only treat integer values as positions.
          * False : only treat integer values as labels.
    """

[fujiaxiang]

Hi, can I look into this? I will add the "positional" parameter and probably refactor MultiIndex._get_level_number a bit

[fujiaxiang]

I added the positional parameter in both DataFrame.reorder_levels and MultiIndex.reorder_levels methods. In GroupBy.quantile this positional argument is set to True. In all other existing codes positional is defaulted to None.