groupby.nth with dropna is buggy

[behzadnouri]

>>> df
   1st  2nd
0    1  NaN
1    2    3
2    1  NaN
3    2    4
>>> df.groupby('1st')['2nd'].nth(0, dropna=True)
1st
1    3
2    4
Name: 2nd, dtype: float64

[tnir]

@behzadnouri You should pass either of None, 'any' or 'all' as dropna, instead of True. We should deprecate passing True to nth.

When you however pass either 'any' or 'all', there is still a bug.

>>> df = pd.DataFrame([[1],[2,3],[1],[2,4]], columns=['1st', '2nd'])
>>> df
   1st  2nd
0    1  NaN
1    2    3
2    1  NaN
3    2    4
>>> df.groupby('1st')['2nd'].nth(0, dropna='any')
1st
1    3
2    4
Name: 2nd, dtype: float64

>>> df.groupby('1st')['2nd'].nth(0, dropna='all')
1st
1    3
2    4
Name: 2nd, dtype: float64

Exchanging the order of selecting a column and nth(), we expect the correct results as follows:

>>> df.groupby('1st').nth(0, dropna='any')['2nd']
1st
1   NaN
2     3
Name: 2nd, dtype: float64

>>> df.groupby('1st').nth(0, dropna='all')['2nd']
1st
1   NaN
2     3
Name: 2nd, dtype: float64

[jreback]

@tnir yep, the argument is a bit too free and should be checked at the top of nth, rather than in the conditional (which misses some cases).

As always pull-requests to fix are welcome.

[tnir]

This is still reproducible with pandas-0.20.3 on Python 3.6:

$ python
Python 3.6.0 (default, Dec 26 2016, 15:43:50)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin

>>> import pandas as pd
>>> df = pd.DataFrame([[1],[2,3],[1],[2,4]], columns=['1st', '2nd'])
>>> df
   1st  2nd
0    1  NaN
1    2  3.0
2    1  NaN
3    2  4.0
>>> df.groupby('1st')['2nd'].nth(0, dropna='any')
1st
1    3.0
2    4.0
Name: 2nd, dtype: float64
>>> df.groupby('1st')['2nd'].nth(0, dropna='all')
1st
1    3.0
2    4.0
Name: 2nd, dtype: float64
>>> df.groupby('1st').nth(0, dropna='any')['2nd']
1st
1    NaN
2    3.0
Name: 2nd, dtype: float64
>>> df.groupby('1st').nth(0, dropna='all')['2nd']
1st
1    NaN
2    3.0
Name: 2nd, dtype: float64

[rhshadrach]

This now raises ValueError: For a DataFrame or Series groupby.nth, dropna must be either None, 'any' or 'all', (was passed True)..

The code from #11038 (comment) now results in

1    3.0
Name: 2nd, dtype: float64

1    3.0
Name: 2nd, dtype: float64

This deviates from the expected behavior mentioned there, but looks right to me. In particular, by performing selection dropna is now dropping all of group 1. Also note that the 1 in the result is not an indication of the group, but rather the index value in the row 1 2 3.0 of the original DataFrame for group 2.