BUG: groupby.first/last with nans

[jreback]

This is incorrect, as this is applied column by column (as they are different dtypes)
so _first_compat should first compute the mask then use it.

from SO

In [18]: df = pd.DataFrame([{'id':"a",'val':np.nan, 'val2':-1},{'id':"a",'val':'TREE','val2':15}])

In [19]: df
Out[19]: 
  id   val  val2
0  a   NaN    -1
1  a  TREE    15

In [20]: df.groupby('id').first()
Out[20]: 
     val  val2
id            
a   TREE    -1

[behzadnouri]

this is designed behavior, and does not depend on types:

>>> df
   jim  joe  jolie
0    0    1    NaN
1    0  NaN      2
>>> df.dtypes
jim      float64
joe      float64
jolie    float64
dtype: object
>>> df._data.nblocks
1
>>> df.groupby('jim').first()
     joe  jolie
jim            
0      1      2

[jreback]

I think it's incorrect for the frame case (series case ok)

[jorisvandenbossche]

Related: #6732

[jreback]

I think this is confusing at the least. So you have multiple values on the first row, some of which are NaN. I guess .head(1) is the 'correct' answer here. And .first() does its thing. I think maybe this should NOT be the default behavior.

[TomAugspurger]

So the documented behavior of Groupby.first / Groupby.last is the first / last non-NA value? Anything to do here?

[WillAyd]

There's some inconsistency between first and nth(0) that needs to be addressed here:

In [10]: df.groupby('id').nth(0)
Out[10]: 
    val  val2
id           
a   NaN    -1

[rhshadrach]

@WillAyd - first is designed to work column by column whereas nth selects a single row. So it seems to me this inconsistency is purposeful (but agree it's confusing).

To me an ideal solution would allow a user to

• get the nth value, either selecting a single row or column-by-column (note: this is different than using axis=1 in the grouby(...) call)
• first/last as a shortcut for nth(0) and nth(-1) as these are very common, again by either selecting a single row or column-by-column

If we had this, then it'd be best to align nth/first so the default behavior of nth(0) is the same as first, and similarly with last.

[WillAyd]

I think your second bullet feels right, and might have the added bonus of simplifying our code paths

[NumberPiOso]

take

[NumberPiOso]

@rhshadrach , I understand that modifying first and last methods to point to nth would be the way to go. This would cause this behaviour.

In [10]: df.groupby('id').nth(0)
Out[10]: 
    val  val2
id           
a   NaN    -1

In [10]: df.groupby('id').first()
Out[10]: 
    val  val2
id           
a   NaN    -1

Then, we would include another parameter to decide if it is the nth value by row or column-by-column.

[rhshadrach]

@NumberPiOso - Certainly we should include the parameter first if going that route before changing the default behavior of first/last. However, I do not feel certain that is the correct approach. I would have to guess that first/last are common operations, at least more so than nth. Is changing the default of first to mean "first row" more useful than "first value in each column"? That is very unclear to me. It's at this point that I wonder if changing the behavior here is more disruptive than helpful.

As a tangential aside, I've always found treating nans differently in first/last/etc more surprising than beneficial, but perhaps that's just my use case. However this is pretty consistent across pandas and I certainly don't find it sever enough to merit a change to the default behavior (or at the very least, merit me arguing for a change!).

[NumberPiOso]

This is true.
As a user, I tend to use first and last a lot more than nth and by themselves each one of this methods are consistent. So I would not change the methods either.