BUG: Groupby NaT Handling

[sinhrks]

xref #9236

There seems to be an inconsistency in some GroupBy methods when NaT is included in the group key.

• GroupBy.groups includes NaT as a key.
• GroupBy.ngroups doesn't count NaT.
• GroupBy.__iter__ doesn't return NaT group.
• GroupBy.get_group fails when NaT is specified.

I understand NaT should be included in the group key according to other function's behaviour, such as dropna. Is it OK to fix it to include NaT?

import pandas as pd
import numpy as np
>>> df = pd.DataFrame({'values': np.random.randn(8), 
                   'dt': [np.nan, pd.Timestamp('2013-01-01'), np.nan, pd.Timestamp('2013-02-01'),
                          np.nan, pd.Timestamp('2013-02-01'), np.nan, pd.Timestamp('2013-01-01')]})
>>> grouped = df.groupby('dt')

>>> grouped.groups
{numpy.datetime64('NaT'): [0, 2, 4, 6], numpy.datetime64('2013-01-01T09:00:00.000000000+0900'): [1, 7], numpy.datetime64('2013-02-01T09:00:00.000000000+0900'): [3, 5]}

>>> grouped.ngroups
2

>>> grouped.indices
# ValueError: DatetimeIndex with NaT cannot be converted to object

>>> grouped.get_group(pd.NaT)
ValueError: DatetimeIndex with NaT cannot be converted to object

[jreback]

nope, these should be handled like nan groups (excluded). http://pandas-docs.github.io/pandas-docs-travis/groupby.html#na-group-handling

maybe add some tests for this explicity (and NaT if its the only key should not be a group)

[jreback]

see also #5456

[sinhrks]

Thanks. I'm working on #3729 now (hopefully finish it soon), and will handle NaT like nan. I'll add some tests at that time.

[sinhrks]

If group key contains NaT, get_group for valid group also doesn't work... It looks to be caused by the spec #1348.

# continued from above example
>>> grouped.get_group(pd.Timestamp('2013-02-01'))
# ValueError: DatetimeIndex with NaT cannot be converted to object

[jreback]

group key should not contain NaT that is an error (just as it should not contain nan).

[sinhrks]

Understood the principle, and I think it is better to get_group works even if the key contains NaT (because it works when the key contains nan).