API: Should extended dtype work as the same as np.dtype?

[sinhrks]

We can create / convert using datetimetz dtype, but it doesn't work some cases.

s = pd.Series([pd.Timestamp('2011-01-31', tz='US/Eastern')])
s
#0   2011-01-31 00:00:00-05:00
# dtype: datetime64[ns, US/Eastern]

# OK
s.astype('datetime64[ns, Asia/Tokyo]')
#0   2011-01-31 14:00:00+09:00
# dtype: datetime64[ns, Asia/Tokyo]

astype

# numpy (OK)
pd.Series([1296432000000000000]).astype('datetime64[ns]')
#0   2011-01-31
# dtype: datetime64[ns]

# extended (NG)
pd.Series([1296432000000000000]).astype('datetime64[ns, Asia/Tokyo]')
# TypeError: Invalid datetime unit in metadata string "[ns, Asia/Tokyo]"

dtype arg

# extended (OK ? I think the result should be 2011-01-31 00:00:00-05:00... see below)
pd.Series([1296432000000000000], dtype='datetime64[ns, US/Eastern]')
#0   2011-01-31 05:00:00-05:00
# dtype: datetime64[ns, US/Eastern]

# ref
pd.Series([1296432000000000000], dtype='datetime64[ns]').dt.tz_localize('US/Eastern')
#0   2011-01-31 00:00:00-05:00
# dtype: datetime64[ns, US/Eastern]

# extended (NG)
pd.Series([pd.Timestamp('2011-01-01', tz='US/Eastern')], dtype='datetime64[ns, US/Eastern]')
# TypeError: data type not understood

[sinhrks]

Found dtype arg issue affects to boxing (#12752) issue. Must fix this first.

s = pd.Series([pd.Timestamp('2011-01-01 09:00', tz='US/Eastern')])
s
# 0   2011-01-01 09:00:00-05:00
# dtype: datetime64[ns, US/Eastern]

s._data._block.values.asi8
# array([1293890400000000000])

# NG, must be 2011-01-01 09:00:00-05:00
pd.Series(s._data._block.values.asi8, dtype='datetime64[ns, US/Eastern]')
# 0   2011-01-01 14:00:00-05:00
# dtype: datetime64[ns, US/Eastern]

[sinhrks]

Timestamp/TDI holds internal repr in int, and it refers to absolute time of GMT.

int(pd.Timestamp('2011-01-01').asm8)
# 1293840000000000000

int(pd.Timestamp('2011-01-01', tz='US/Eastern').asm8)
# 1293858000000000000

Thus, Timestamp creation using int should have the same internal repr.

pd.Timestamp(1293858000000000000)
# Timestamp('2011-01-01 05:00:00')

int(pd.Timestamp(1293858000000000000).asm8)
# 1293858000000000000

pd.Timestamp(1293858000000000000, tz='US/Eastern')
# Timestamp('2011-01-01 00:00:00-0500', tz='US/Eastern')

int(pd.Timestamp(1293858000000000000, tz='US/Eastern').asm8)
# 1293858000000000000

However, the rule is not applied to DTI. DTI must work the same as Timestamp, otherwise boxing against scalar / array outputs different results.

# OK, without TZ
pd.DatetimeIndex([1293858000000000000])
# DatetimeIndex(['2011-01-01 05:00:00'], dtype='datetime64[ns]', freq=None)

pd.DatetimeIndex([1293858000000000000]).asi8
# array([1293858000000000000])

# NG, with TZ slides internal repr
pd.DatetimeIndex([1293858000000000000], tz='US/Eastern')
# DatetimeIndex(['2011-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

pd.DatetimeIndex([1293858000000000000], tz='US/Eastern').asi8
# array([1293876000000000000])

[jreback]

yeah, I think there's a bug somewhere where I am converting on a localized UTC somewhere