DataFrame combine_first() loses timezone information for datetime columns

[iyer]

xref addl example in #13650

combine_first() loses timezone information for datetime columns

dts1 = pd.date_range('20150101','20150105',tz='UTC')
df1 = pd.DataFrame({'DATE':dts1})
dts2 = pd.date_range('20150103','20150105',tz='UTC')
df2 = pd.DataFrame({'DATE':dts2})
df = df1.combine_first(df2)
df.DATE[0].tz # this shows up as None

[jreback]

this is actually some older code combine which really needs to be pushed into the internal classes, e.g. see this issue #3025 so a bit non-trivial. that said, pull-requests are welcome!

[iyer]

Thanks. I'll await the change when it happens
In the meantime I'll consume the DATE in my index, so as to work around the issue

[jreback]

well pull-requests are always welcome. There are a ton of issues. So this will prob not be addressed for quite some time.

[terrytangyuan]

@jreback Could you give some instructions on how to fix this bug?

[jreback]

well, this needs to be pushed to the block manager (e.g. need a method Block.combine), then all of this dtype handling can be dispatched via the internal block types. Further #10477 will allow this to handle nicely.

Doing it now is going to be a bit hacky.

So if you want to look to start moving it internally as in #3025 would be a good start.

[phretor]

FWIW, I noticed that, in certain cases (i.e., when you are well aware of the TZs), this workaround can be used:

In [16]: n['ts'].index.tz
Out[16]: <DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>

In [17]: n['ev'].index.tz
Out[17]: <DstTzInfo 'America/Los_Angeles' LMT-1 day, 16:07:00 STD>

In [18]: n['ts'].combine_first(n['ts']).index.tz
Out[18]: <DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>

In [19]: n['ts'].combine_first(n['ev']).index.tz
Out[19]: <UTC>

In [20]: n['ts'].combine_first(n['ev']).tz_convert('America/Los_Angeles').index.tz
Out[20]: <DstTzInfo 'America/Los_Angeles' LMT-1 day, 16:07:00 STD>

Warning: this is my first dig into pandas.

[sinhrks]

This looks to be fixed on master (I don't look into detail, but maybe by changes in concat_compat?)

Adding tests.

[sinhrks]

Ah, even though tz is preserved, datetimes are incorrectly shifts (maybe the same as #12619).

dts1 = pd.DatetimeIndex(['2011-01-01', 'NaT', '2011-01-03', '2011-01-04'], tz='US/Eastern')
df1 = pd.DataFrame({'DATE': dts1}, index=[1, 3, 5, 7])
dts2 = pd.DatetimeIndex(['2012-01-01', '2012-01-02', '2012-01-03'], tz='US/Eastern')
df2 = pd.DataFrame({'DATE':dts2}, index=[2, 4, 5])
df = df1.combine_first(df2)
df
#                        DATE
# 1 2011-01-01 05:00:00-05:00
# 2 2012-01-01 05:00:00-05:00
# 3                       NaT
# 4 2012-01-02 05:00:00-05:00
# 5 2011-01-03 05:00:00-05:00
# 7 2011-01-04 05:00:00-05:00

[jorisvandenbossche]

Note that the example at the top is not really showing the issue (that seems to work in master)

[jackalack]

why was this closed? This still appears to be an issue

[jorisvandenbossche]

@jackalack Can you provide a reproducible example that shows this?