Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: add Series.astype with the new tz dtype #11003

Merged
merged 2 commits into from
Sep 7, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 45 additions & 11 deletions doc/source/timeseries.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1753,22 +1753,56 @@ TZ aware Dtypes

.. versionadded:: 0.17.0

``Series/DatetimeIndex`` with a timezone naive value are represented with a dtype of ``datetime64[ns]``.
``Series/DatetimeIndex`` with a timezone **naive** value are represented with a dtype of ``datetime64[ns]``.

.. ipython:: python

dr = pd.date_range('20130101',periods=3)
dr
s = Series(dr)
s
dr_naive = pd.date_range('20130101',periods=3)
dr_naive
s_naive = Series(dr_naive)
s_naive

``Series/DatetimeIndex`` with a timezone aware value are represented with a dtype of ``datetime64[ns, tz]``.
``Series/DatetimeIndex`` with a timezone **aware** value are represented with a dtype of ``datetime64[ns, tz]``.

.. ipython:: python

dr = pd.date_range('20130101',periods=3,tz='US/Eastern')
dr
s = Series(dr)
s
dr_aware = pd.date_range('20130101',periods=3,tz='US/Eastern')
dr_aware
s_aware = Series(dr_aware)
s_aware

Both of these ``Series`` can be manipulated via the ``.dt`` accessor, see :ref:`here <basics.dt_accessors>`.
See the :ref:`docs <timeseries.dtypes>` for more details.

Further more you can ``.astype(...)`` timezone aware (and naive).

.. ipython:: python

# make this naive
s_aware.astype('datetime64[ns]')

# convert
s_aware.astype('datetime64[ns, CET]')
s_naive.astype('datetime64[ns, CET]')

.. note::

Using the ``.values`` accessor on a ``Series``, returns an numpy array of the data.
These values are converted to UTC, as numpy does not currently support timezones (even though it is *printing* in the local timezone!).

.. ipython:: python

s_naive.values
s_aware.values

Further note that once converted to a numpy array these would lose the tz tenor.

.. ipython:: python

Series(s_aware.values)

However, these can be easily converted

.. ipython:: python

Both of these ``Series`` can be manipulated via the ``.dt`` accessor, see the :ref:`docs <basics.dt_accessors>` as well.
Series(s_aware).dt.tz_localize('UTC').dt.tz_convert('US/Eastern')
11 changes: 7 additions & 4 deletions doc/source/whatsnew/v0.17.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -447,9 +447,9 @@ Datetime with TZ

We are adding an implementation that natively supports datetime with timezones. A ``Series`` or a ``DataFrame`` column previously
*could* be assigned a datetime with timezones, and would work as an ``object`` dtype. This had performance issues with a large
number rows. (:issue:`8260`, :issue:`10763`)
number rows. See the :ref:`docs <timeseries.timezone_series>` for more details. (:issue:`8260`, :issue:`10763`).

The new implementation allows for having a single-timezone across all rows, and operating on it in a performant manner.
The new implementation allows for having a single-timezone across all rows, with operations in a performant manner.

.. ipython:: python

Expand All @@ -469,13 +469,15 @@ This uses a new-dtype representation as well, that is very similar in look-and-f
.. ipython:: python

df['B'].dtype
type(df['B']).dtype
type(df['B'].dtype)

.. note::

There is a slightly different string repr for the underlying ``DatetimeIndex`` as a result of the dtype changes, but
functionally these are the same.

Previous Behavior:

.. code-block:: python

In [1]: pd.date_range('20130101',periods=3,tz='US/Eastern')
Expand All @@ -486,12 +488,13 @@ This uses a new-dtype representation as well, that is very similar in look-and-f
In [2]: pd.date_range('20130101',periods=3,tz='US/Eastern').dtype
Out[2]: dtype('<M8[ns]')

New Behavior:

.. ipython:: python

pd.date_range('20130101',periods=3,tz='US/Eastern')
pd.date_range('20130101',periods=3,tz='US/Eastern').dtype


.. _whatsnew_0170.api_breaking.convert_objects:

Changes to convert_objects
Expand Down
21 changes: 21 additions & 0 deletions pandas/core/internals.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
array_equivalent, _maybe_convert_string_to_object,
is_categorical, needs_i8_conversion, is_datetimelike_v_numeric,
is_internal_type)
from pandas.core.dtypes import DatetimeTZDtype

from pandas.core.index import Index, MultiIndex, _ensure_index
from pandas.core.indexing import maybe_convert_indices, length_of_indexer
Expand Down Expand Up @@ -1868,6 +1869,26 @@ def __init__(self, values, placement,
fastpath=True, placement=placement,
**kwargs)

def _astype(self, dtype, **kwargs):
"""
these automatically copy, so copy=True has no effect
raise on an except if raise == True
"""

# if we are passed a datetime64[ns, tz]
if com.is_datetime64tz_dtype(dtype):
dtype = DatetimeTZDtype(dtype)

values = self.values
if getattr(values,'tz',None) is None:
values = DatetimeIndex(values).tz_localize('UTC')
values = values.tz_convert(dtype.tz)
return self.make_block(values)

# delegate
return super(DatetimeBlock, self)._astype(dtype=dtype, **kwargs)


def _can_hold_element(self, element):
if is_list_like(element):
element = np.array(element)
Expand Down
14 changes: 14 additions & 0 deletions pandas/tests/test_series.py
Original file line number Diff line number Diff line change
Expand Up @@ -1072,6 +1072,20 @@ def test_constructor_with_datetime_tz(self):
expected = Series(DatetimeIndex(s._values).asobject)
assert_series_equal(result, expected)

result = Series(s.values).dt.tz_localize('UTC').dt.tz_convert(s.dt.tz)
assert_series_equal(result, s)

# astype - datetime64[ns, tz]
result = Series(s.values).astype('datetime64[ns, US/Eastern]')
assert_series_equal(result, s)

result = Series(s.values).astype(s.dtype)
assert_series_equal(result, s)

result = s.astype('datetime64[ns, CET]')
expected = Series(date_range('20130101 06:00:00',periods=3,tz='CET'))
assert_series_equal(result, expected)

# short str
self.assertTrue('datetime64[ns, US/Eastern]' in str(s))

Expand Down