Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

REF: DatetimeLikeArray #24024

Merged
merged 220 commits into from
Jan 2, 2019
Merged

REF: DatetimeLikeArray #24024

merged 220 commits into from
Jan 2, 2019

Conversation

TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Nov 30, 2018 •
edited
Loading

TODO:

n.b.: right now I have a few other PRs in this branch (#23601, #23990, #24023). The eventual diff is at TomAugspurger/pandas@disown-tz-only-target...TomAugspurger:disown-tz-only-rebased.

Also, I would say this isn't quite ready for review yet, but I've been promising it for a while. I'm going to try to split off a bit more into separate PRs.

This implements DatetimeArray and TimedeltaArray, arrays for storing datetime
(tz naive or aware) and timedelta data.

High-level Design:

Everything here applies equally to TimedeltaIndex / TimedeltaArray, other than
references to DatetimeTZDtype. We don't have an ExtensionType for Timedelta.

DatetimeIndex's data is now a DatetimeArray, rather than an ndarray, and its dtype
is dtype::Union['datetime64[ns]', DatetimeTZDtype].

Class Hierarchy: We've split DatetimeIndex to compose a DatetimeArray,
rather than inherit it.

Old - DatetimeIndex: dti_master

New - DatetimeIndex: dti_pr

New - DatetimeArray: dta_pr

Internals I've mostly given up here... Things are basically as they were
before, with the exception that DatetimeTZBlock now extends ExtensionBlock. In
particular tz-naive data is still stored in a DatetimeBlock, and is still
consolidatable. This isn't elegant, but we've lived with it for this long.
Future releases can clean it up, possibly when we rewrite / cythonize the block
manager, possibly before.

Types of changes

  1. Some changes from x.view('i8') to x.astype('i8', copy=False) to
    support getting i8 values from {ndarray, DatetimeIndex, DatetimeArray}.
    We could revert those if we implemented DatetimeArray.view, but I don't
    think we should. However, this is slower than .view for ndarrays, so
    I'm going to find another way.
  2. Many places where we did series.values instead of series._values.
  3. Moves from PeriodArray / PeriodIndex to a base class. I've annotated these
    with NB: Moved from...
  4. Test skips / xfails. I intended to resolve all of these over the next few
    days.

TODO:

  • Split out DatelikeOps and TimelikeOps changes (REF: Refactor Date/TimeLikeOps #24038)
    • independent tests for DatelikeOps & TimelikeOps methods on arrays.
  • re-clean up the constructors. I may have some cruft from rebasing
  • identify shared code in PeriodArray / PeriodIndex that can be moved up
  • all the xfailed tests
  • identify code that was special casing Datetime-dtyps (like is_extension_type).
  • API: Series[datetimetz].unique: ndarray[object] or DatetimeArray?

Closes #23185
Closes #23932
closes #24465

@TomAugspurger
Squashed commit of the following:
e9c8418 
commit c23c9e2
Merge: 50e1aeb dc8d35a
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 09:10:00 2018 -0600

    Merge remote-tracking branch 'upstream/master' into dtype-only

commit 50e1aeb
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 13:00:35 2018 -0600

    PeriodDtype needs freq

commit 10d2c8a
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:30:29 2018 -0600

    refactor construct_from_string

commit c14b45f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:19:29 2018 -0600

    fix unpickling

commit 7ab2a74
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 10:16:32 2018 -0600

    Remove _coerce_to_dtype

commit 6cc9ce5
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 08:54:35 2018 -0600

    Fixed tz name

commit e0b7b77
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 07:05:04 2018 -0600

    Updates

    * Use pandas_dtype
    * removed cache_readonly

commit ad2723c
Merge: 9e4faf8 d9a037e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 06:39:25 2018 -0600

    Merge remote-tracking branch 'upstream/master' into dtype-only

commit 9e4faf8
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 29 06:14:05 2018 -0600

    cache readonly

commit 7e6d8ea
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 21:45:21 2018 -0600

    Restore construct_array_type

commit 2fa4bb0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 21:14:10 2018 -0600

    unxfail test, remove caching bit

commit 1ca7fa4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 17:04:40 2018 -0600

    REF/API: DatetimeTZDtype

    * Remove magic constructor from string
    * Remove Caching

    The remaining changes in the DatetimeArray PR will be to

    1. Inherit from ExtensionDtype
    2. Implement construct_array_type
    3. Register
@TomAugspurger
Squashed commit of the following:
80d9576 
commit e7cc2ac
Merge: 740f9e5 30c1290
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 16:16:03 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 740f9e5
Merge: a35399e db8d33e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 28 07:42:38 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit a35399e
Merge: d9df6bf 6fad5a0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:07:41 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit d9df6bf
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:04:56 2018 -0600

    correct boxing

commit d84cc02
Merge: 2b5fe25 deb7b4d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 19:02:02 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 2b5fe25
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:38:03 2018 -0600

    BUG: Fixed SparseArray formatter

    We want to fall back to the implementation in formats.

commit ef390fc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:29:42 2018 -0600

    Updates: misc

    * whatsnew
    * docstrings

commit 5c253a4
Merge: 27db397 2946745
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 19 06:22:28 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 27db397
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 15 09:05:47 2018 -0600

    simplify formatter

commit fc4279d
Merge: a926dca 8af7637
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Thu Nov 15 08:06:48 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit a926dca
Merge: 62b1e2f e413c49
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Nov 14 15:21:09 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 62b1e2f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 11:19:07 2018 -0600

    remove bytes

commit 2364546
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:11:27 2018 -0600

    fixup! fixup! use repr

commit 439f2f8
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:05:50 2018 -0600

    fixup! use repr

commit 221cee9
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 09:04:31 2018 -0600

    use repr

commit e5f6976
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:19:01 2018 -0600

    wip

commit ebadf6f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:11:12 2018 -0600

    FutureWarning -> DeprecationWarning

commit 9116930
Merge: 0f4083e 011b79f
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 08:10:53 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 0f4083e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 06:59:20 2018 -0600

    remove periodarray

commit 708dd75
Merge: 1b93bf0 3592a46
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Mon Nov 12 06:18:59 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 1b93bf0
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sun Nov 11 14:52:27 2018 -0600

    update repr tests

commit 5b291d5
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:33:20 2018 -0600

    lint

commit 4d343ea
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:30:55 2018 -0600

    unicode

commit baee6b2
Merge: 5d8d2fc 383d052
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 16:27:49 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit 5d8d2fc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:45:55 2018 -0600

    unicode

commit 2fd3d5d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:45:22 2018 -0600

    unicode

commit ff0c998
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 14:41:43 2018 -0600

    fixup

commit 5b07906
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:52:16 2018 -0600

    py3 fixup

commit 60e0d02
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:36:39 2018 -0600

    isort

commit 445736d
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:30:57 2018 -0600

    unicode, bytes

commit b312fe4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Sat Nov 10 06:22:33 2018 -0600

    revert interval

commit d8e7ba4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 21:08:41 2018 -0600

    py2 compat

commit 48e55cc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 20:52:28 2018 -0600

    fixup interval

commit e2b1941
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 16:05:33 2018 -0600

    updates

commit 1635b73
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 15:35:53 2018 -0600

    try this

commit 5a2e1e4
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 15:02:39 2018 -0600

    format

commit 193747e
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:56:09 2018 -0600

    update docs, type

commit 6e64b7b
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:47:45 2018 -0600

    more cleanup

commit 37638cc
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 14:29:25 2018 -0600

    wip

commit 4e0d91f
Merge: ecfcd72 efd1844
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:36:13 2018 -0600

    Merge remote-tracking branch 'upstream/master' into ea-repr

commit ecfcd72
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:33:25 2018 -0600

    clean

commit 1885a97
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 13:22:29 2018 -0600

    na formatter

commit fef04e6
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:19:58 2018 -0600

    compat

commit 6e76b51
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:16:03 2018 -0600

    test for warning

commit ace62aa
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 12:07:47 2018 -0600

    Deprecate formatting_values

commit 0fdbfd3
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 9 09:26:57 2018 -0600

    wip
@TomAugspurger
Squashed commit of the following:
acd5b6d 
commit f35e8dd
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:12:22 2018 -0600

    extraneous comment

commit ce353af
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:11:30 2018 -0600

    release note

commit 09e07d2
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Fri Nov 30 13:00:39 2018 -0600

    PERF: fixup
@TomAugspurger
REF: DatetimeArray
f364f77
@TomAugspurger TomAugspurger added Refactor Internal refactoring of code Datetime Datetime data dtype labels Nov 30, 2018
@TomAugspurger TomAugspurger added this to the 0.24.0 milestone Nov 30, 2018
@jbrockmendel
Copy link
Member

Thanks for putting this up, I'll take a look. How did you produce those inheritance graphs? Those are neat.

@TomAugspurger
Copy link
Contributor Author

Thanks for putting this up, I'll take a look. How did you produce those inheritance graphs? Those are neat.

lightly modified http://www.phyast.pitt.edu/~micheles/python/drawMRO.html

@jbrockmendel jbrockmendel mentioned this pull request Nov 30, 2018
Merged
TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Dec 1, 2018
@TomAugspurger
REF: Refactor Datetimelike delegation
e614a80 
This is a generalization of PeriodIndex's dispatching to
PeriodArray, without any actual changes yet. This is split
from pandas-dev#24024, where DatetimeIndex and TimedeltaIndex will
implement and inherit from delgates similiar to PeriodDelegateMixin.
@TomAugspurger TomAugspurger mentioned this pull request Dec 1, 2018
Merged
@TomAugspurger TomAugspurger mentioned this pull request Dec 2, 2018
Merged
jreback pushed a commit that referenced this pull request Jan 3, 2019
@jbrockmendel @jreback
CLN: Follow-ups to #24024 (#24573)
1fc76b8
@jreback jreback mentioned this pull request Jan 3, 2019
Closed
@jorisvandenbossche jorisvandenbossche mentioned this pull request Jan 3, 2019
Merged
This was referenced Jan 6, 2019
Merged
Closed
@shashank88 shashank88 mentioned this pull request Jan 28, 2019
Closed
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement independent parts of pandas-dev#24024 (pandas-dev#24276)
cc6ac5c 
* implement independent parts of pandas-dev#24024

* move monotonic checks up
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement _index_data parts of pandas-dev#24024 (pandas-dev#24379)
64ec6e1
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Rendering Methods portions of pandas-dev#24024 (pandas-dev#24392)
2a28e53
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
REF: Pieces broken off of pandas-dev#24024 (pandas-dev#24364)
dd6c082
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement Delegate portions of pandas-dev#24024 (pandas-dev#24463)
06f2c42
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement astype portion of pandas-dev#24024 (pandas-dev#24405)
0277ee7
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
searchsorted, repeat broken off from pandas-dev#24024 (pandas-dev#24461)
5a9816f
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
REF: Stop mixing DTA/TDA into DTI/TDI (pandas-dev#24476)
46aa691 
* implement _index_data parts of pandas-dev#24024

* implement _eadata, dispatch arithmetic methods to it

* dont mix DatetimeLikeArrayMixin into DatetimeIndexOpsMixin

* dont inherit TimedeltaIndex from TimedeltaArray

* dont inherit from DatetimeArray

* use ea_passthrough

* remove previously-overriden overridings

* stop double-mixing

* stop over-writing

* handle+test object arrays

* Remove unused import

* flake8 fixup

* edits per comments
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Implement reductions from pandas-dev#24024 (pandas-dev#24484)
c710e5a
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Misc separable pieces of pandas-dev#24024 (pandas-dev#24488)
190290e
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@TomAugspurger @Pingviinituutti
REF: DatetimeLikeArray (pandas-dev#24024)
c9203eb
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
CLN: Follow-ups to pandas-dev#24024 (pandas-dev#24573)
72c8401
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement independent parts of pandas-dev#24024 (pandas-dev#24276)
4205e45 
* implement independent parts of pandas-dev#24024

* move monotonic checks up
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement _index_data parts of pandas-dev#24024 (pandas-dev#24379)
7d8cd68
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Rendering Methods portions of pandas-dev#24024 (pandas-dev#24392)
9ffc15d
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
REF: Pieces broken off of pandas-dev#24024 (pandas-dev#24364)
33ae55e
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement Delegate portions of pandas-dev#24024 (pandas-dev#24463)
672ae96
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
implement astype portion of pandas-dev#24024 (pandas-dev#24405)
83b902c
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
searchsorted, repeat broken off from pandas-dev#24024 (pandas-dev#24461)
e239ab3
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
REF: Stop mixing DTA/TDA into DTI/TDI (pandas-dev#24476)
73bdcdc 
* implement _index_data parts of pandas-dev#24024

* implement _eadata, dispatch arithmetic methods to it

* dont mix DatetimeLikeArrayMixin into DatetimeIndexOpsMixin

* dont inherit TimedeltaIndex from TimedeltaArray

* dont inherit from DatetimeArray

* use ea_passthrough

* remove previously-overriden overridings

* stop double-mixing

* stop over-writing

* handle+test object arrays

* Remove unused import

* flake8 fixup

* edits per comments
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Implement reductions from pandas-dev#24024 (pandas-dev#24484)
f56966a
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
Misc separable pieces of pandas-dev#24024 (pandas-dev#24488)
85663ea
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@TomAugspurger @Pingviinituutti
REF: DatetimeLikeArray (pandas-dev#24024)
782e8b6
Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019
@jbrockmendel @Pingviinituutti
CLN: Follow-ups to pandas-dev#24024 (pandas-dev#24573)
59becee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WillAyd WillAyd WillAyd left review comments

@jreback jreback jreback approved these changes

@jorisvandenbossche jorisvandenbossche jorisvandenbossche left review comments

@jbrockmendel jbrockmendel jbrockmendel left review comments

Assignees
No one assigned
Labels
Datetime Datetime data dtype Refactor Internal refactoring of code
Projects
None yet
Milestone
0.24.0
7 participants
@TomAugspurger @jbrockmendel @jorisvandenbossche @jreback @h-vetinari @alimcmaster1 @WillAyd