BUG/TST/REF: Datetimelike Arithmetic Methods #23215

jbrockmendel · 2018-10-17T23:09:56Z

@TomAugspurger @jorisvandenbossche @jreback as discussed

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…, add missing tests

jbrockmendel · 2018-10-17T23:12:55Z

pandas/core/arrays/datetimelike.py

-        attribs = self._get_attributes_dict()
-        if not is_period_dtype(self):
-            attribs['freq'] = None
-        return self._simple_new(result, **attribs)


This is only used once outside of tests. Much less verbose to inline.

hah I just made a comment on PeriodArray about re-using this.

jbrockmendel · 2018-10-17T23:15:05Z

pandas/core/arrays/timedeltas.py

+        if other is NaT:
+            # In this case we specifically interpret NaT as a datetime, not
+            # the timedelta interpretation we would get by returning self + NaT
+            result = self.asi8.view('m8[ms]') + NaT.to_datetime64()
            return DatetimeArrayMixin(result)


This fixes the bug in TimedeltaIndex + np.timedelta64('NaT')

jbrockmendel · 2018-10-17T23:15:41Z

pandas/core/arrays/timedeltas.py

+        result = checked_add_with_arr(i8, other.value,
+                                      arr_mask=self._isnan)
+        result = self._maybe_mask_results(result, fill_value=iNaT)
+        return DatetimeArrayMixin(result, tz=other.tz)


This fixes a bug with failing to attach tz

jbrockmendel · 2018-10-17T23:16:02Z

pandas/tests/arithmetic/test_period.py

+    @pytest.mark.parametrize('other', [pd.Timestamp.now(),
+                                       pd.Timestamp.now().to_pydatetime(),
+                                       pd.Timestamp.now().to_datetime64()])
+    def test_pi_add_sub_datetime(self, other):


This behavior is unchanged; it was just an untested case.

pandas/core/arrays/datetimes.py

codecov · 2018-10-18T03:49:19Z

Codecov Report

Merging #23215 into master will decrease coverage by <.01%.
The diff coverage is 95%.

@@            Coverage Diff             @@
##           master   #23215      +/-   ##
==========================================
- Coverage   92.16%   92.16%   -0.01%     
==========================================
  Files         166      166              
  Lines       51224    51206      -18     
==========================================
- Hits        47212    47192      -20     
- Misses       4012     4014       +2

Flag	Coverage Δ
#multiple	`90.59% <95%> (-0.01%)`	⬇️
#single	`42.24% <21.66%> (+0.01%)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimelike.py	`94.01% <100%> (+0.79%)`	⬆️
pandas/core/arrays/timedeltas.py	`94.3% <100%> (-0.18%)`	⬇️
pandas/core/indexes/timedeltas.py	`90.62% <100%> (ø)`	⬆️
pandas/core/indexes/period.py	`91.04% <84.21%> (-1.19%)`	⬇️
pandas/core/arrays/period.py	`97.48% <93.54%> (-0.09%)`	⬇️
pandas/core/arrays/datetimes.py	`97.9% <96%> (+0.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62a15fa...0466b9c. Read the comment docs.

pep8speaks · 2018-10-18T17:47:00Z

Hello @jbrockmendel! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/core/arrays/datetimelike.py !
There are no PEP8 issues in the file pandas/core/arrays/datetimes.py !
There are no PEP8 issues in the file pandas/core/arrays/period.py !
There are no PEP8 issues in the file pandas/core/arrays/timedeltas.py !
There are no PEP8 issues in the file pandas/tests/arithmetic/test_period.py !
There are no PEP8 issues in the file pandas/tests/arithmetic/test_timedelta64.py !
There are no PEP8 issues in the file pandas/tests/indexes/datetimes/test_ops.py !
There are no PEP8 issues in the file pandas/tests/indexes/period/test_ops.py !
There are no PEP8 issues in the file pandas/tests/indexes/timedeltas/test_ops.py !
There are no PEP8 issues in the file pandas/util/testing.py !
Complete extra results for this file :

file_to_check.py:2522:-258: W605 invalid escape sequence '('
file_to_check.py:2522:-255: W605 invalid escape sequence ')'

jbrockmendel · 2018-10-18T19:11:42Z

@datapythonista where should I be looking for the explanation of why the linting is failing? It used to tell me right above the "exited wth code 1" line

From Travis log:

============== 4 passed, 4 deselected, 1 warnings in 0.24 seconds ==============
Doctests top-level reshaping functions DONE
The command "ci/code_checks.sh" exited with 1.

TomAugspurger · 2018-10-18T19:45:33Z

https://travis-ci.org/pandas-dev/pandas/jobs/443307753#L2918

I search for ci/code_checks.sh. We should be passing a -q to the doctests I think.

jorisvandenbossche · 2018-10-18T20:27:54Z

It's the imports I think?

Check import format using isort
ERROR: /home/travis/build/pandas-dev/pandas/pandas/core/indexes/range.py Imports are incorrectly sorted.
Check import format using isort DONE

pandas/core/arrays/datetimelike.py

jbrockmendel · 2018-10-18T20:46:15Z

It's the imports I think?

Yah, I think #23231 should have just fixed it. I'll re-push in a bit.

jbrockmendel · 2018-10-19T02:24:52Z

Updated with additional bugfixes:

DateOffset + (PeriodIndex|DataFrame) no longer raises TypeError
PeriodIndex[freq.n!=1] + DateOffset is no longer incorrect

pandas/core/arrays/datetimes.py

pandas/core/arrays/period.py

jreback · 2018-10-26T01:58:41Z

ping on green

jorisvandenbossche

You did a rename of things like _sub_datelike_dti to _sub_datetime_arraylike, which is good, but the same could be done for _sub_delta_tdi and others (can maybe also be left for another PR, as those are touched less here)

Added some minor comments/questions, looks good to me for the rest

pandas/core/arrays/period.py

jorisvandenbossche · 2018-10-26T10:28:27Z

pandas/core/arrays/period.py

@@ -844,7 +753,7 @@ def _add_delta_td(self, other):
        #  delta_to_nanoseconds(delta).  Because delta here is an integer,
        #  delta_to_nanoseconds will return it unchanged.
        ordinals = super(PeriodArray, self)._add_delta_td(delta)
-        return type(self)(ordinals, self.freq)
+        return ordinals

    def _add_delta_tdi(self, other):


The superclass version seems to return integers, but this an actual PeriodArray?

Can you add a comment indicating the types?

The superclass version seems to return integers, but this an actual PeriodArray?

Both this and the superclass version return i8 array. The superclass has a brief docstring to that effect.

Can you then add a comment on this function indicating the types? That really makes it easier to read through this code

But, to my original question: this function uses _addsub_int_array, which is defined a little bit above to return a PeriodArray, and not a i8 array.
I assume in practice this is not really a problem, because the code that converts the i8 array to PeriodArray, will also accept a PeriodArray.

Good catch, I must have lost track of this somewhere in the rebase process. Fixed to correctly return an i8 array, and brief docstrings added indicating types.

pandas/core/arrays/period.py

jbrockmendel · 2018-10-26T20:34:11Z

@jorisvandenbossche anything else to address or can i ping jeff?

jreback · 2018-10-26T20:34:55Z

will look tonight

…rith

jreback

minor comments, needds a rebase, ping on green.

jreback · 2018-10-28T03:39:37Z

pandas/core/arrays/datetimelike.py

@@ -221,7 +221,7 @@ def hasnans(self):
        """ return if I have any nans; enables various perf speedups """
        return self._isnan.any()

-    def _maybe_mask_results(self, result, fill_value=None, convert=None):
+    def _maybe_mask_results(self, result, fill_value=iNaT, convert=None):
        """
        Parameters
        ----------


can you update the doc-string for fill_value

jreback · 2018-10-28T03:41:18Z

pandas/core/arrays/period.py

+        #  delta_to_nanoseconds(delta).  Because delta here is an integer,
+        #  delta_to_nanoseconds will return it unchanged.
+        result = super(PeriodArray, self)._add_delta_td(other.n)
+        return type(self)(result, freq=self.freq)

    def _add_delta_td(self, other):


can you rename this to _add_timedelta? (or _add_delta_timedelta)

jbrockmendel · 2018-10-28T12:36:37Z

Ping

jreback · 2018-10-28T13:34:03Z

thanks @jbrockmendel

…y_tests * repo_org/master: (52 commits) ENH: Allow rename_axis to specify index and columns arguments (pandas-dev#20046) STY: proposed isort settings [ci skip] [skip ci] [ciskip] [skipci] (pandas-dev#23366) MAINT: Remove extraneous test.parquet file CLN: Follow-up comments to pandas-devgh-23392 (pandas-dev#23401) BUG GH23282 calling min on series of NaT returns NaT (pandas-dev#23289) unpin openpyxl (pandas-dev#23361) REF: collect ops dispatch functions in one place, try to de-duplicate SparseDataFrame methods (pandas-dev#23060) CLN: Remove pandas.tools module (pandas-dev#23376) CLN: Remove some dtype methods from API (pandas-dev#23390) CLN: Cleanup toplevel namespace shims (pandas-dev#23386) DOC: fixup whatsnew note for GH21394 (pandas-dev#23355) Fix import format at pandas/tests/extension directory (pandas-dev#23365) DOC: Remove Series.sortlevel from api.rst (pandas-dev#23395) API: Disallow dtypes w/o frequency when casting (pandas-dev#23392) BUG/TST/REF: Datetimelike Arithmetic Methods (pandas-dev#23215) STYLE: lint add np.nan* funcs to cython_table (pandas-dev#22109) Run Isort on tests/util single PR (pandas-dev#23347) BUG: Fix date_range overflow (pandas-dev#23345) Run Isort on tests/arrays single PR (pandas-dev#23346) ...

jbrockmendel added 8 commits October 16, 2018 19:05

Make arithmetic code dispatch less redundant, fix datetime64 addition…

dc2280f

…, add missing tests

remove unnecessary nat_new

37728ff

Fix interpretation of NaT

15ad0a6

move test

94f1745

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

bf5e2fb

whatsnew, fix dropped timezone

3bdf104

remove comment

982ea30

Add GH references

d046038

jbrockmendel commented Oct 17, 2018

View reviewed changes

jbrockmendel commented Oct 18, 2018

View reviewed changes

pandas/core/arrays/datetimes.py Show resolved Hide resolved

remove unused imports

a0c1a85

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

33d82b3

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

8a7c249

TomAugspurger mentioned this pull request Oct 18, 2018

CI: Quieter doctests #23232

Merged

jorisvandenbossche reviewed Oct 18, 2018

View reviewed changes

pandas/core/arrays/datetimelike.py Show resolved Hide resolved

jbrockmendel added 3 commits October 18, 2018 13:49

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

8e57fd8

dummy commit to force CI

b932121

Fix bug in adding DateOffset to PeriodIndex, Series, Frame

9f3b18d

Dummy commit to force CI

7a8232e

jreback requested changes Oct 26, 2018

View reviewed changes

pandas/core/arrays/datetimes.py Show resolved Hide resolved

pandas/core/arrays/datetimes.py Show resolved Hide resolved

pandas/core/arrays/period.py Show resolved Hide resolved

jbrockmendel added 2 commits October 25, 2018 18:22

fixup super usage

a6eb01c

flesh out TODO comment

acf1f74

jreback approved these changes Oct 26, 2018

View reviewed changes

jbrockmendel added 3 commits October 25, 2018 20:06

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

e89e3ef

move test for moved method

d306277

Fixup name

f6e4073

jorisvandenbossche reviewed Oct 26, 2018

View reviewed changes

jbrockmendel added 4 commits October 27, 2018 07:36

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

9146a72

change NotImplementedError to TypeError

4e4b9ed

Fix add_delta_tdi return type; docstrings

f35b3b6

Merge branch 'arith' of https://github.com/jbrockmendel/pandas into a…

87e07ef

…rith

jorisvandenbossche approved these changes Oct 27, 2018

View reviewed changes

jreback requested changes Oct 28, 2018

View reviewed changes

jbrockmendel added 2 commits October 27, 2018 22:19

Merge branch 'master' of https://github.com/pandas-dev/pandas into arith

343ee30

docstring edit, rename

0466b9c

jreback approved these changes Oct 28, 2018

View reviewed changes

jreback merged commit f0746fc into pandas-dev:master Oct 28, 2018

jbrockmendel deleted the arith branch October 28, 2018 16:16

tm9k1 pushed a commit to tm9k1/pandas that referenced this pull request Nov 19, 2018

BUG/TST/REF: Datetimelike Arithmetic Methods (pandas-dev#23215)

24f5e5b

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG/TST/REF: Datetimelike Arithmetic Methods (pandas-dev#23215)

badd26b

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG/TST/REF: Datetimelike Arithmetic Methods (pandas-dev#23215)

40d6ada

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG/TST/REF: Datetimelike Arithmetic Methods #23215

BUG/TST/REF: Datetimelike Arithmetic Methods #23215

jbrockmendel commented Oct 17, 2018

jbrockmendel Oct 17, 2018

jreback Oct 19, 2018

jbrockmendel Oct 17, 2018

jbrockmendel Oct 17, 2018

jbrockmendel Oct 17, 2018

codecov bot commented Oct 18, 2018 •

edited

Loading

pep8speaks commented Oct 18, 2018

jbrockmendel commented Oct 18, 2018

TomAugspurger commented Oct 18, 2018

jorisvandenbossche commented Oct 18, 2018

jbrockmendel commented Oct 18, 2018

jbrockmendel commented Oct 19, 2018

jreback commented Oct 26, 2018

jorisvandenbossche left a comment

jorisvandenbossche Oct 26, 2018

jbrockmendel Oct 26, 2018

jorisvandenbossche Oct 27, 2018

jbrockmendel Oct 27, 2018

jbrockmendel commented Oct 26, 2018

jreback commented Oct 26, 2018

jreback left a comment

jreback Oct 28, 2018

jreback Oct 28, 2018

jbrockmendel commented Oct 28, 2018

jreback commented Oct 28, 2018

BUG/TST/REF: Datetimelike Arithmetic Methods #23215

BUG/TST/REF: Datetimelike Arithmetic Methods #23215

Conversation

jbrockmendel commented Oct 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 18, 2018 • edited Loading

Codecov Report

pep8speaks commented Oct 18, 2018

jbrockmendel commented Oct 18, 2018

TomAugspurger commented Oct 18, 2018

jorisvandenbossche commented Oct 18, 2018

jbrockmendel commented Oct 18, 2018

jbrockmendel commented Oct 19, 2018

jreback commented Oct 26, 2018

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Oct 26, 2018

jreback commented Oct 26, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Oct 28, 2018

jreback commented Oct 28, 2018

codecov bot commented Oct 18, 2018 •

edited

Loading