Squashed commit of the following:

commit 7714e79 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Wed Oct 3 10:13:06 2018 -0500 Always return ndarray commit 1921c6f Merge: 01f7366 fea27f0 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Wed Oct 3 09:50:30 2018 -0500 Merge remote-tracking branch 'upstream/master' into combine-exception commit fea27f0 Author: Tom Augspurger <TomAugspurger@users.noreply.github.com> Date: Wed Oct 3 08:49:44 2018 -0500 CI: pin moto to 1.3.4 (pandas-dev#22959) commit 15d32bb Author: jbrockmendel <jbrockmendel@gmail.com> Date: Wed Oct 3 04:32:35 2018 -0700 [CLN] Dispatch (some) Frame ops to Series, avoiding _data.eval (pandas-dev#22019) * avoid casting to object dtype in mixed-type frames * Dispatch to Series ops in _combine_match_columns * comment * docstring * flake8 fixup * dont bother with try_cast_result * revert non-central change * simplify * revert try_cast_results * revert non-central changes * Fixup typo syntaxerror * simplify assertion * use dispatch_to_series in combine_match_columns * Pass unwrapped op where appropriate * catch correct error * whatsnew note * comment * whatsnew section * remove unnecessary tester * doc fixup commit 3e3256b Author: alimcmaster1 <alimcmaster1@gmail.com> Date: Wed Oct 3 12:23:22 2018 +0100 Allow passing a mask to NanOps (pandas-dev#22865) commit e756e99 Author: jbrockmendel <jbrockmendel@gmail.com> Date: Wed Oct 3 02:19:27 2018 -0700 CLN: Use is_period_dtype instead of ABCPeriodIndex checks (pandas-dev#22958) commit 03181f0 Author: Wenhuan <lixx0880@gmail.com> Date: Wed Oct 3 15:28:07 2018 +0800 BUG: fix Series(extension array) + extension array values addition (pandas-dev#22479) commit 04ea51d Author: Joris Van den Bossche <jorisvandenbossche@gmail.com> Date: Wed Oct 3 09:24:36 2018 +0200 CLN: small clean-up of IntervalIndex (pandas-dev#22956) commit b0f9a10 Author: Tony Tao <34781056+tonytao2012@users.noreply.github.com> Date: Tue Oct 2 19:01:08 2018 -0500 DOC GH22893 Fix docstring of groupby in pandas/core/generic.py (pandas-dev#22920) commit 08ecba8 Author: jbrockmendel <jbrockmendel@gmail.com> Date: Tue Oct 2 14:22:53 2018 -0700 BUG: fix DataFrame+DataFrame op with timedelta64 dtype (pandas-dev#22696) commit c44bad2 Author: Pamela Wu <pambot@users.noreply.github.com> Date: Tue Oct 2 17:16:25 2018 -0400 CLN GH22873 Replace base excepts in pandas/core (pandas-dev#22901) commit 8e749a3 Author: Pamela Wu <pambot@users.noreply.github.com> Date: Tue Oct 2 17:14:48 2018 -0400 CLN GH22874 replace bare excepts in pandas/io/pytables.py (pandas-dev#22919) commit 1102a33 Author: Joris Van den Bossche <jorisvandenbossche@gmail.com> Date: Tue Oct 2 22:31:36 2018 +0200 DOC/CLN: clean-up shared_docs in generic.py (pandas-dev#20074) commit 01f7366 Merge: 5372134 9caf048 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 13:50:28 2018 -0500 Merge remote-tracking branch 'upstream/master' into combine-exception commit 9caf048 Author: Tom Augspurger <TomAugspurger@users.noreply.github.com> Date: Tue Oct 2 13:25:22 2018 -0500 CI: change windows vm image (pandas-dev#22948) commit 5372134 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:35:07 2018 -0500 fixed move commit ce1a3c6 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:32:11 2018 -0500 fixed move commit b9c7e4b Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:28:57 2018 -0500 remove old note commit a4a2933 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:24:48 2018 -0500 handle test commit be63feb Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:19:17 2018 -0500 move test commit 0eef0cf Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:18:18 2018 -0500 move back commit 2183f7b Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:17:28 2018 -0500 api commit 85fc5d8 Merge: 9059c0d 1d9f76c Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 11:15:52 2018 -0500 Merge remote-tracking branch 'upstream/master' into combine-exception commit 1d9f76c Author: Joris Van den Bossche <jorisvandenbossche@gmail.com> Date: Tue Oct 2 17:11:11 2018 +0200 CLN: remove Index._to_embed (pandas-dev#22879) * CLN: remove Index._to_embed * pep8 commit 6247da0 Author: Tom Augspurger <TomAugspurger@users.noreply.github.com> Date: Tue Oct 2 08:50:41 2018 -0500 Provide default implementation for `data_repated` (pandas-dev#22935) commit 9059c0d Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 06:33:15 2018 -0500 Note commit 0c53f08 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 06:30:54 2018 -0500 Imports commit ce94bf9 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Tue Oct 2 06:28:16 2018 -0500 Moves commit fdd43c4 Author: Tom Augspurger <tom.w.augspurger@gmail.com> Date: Mon Oct 1 21:26:09 2018 -0500 Closes pandas-dev#22850 commit 5ce06b5 Author: Matthew Roeschke <emailformattr@gmail.com> Date: Mon Oct 1 14:22:20 2018 -0700 BUG: to_datetime preserves name of Index argument in the result (pandas-dev#22918) * BUG: to_datetime preserves name of Index argument in the result * correct test
TomAugspurger · Oct 3, 2018 · c2d57bd · c2d57bd
1 parent ff7c06c
commit c2d57bd
Show file tree

Hide file tree

Showing 10 changed files with 87 additions and 38 deletions.
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
@@ -739,14 +739,22 @@ def _create_method(cls, op, coerce_to_dtype=True):
         ----------
         op : function
             An operator that takes arguments op(a, b)
-        coerce_to_dtype :  bool
+        coerce_to_dtype :  bool, default True
             boolean indicating whether to attempt to convert
-            the result to the underlying ExtensionArray dtype
-            (default True)
+            the result to the underlying ExtensionArray dtype.
+            If it's not possible to create a new ExtensionArray with the
+            values, an ndarray is returned instead.
 
         Returns
         -------
-        A method that can be bound to a method of a class
+        Callable[[Any, Any], Union[ndarray, ExtensionArray]]
+            A method that can be bound to a class. When used, the method
+            receives the two arguments, one of which is the instance of
+            this class, and should return an ExtensionArray or an ndarray.
+
+            Returning an ndarray may be necessary when the result of the
+            `op` cannot be stored in the ExtensionArray. The dtype of the
+            ndarray uses NumPy's normal inference rules.
 
         Example
         -------
@@ -757,7 +765,6 @@ def _create_method(cls, op, coerce_to_dtype=True):
         in the class definition of MyExtensionArray to create the operator
         for addition, that will be based on the operator implementation
         of the underlying elements of the ExtensionArray
-
         """
 
         def _binop(self, other):
@@ -780,12 +787,12 @@ def convert_values(param):
                         a, b = zip(*res)
                         res = (self._from_sequence(a),
                                self._from_sequence(b))
-                    except TypeError:
+                    except Exception:
                         pass
                 else:
                     try:
                         res = self._from_sequence(res)
-                    except TypeError:
+                    except Exception:
                         pass
 
             return res

diff --git a/pandas/core/series.py b/pandas/core/series.py
@@ -2323,10 +2323,14 @@ def combine(self, other, func, fill_value=None):
             pass
         elif is_extension_array_dtype(self.values):
             # The function can return something of any type, so check
-            # if the type is compatible with the calling EA
+            # if the type is compatible with the calling EA.
             try:
                 new_values = self._values._from_sequence(new_values)
             except Exception:
+                # https://github.com/pandas-dev/pandas/issues/22850
+                # pandas has no control over what 3rd-party ExtensionArrays
+                # do in _values_from_sequence. We still want ops to work
+                # though, so we catch any regular Exception.
                 pass
 
         return self._constructor(new_values, index=new_index, name=new_name)

diff --git a/pandas/tests/extension/base/ops.py b/pandas/tests/extension/base/ops.py
@@ -78,10 +78,6 @@ def test_divmod(self, data):
         self._check_divmod_op(s, divmod, 1, exc=TypeError)
         self._check_divmod_op(1, ops.rdivmod, s, exc=TypeError)
 
-    def test_divmod_series_array(self, data):
-        s = pd.Series(data)
-        self._check_divmod_op(s, divmod, data)
-
     def test_add_series_with_extension_array(self, data):
         s = pd.Series(data)
         result = s + data

diff --git a/pandas/tests/extension/decimal/__init__.py b/pandas/tests/extension/decimal/__init__.py
@@ -0,0 +1,4 @@
+from .array import DecimalArray, DecimalDtype, to_decimal, make_data
+
+
+__all__ = ['DecimalArray', 'DecimalDtype', 'to_decimal', 'make_data']
diff --git a/pandas/tests/extension/decimal/array.py b/pandas/tests/extension/decimal/array.py
@@ -1,5 +1,6 @@
 import decimal
 import numbers
+import random
 import sys
 
 import numpy as np
@@ -142,5 +143,9 @@ def to_decimal(values, context=None):
     return DecimalArray([decimal.Decimal(x) for x in values], context=context)
 
 
+def make_data():
+    return [decimal.Decimal(random.random()) for _ in range(100)]
+
+
 DecimalArray._add_arithmetic_ops()
 DecimalArray._add_comparison_ops()
diff --git a/pandas/tests/extension/decimal/test_decimal.py b/pandas/tests/extension/decimal/test_decimal.py
@@ -1,18 +1,14 @@
+import operator
 import decimal
 
-import random
 import numpy as np
 import pandas as pd
 import pandas.util.testing as tm
 import pytest
 
 from pandas.tests.extension import base
 
-from .array import DecimalDtype, DecimalArray, to_decimal
-
-
-def make_data():
-    return [decimal.Decimal(random.random()) for _ in range(100)]
+from .array import DecimalDtype, DecimalArray, make_data
 
 
 @pytest.fixture
@@ -294,3 +290,47 @@ def test_compare_array(self, data, all_compare_operators):
         other = pd.Series(data) * [decimal.Decimal(pow(2.0, i))
                                    for i in alter]
         self._compare_other(s, data, op_name, other)
+
+
+class DecimalArrayWithoutFromSequence(DecimalArray):
+    """Helper class for testing error handling in _from_sequence."""
+    def _from_sequence(cls, scalars, dtype=None, copy=False):
+        raise KeyError("For the test")
+
+
+class DecimalArrayWithoutCoercion(DecimalArrayWithoutFromSequence):
+    @classmethod
+    def _create_arithmetic_method(cls, op):
+        return cls._create_method(op, coerce_to_dtype=False)
+
+
+DecimalArrayWithoutCoercion._add_arithmetic_ops()
+
+
+def test_combine_from_sequence_raises():
+    # https://github.com/pandas-dev/pandas/issues/22850
+    ser = pd.Series(DecimalArrayWithoutFromSequence([
+        decimal.Decimal("1.0"),
+        decimal.Decimal("2.0")
+    ]))
+    result = ser.combine(ser, operator.add)
+
+    # note: object dtype
+    expected = pd.Series([decimal.Decimal("2.0"),
+                          decimal.Decimal("4.0")], dtype="object")
+    tm.assert_series_equal(result, expected)
+
+
+@pytest.mark.parametrize("class_", [DecimalArrayWithoutFromSequence,
+                                    DecimalArrayWithoutCoercion])
+def test_scalar_ops_from_sequence_raises(class_):
+    # op(EA, EA) should return an EA, or an ndarray if it's not possible
+    # to return an EA with the return values.
+    arr = class_([
+        decimal.Decimal("1.0"),
+        decimal.Decimal("2.0")
+    ])
+    result = arr + arr
+    expected = np.array([decimal.Decimal("2.0"), decimal.Decimal("4.0")],
+                        dtype="object")
+    tm.assert_numpy_array_equal(result, expected)
diff --git a/pandas/tests/extension/json/__init__.py b/pandas/tests/extension/json/__init__.py
@@ -0,0 +1,3 @@
+from .array import JSONArray, JSONDtype, make_data
+
+__all__ = ['JSONArray', 'JSONDtype', 'make_data']
diff --git a/pandas/tests/extension/json/array.py b/pandas/tests/extension/json/array.py
@@ -13,6 +13,8 @@
 import collections
 import itertools
 import numbers
+import random
+import string
 import sys
 
 import numpy as np
@@ -179,3 +181,10 @@ def _values_for_argsort(self):
         # cast them to an (N, P) array, instead of an (N,) array of tuples.
         frozen = [()] + [tuple(x.items()) for x in self]
         return np.array(frozen, dtype=object)[1:]
+
+
+def make_data():
+    # TODO: Use a regular dict. See _NDFrameIndexer._setitem_with_indexer
+    return [collections.UserDict([
+        (random.choice(string.ascii_letters), random.randint(0, 100))
+        for _ in range(random.randint(0, 10))]) for _ in range(100)]
diff --git a/pandas/tests/extension/json/test_json.py b/pandas/tests/extension/json/test_json.py
@@ -1,7 +1,5 @@
 import operator
 import collections
-import random
-import string
 
 import pytest
 
@@ -10,18 +8,11 @@
 from pandas.compat import PY2, PY36
 from pandas.tests.extension import base
 
-from .array import JSONArray, JSONDtype
+from .array import JSONArray, JSONDtype, make_data
 
 pytestmark = pytest.mark.skipif(PY2, reason="Py2 doesn't have a UserDict")
 
 
-def make_data():
-    # TODO: Use a regular dict. See _NDFrameIndexer._setitem_with_indexer
-    return [collections.UserDict([
-        (random.choice(string.ascii_letters), random.randint(0, 100))
-        for _ in range(random.randint(0, 10))]) for _ in range(100)]
-
-
 @pytest.fixture
 def dtype():
     return JSONDtype()
@@ -266,11 +257,6 @@ def test_add_series_with_extension_array(self, data):
         with tm.assert_raises_regex(TypeError, "unsupported"):
             ser + data
 
-    def _check_divmod_op(self, s, op, other, exc=NotImplementedError):
-        return super(TestArithmeticOps, self)._check_divmod_op(
-            s, op, other, exc=TypeError
-        )
-
 
 class TestComparisonOps(BaseJSON, base.BaseComparisonOpsTests):
     pass
diff --git a/pandas/tests/extension/test_categorical.py b/pandas/tests/extension/test_categorical.py
@@ -208,11 +208,6 @@ def test_add_series_with_extension_array(self, data):
         with tm.assert_raises_regex(TypeError, "cannot perform"):
             ser + data
 
-    def _check_divmod_op(self, s, op, other, exc=NotImplementedError):
-        return super(TestArithmeticOps, self)._check_divmod_op(
-            s, op, other, exc=TypeError
-        )
-
 
 class TestComparisonOps(base.BaseComparisonOpsTests):