TYP: core/dtypes/cast.py #37024

arw2019 · 2020-10-10T06:32:00Z

jbrockmendel · 2020-10-10T14:58:45Z

pandas/core/dtypes/cast.py

@@ -181,7 +181,7 @@ def maybe_downcast_to_dtype(result, dtype):
    return result


-def maybe_downcast_numeric(result, dtype, do_round: bool = False):
+def maybe_downcast_numeric(result, dtype: Dtype, do_round: bool = False):


Switched to that

jbrockmendel · 2020-10-10T14:59:24Z

pandas/core/dtypes/cast.py

-def maybe_cast_to_extension_array(cls: Type["ExtensionArray"], obj, dtype=None):
+def maybe_cast_to_extension_array(
+    cls: Type["ExtensionArray"], obj, dtype: Dtype = None
+):


docstring says ExtensionDtype, optional, so Optional[ExtensionDtype]?

jbrockmendel · 2020-10-10T15:00:07Z

pandas/core/dtypes/cast.py

-def infer_dtype_from_array(arr, pandas_dtype: bool = False) -> Tuple[DtypeObj, Any]:
+def infer_dtype_from_array(
+    arr, pandas_dtype: bool = False
+) -> Tuple[DtypeObj, AnyArrayLike]:


i think ArrayLike, not AnyArrayLike

jbrockmendel · 2020-10-10T15:00:57Z

pandas/core/dtypes/cast.py

-def maybe_convert_objects(values: np.ndarray, convert_numeric: bool = True):
+def maybe_convert_objects(
+    values: np.ndarray, convert_numeric: bool = True
+) -> Union[np.ndarray, ABCDatetimeIndex]:


use "DatetimeIndex" instead of ABCDatetimeIndex

jbrockmendel · 2020-10-10T15:01:13Z

pandas/core/dtypes/cast.py

@@ -1183,7 +1205,7 @@ def convert_dtypes(
    return inferred_dtype


-def maybe_castable(arr) -> bool:
+def maybe_castable(arr: np.ndarray) -> bool:


we never get EAs here?

Atm the only use case is here:

pandas/pandas/core/construction.py

Lines 538 to 557 in 126f3cc

def _try_cast(arr, dtype: Optional[DtypeObj], copy: bool, raise_cast_failure: bool):

"""

Convert input to numpy ndarray and optionally cast to a given dtype.

Parameters

----------

arr : ndarray, scalar, list, tuple, iterator (catchall)

Excludes: ExtensionArray, Series, Index.

dtype : np.dtype, ExtensionDtype or None

copy : bool

If False, don't copy the data if not needed.

raise_cast_failure : bool

If True, and if a dtype is specified, raise errors during casting.

Otherwise an object array is returned.

"""

# perf shortcut as this is the most common case

if isinstance(arr, np.ndarray):

if maybe_castable(arr) and not copy and dtype is None:

return arr

which according to docstring excludes ExtensionArray

can you add an assert in the body itself to lock this down. (whether we want it or not is different, but guarantee is good for now)

pandas/core/dtypes/cast.py

jreback · 2020-10-11T16:39:49Z

pandas/core/dtypes/cast.py

@@ -904,7 +936,7 @@ def maybe_upcast(values, fill_value=np.nan, dtype=None, copy: bool = False):
    return values, fill_value


-def invalidate_string_dtypes(dtype_set):
+def invalidate_string_dtypes(dtype_set: Set):


Set[DtypeObj] I think?

jreback · 2020-10-11T16:40:37Z

pandas/core/dtypes/cast.py

@@ -1183,7 +1205,7 @@ def convert_dtypes(
    return inferred_dtype


-def maybe_castable(arr) -> bool:
+def maybe_castable(arr: np.ndarray) -> bool:


can you add an assert in the body itself to lock this down. (whether we want it or not is different, but guarantee is good for now)

jreback · 2020-10-11T16:41:01Z

cc @simonjayhawkins

jreback · 2020-10-12T15:34:33Z

@simonjayhawkins i fyou can have a quick look and merge if good

simonjayhawkins · 2020-10-12T19:02:01Z

pandas/core/dtypes/cast.py

-def maybe_cast_to_extension_array(cls: Type["ExtensionArray"], obj, dtype=None):
+def maybe_cast_to_extension_array(
+    cls: Type["ExtensionArray"], obj: ArrayLike, dtype: Optional[ExtensionDtype] = None
+) -> ArrayLike:


ArrayLike is a TypeVar. so the return type is typed to be the same as obj.

I think mypy is accepting this since np.ndarray resolves to Any

We probably need an ArrayLikeUnion, but this is perhaps out of scope here as the discussion has been started elsewhere. xref #36100

Shall I remove the annotation and add a TODO?

As an aside FWIW I like the ArrayLike/ArrayLikeT idea in https://github.com/pandas-dev/pandas/pull/36100/files#r483753222

you can leave as is for now. we have #36092 to track the issues once np.ndarray no longer resolves to Any.

simonjayhawkins · 2020-10-12T19:06:36Z

pandas/core/dtypes/cast.py

@@ -468,7 +487,7 @@ def maybe_casted_values(index, codes=None):

    # if we have the codes, extract the values with a mask
    if codes is not None:
-        mask = codes == -1
+        mask: np.ndarray = codes == -1


why was this added? mypy is not reporting any errors for me with this removed.

generally, try to avoid adding variable type annotations. should be inferred from expression.

mistake, will revert

I had tried typing index and codes but there were a lot of errors - this was part of an attempted fix.

simonjayhawkins · 2020-10-12T19:12:03Z

pandas/core/dtypes/cast.py

@@ -1184,7 +1220,7 @@ def soft_convert_objects(


 def convert_dtypes(
-    input_array,
+    input_array: AnyArrayLike,


Can you update the docstring to match?

simonjayhawkins · 2020-10-12T19:22:10Z

pandas/core/dtypes/cast.py

@@ -1566,7 +1606,9 @@ def find_common_type(types: List[DtypeObj]) -> DtypeObj:
    return np.find_common_type(types, [])


-def cast_scalar_to_array(shape, value, dtype: Optional[DtypeObj] = None) -> np.ndarray:
+def cast_scalar_to_array(
+    shape: Tuple, value: Scalar, dtype: Optional[DtypeObj] = None


we just pass shape onto np.empty, https://numpy.org/doc/1.19/reference/generated/numpy.empty.html

so could be int or tuple of int

I recall a discussion about adding an alias to pandas._typing or

Suggested change

shape: Tuple, value: Scalar, dtype: Optional[DtypeObj] = None

shape: Union[int, Tuple[int, ...]], value: Scalar, dtype: Optional[DtypeObj] = None

and the docstring would need to be updated too.

Pushed the suggestion.

Adding an alias to _typing sounds like a good idea. Do we already have a PR on that or shall I submit one?

Do we already have a PR on that or shall I submit one?

I don't think so, go ahead.

simonjayhawkins · 2020-10-12T19:40:33Z

pandas/core/dtypes/cast.py

@@ -1698,7 +1742,7 @@ def construct_1d_ndarray_preserving_na(
    return subarr


-def maybe_cast_to_integer_array(arr, dtype, copy: bool = False):
+def maybe_cast_to_integer_array(arr, dtype: Union[str, np.dtype], copy: bool = False):


is this limited to numpy dtypes? if not should use Dtype from typing.

it's not. Switched to Dtype here

simonjayhawkins · 2020-10-12T19:42:05Z

Thanks @arw2019 lgtm one more comment #37024 (comment) ping when green

jreback · 2020-10-14T12:35:20Z

thanks @arw2019

arw2019 marked this pull request as draft October 10, 2020 06:32

arw2019 force-pushed the TYP-core_dtypes_cast branch from 8706ee1 to 88d793e Compare October 10, 2020 06:57

arw2019 mentioned this pull request Oct 10, 2020

CLN: move maybe_casted_values from pandas/core/frame.py to pandas/core/dtype/cast.py #36985

Merged

5 tasks

jbrockmendel reviewed Oct 10, 2020

View reviewed changes

jreback added Dtype Conversions Unexpected or buggy dtype conversions Typing type annotations, mypy/pyright type checking labels Oct 10, 2020

arw2019 force-pushed the TYP-core_dtypes_cast branch from 88d793e to 40cf38f Compare October 10, 2020 16:17

arw2019 commented Oct 10, 2020

View reviewed changes

pandas/core/dtypes/cast.py Outdated Show resolved Hide resolved

arw2019 force-pushed the TYP-core_dtypes_cast branch 3 times, most recently from 136d7db to 5ae2c02 Compare October 11, 2020 03:57

arw2019 marked this pull request as ready for review October 11, 2020 04:01

arw2019 force-pushed the TYP-core_dtypes_cast branch from aa53270 to 9872ea2 Compare October 11, 2020 16:32

jreback requested changes Oct 11, 2020

View reviewed changes

arw2019 force-pushed the TYP-core_dtypes_cast branch from cb90e32 to 07d3929 Compare October 11, 2020 20:47

jreback added this to the 1.2 milestone Oct 12, 2020

jreback approved these changes Oct 12, 2020

View reviewed changes

simonjayhawkins reviewed Oct 12, 2020

View reviewed changes

arw2019 force-pushed the TYP-core_dtypes_cast branch from 9d6ac87 to a5d001a Compare October 12, 2020 19:24

simonjayhawkins reviewed Oct 12, 2020

View reviewed changes

arw2019 force-pushed the TYP-core_dtypes_cast branch 2 times, most recently from 2e85988 to a3ae1be Compare October 12, 2020 23:00

arw2019 added 13 commits October 13, 2020 16:40

add type hints

b4e582f

feedback

2cc7e97

more type hints

8da843d

more type hints

fc757c0

remove addressed TODO

8999b35

add type hints to maybe_casted_values

d054868

remove type hint from codes pending followon PR

92d94b0

feedback: invalidate_string_dtype

a5e9800

feedback: add assert to maybe_castable

afd887b

feedback: update docstring [convert_dtypes]

7f4e93e

feedback: revert variable type annotation in maybe_casted_values

3be4aba

revert annotations and add TODO for maybe_cast_to_EA

25ce7ed

use Dtype in maybe_cast_to_integer_array

3aa7e4d

arw2019 force-pushed the TYP-core_dtypes_cast branch from a3ae1be to 3aa7e4d Compare October 13, 2020 16:41

jreback merged commit 85260bf into pandas-dev:master Oct 14, 2020

charlesdong1991 mentioned this pull request Oct 14, 2020

VIS: Accept xlabel and ylabel for scatter and hexbin plots #37102

Merged

5 tasks

simonjayhawkins mentioned this pull request Oct 14, 2020

CLN: core/dtypes/cast.py::maybe_downcast_to_dtype #37050

Merged

5 tasks

This was referenced Oct 15, 2020

[REDO] CLN: core/dtypes/cast.py::maybe_downcast_to_dtype #37126

Merged

TYP: add Shape alias to pandas._typing #37128

Merged

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Oct 26, 2020

TYP: core/dtypes/cast.py (pandas-dev#37024)

8a99d86

kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020

TYP: core/dtypes/cast.py (pandas-dev#37024)

e71dfb8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TYP: core/dtypes/cast.py #37024

TYP: core/dtypes/cast.py #37024

arw2019 commented Oct 10, 2020

jbrockmendel Oct 10, 2020

arw2019 Oct 10, 2020

jbrockmendel Oct 10, 2020

arw2019 Oct 10, 2020

jbrockmendel Oct 10, 2020

arw2019 Oct 10, 2020

jbrockmendel Oct 10, 2020

arw2019 Oct 10, 2020

jbrockmendel Oct 10, 2020

arw2019 Oct 10, 2020

jreback Oct 11, 2020

arw2019 Oct 11, 2020

jreback Oct 11, 2020

arw2019 Oct 11, 2020

jreback Oct 11, 2020

jreback commented Oct 11, 2020

jreback commented Oct 12, 2020

simonjayhawkins Oct 12, 2020

arw2019 Oct 12, 2020

simonjayhawkins Oct 12, 2020

simonjayhawkins Oct 12, 2020 •

edited

Loading

arw2019 Oct 12, 2020

simonjayhawkins Oct 12, 2020

arw2019 Oct 12, 2020

simonjayhawkins Oct 12, 2020

arw2019 Oct 12, 2020

simonjayhawkins Oct 12, 2020

simonjayhawkins Oct 12, 2020

arw2019 Oct 12, 2020

simonjayhawkins commented Oct 12, 2020

jreback commented Oct 14, 2020

	def _try_cast(arr, dtype: Optional[DtypeObj], copy: bool, raise_cast_failure: bool):
	"""
	Convert input to numpy ndarray and optionally cast to a given dtype.

	Parameters
	----------
	arr : ndarray, scalar, list, tuple, iterator (catchall)
	Excludes: ExtensionArray, Series, Index.
	dtype : np.dtype, ExtensionDtype or None
	copy : bool
	If False, don't copy the data if not needed.
	raise_cast_failure : bool
	If True, and if a dtype is specified, raise errors during casting.
	Otherwise an object array is returned.
	"""
	# perf shortcut as this is the most common case
	if isinstance(arr, np.ndarray):
	if maybe_castable(arr) and not copy and dtype is None:
	return arr

	shape: Tuple, value: Scalar, dtype: Optional[DtypeObj] = None
	shape: Union[int, Tuple[int, ...]], value: Scalar, dtype: Optional[DtypeObj] = None

TYP: core/dtypes/cast.py #37024

TYP: core/dtypes/cast.py #37024

Conversation

arw2019 commented Oct 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Oct 11, 2020

jreback commented Oct 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins commented Oct 12, 2020

jreback commented Oct 14, 2020

simonjayhawkins Oct 12, 2020 •

edited

Loading