-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Selection with datetime64[ns] fails with Pandas 1.1.0 #4283
Labels
Comments
Looks like (to my eye anyway) it stems from: import numpy as np
import pandas as pd
t = np.array(['2017-09-05T12:00:00.000000000', '2017-09-05T15:00:00.000000000'], dtype='datetime64[ns]')
index = pd.DatetimeIndex(t)
index.get_loc(t[0].item()) # Fails with KeyError
index.get_loc(t[0]) # Works Fails on 1.1.0. What I have no idea is whether the |
The OpenDataCube is being affected by this also. |
Closed
shoyer
added a commit
to shoyer/xarray
that referenced
this issue
Jul 31, 2020
Fixes pydata#4283 The underlying issue is that calling `.item()` on a NumPy array with `dtype=datetime64[ns]` returns an _integer_, rather than an `np.datetime64 scalar. This is somewhat baffling but works this way because `.item()` returns native Python types, but `datetime.datetime` doesn't support nanosecond precision. `pandas.Index.get_loc` used to support these integers, but now is more strict. Hence we get errors. We can fix this by using `array[()]` to convert 0d arrays into NumPy scalars instead of calling `array.item()`. I've added a crude regression test. There may well be a better way to test this but I haven't figured it out yet.
shoyer
added a commit
to shoyer/xarray
that referenced
this issue
Jul 31, 2020
Fixes pydata#4283 The underlying issue is that calling `.item()` on a NumPy array with `dtype=datetime64[ns]` returns an _integer_, rather than an `np.datetime64 scalar. This is somewhat baffling but works this way because `.item()` returns native Python types, but `datetime.datetime` doesn't support nanosecond precision. `pandas.Index.get_loc` used to support these integers, but now is more strict. Hence we get errors. We can fix this by using `array[()]` to convert 0d arrays into NumPy scalars instead of calling `array.item()`. I've added a crude regression test. There may well be a better way to test this but I haven't figured it out yet.
2 tasks
This was referenced Aug 4, 2020
leifdenby
added a commit
to leifdenby/lagtraj
that referenced
this issue
Aug 4, 2020
There is an issue with pandas and xarray wrt to datetime indexing (pydata/xarray#4283), until this is fixed in xarray we'll pin the pandas version.
Chilipp
added a commit
to psyplot/psyplot
that referenced
this issue
Aug 27, 2020
shoyer
added a commit
that referenced
this issue
Sep 16, 2020
* Fix indexing with datetime64[ns] with pandas=1.1 Fixes #4283 The underlying issue is that calling `.item()` on a NumPy array with `dtype=datetime64[ns]` returns an _integer_, rather than an `np.datetime64 scalar. This is somewhat baffling but works this way because `.item()` returns native Python types, but `datetime.datetime` doesn't support nanosecond precision. `pandas.Index.get_loc` used to support these integers, but now is more strict. Hence we get errors. We can fix this by using `array[()]` to convert 0d arrays into NumPy scalars instead of calling `array.item()`. I've added a crude regression test. There may well be a better way to test this but I haven't figured it out yet. * lint fix * add a test checking the datetime indexer * use label.item() for non-datetime / timedelta labels * unpin pandas in the docs * ignore the future warning about deprecated arguments to pandas.Grouper * Update xarray/core/indexing.py Co-authored-by: keewis <keewis@users.noreply.github.com> * Add whatsnew note Co-authored-by: Keewis <keewis@posteo.de> Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> Co-authored-by: keewis <keewis@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I ran into this issue with a netCDF file with the following time variable:
but we can reproduce the problem with something as simple as:
this produces:
what's interesting is changing the units of
datetime64
to[s]
works:Environment:
Python 3.8 from conda-forge on macOS 10.15.4
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.8.5 | packaged by conda-forge | (default, Jul 24 2020, 01:06:20)
[Clang 10.0.1 ]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4
xarray: 0.16.0
pandas: 1.1.0
numpy: 1.19.1
scipy: 1.5.2
netCDF4: 1.5.4
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.2.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: 0.9.8.3
iris: None
bottleneck: None
dask: 2.21.0
distributed: 2.21.0
matplotlib: 3.3.0
cartopy: 0.18.0
seaborn: None
numbagg: None
pint: 0.14
setuptools: 49.2.0.post20200712
pip: 20.1.1
conda: None
pytest: 6.0.0
IPython: 7.16.1
sphinx: 2.4.4
The text was updated successfully, but these errors were encountered: