-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
decode_cf() loads chunked arrays #1372
Comments
|
+1 we have come across this recently also in aospy |
A simple test case:
|
Ah, so here's the thing:
You could call You might ask why this separate lazy compute machinery exists. The answer is that dask fails to optimize element-wise operations like See dask/dask#746 for discussion and links to PRs about this. @jcrist had a solution that worked, but it slowed down every dask array operations by 20%, which wasn't a great win. I wonder if this is worth revisiting with a simpler, less general optimization pass that doesn't bother with broadcasting. See the subclasses of
If we could optimize all these operations (and ideally chain them), then we could drop all the lazy loading stuff from xarray in favor of dask, which would be a real win. @mrocklin any thoughts? |
Optimization has come up a bit recently. In other projects like dask-glm I've actually been thinking about just turning it off entirely. The spread of desires here is wide. I think it would be good to make it a bit more customizable so that different applications can more easily customize their optimization suite (see dask/dask#2206) But for this application in particular presumably all of these already tasks |
Yes, that sounds right to me. |
I just hit this issue. I tried to reproduce it with a synthetic dataset, as in @shoyer's example, but I couldn't. I can only reproduce it with data loaded from netcdf4 via open_mfdataset. I downloaded one year of air-temperature data from NARR: I load it this way (preprocessing is necessary to resolve conflict between def preprocess_narr(ds):
del ds.air.attrs['_FillValue']
ds = ds.set_coords(['lon', 'lat', 'Lambert_Conformal', 'time_bnds'])
return ds
ds = xr.open_mfdataset('air.*.nc', preprocess=preprocess_narr, decode_cf=False)
print(ds)
print(ds.chunks)
If I try to decode cf, it returns instantly. ds_decode = xr.decode_cf(ds)
print(ds_decode)
There are no more chunks: This is already a weird situation, since the data is not in memory, but it is not a dask array either. If I try to do anything beyond this with the data, it triggers eager computation. Even if I just call In my case, I could get around this problem if the preprocess function in |
Currently using xarray version 0.9.2 and dask version 0.14.0.
Suppose you load a NetCDF file with the chunks parameter:
The data is loaded as dask arrays, as expected. But if we then manually call
xarray.decode_cf()
, it'll eagerly load the data. Is this the expected behavior, or shoulddecode_cf()
preserve the laziness of the data?The text was updated successfully, but these errors were encountered: