Skip to content

Commit

Permalink
Remove hardcoded single-threaded dask scheduler for dataset combine (#…
Browse files Browse the repository at this point in the history
…614)

* remove hardcoded single-threaded scheduler for dataset combine

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
dougiesquire and pre-commit-ci[bot] committed Jun 30, 2023
1 parent e6b49a8 commit 02571ae
Showing 1 changed file with 5 additions and 10 deletions.
15 changes: 5 additions & 10 deletions intake_esm/source.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,16 +252,11 @@ def _open_dataset(self):
for agg in self.aggregations
),
)
with dask.config.set(
{'scheduler': 'single-threaded', 'array.slicing.split_large_chunks': True}
): # Use single-threaded scheduler
datasets = [
ds.set_coords(set(ds.variables) - set(ds.attrs[OPTIONS['vars_key']]))
for ds in datasets
]
self._ds = xr.combine_by_coords(
datasets, **self.xarray_combine_by_coords_kwargs
)
datasets = [
ds.set_coords(set(ds.variables) - set(ds.attrs[OPTIONS['vars_key']]))
for ds in datasets
]
self._ds = xr.combine_by_coords(datasets, **self.xarray_combine_by_coords_kwargs)

self._ds.attrs[OPTIONS['dataset_key']] = self.key

Expand Down

0 comments on commit 02571ae

Please sign in to comment.