Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make argmin/max work lazy with dask #3244

Merged
merged 8 commits into from
Sep 6, 2019
Merged

Make argmin/max work lazy with dask #3244

merged 8 commits into from
Sep 6, 2019

Conversation

ulijh
Copy link
Contributor

@ulijh ulijh commented Aug 22, 2019 •
edited
Loading

As @shoyer pointed out in #3237, nanargmax/min from numpy or dask should be used when not working on object arrays. Also, nanargmin/max were added to the nputils module so they should be using bottleneck if available.

@ulijh ulijh changed the title Make argmin/max work lazy with dask (#3237). Make argmin/max work lazy with dask Aug 22, 2019
@ulijh ulijh mentioned this pull request Aug 22, 2019
Closed
@max-sixty
Copy link
Collaborator

This looks good @ulijh ! Thanks

Could you add a test? Also feel free to give yourself credit in a whatsnew

@ulijh
Copy link
Contributor Author

ulijh commented Aug 22, 2019

Thanks @max-sixty! Sure, I'll do this tomorrow then.

@ulijh
Copy link
Contributor Author

ulijh commented Aug 23, 2019 •
edited
Loading

Guys, if you could have a look at the tests I modified (a4c3622) to check how many times things get computed. I tried to integrate it with the existing tests. The number of count check could possibly be applied to some other tests as well. May be there is a smarter way of doing this?

shoyer
shoyer reviewed Aug 26, 2019
View reviewed changes
xarray/tests/test_dask.py Outdated Show resolved Hide resolved
xarray/tests/test_dask.py Outdated Show resolved Hide resolved
xarray/tests/test_dask.py Outdated Show resolved Hide resolved
xarray/tests/test_dask.py Show resolved Hide resolved
xarray/core/nanops.py Outdated Show resolved Hide resolved
ulijh and others added 4 commits August 27, 2019 13:48
@ulijh @shoyer
Fix typo
1b4bc3a 
Co-Authored-By: Stephan Hoyer <shoyer@gmail.com>
@ulijh @shoyer
Be more explicit.
abe6655 
Co-Authored-By: Stephan Hoyer <shoyer@gmail.com>
More explicit raise_if_dask_computes
c0d7462
nanargmin/max: only set fill_value when needed
faa7eab
@ulijh
Copy link
Contributor Author

ulijh commented Aug 27, 2019

May be this is a stupid question: What's the best way to resolve this conflict, and get the checks to run? Should I do a merge master? Or rebase (seems somewhat of a pain to get the remote up to date)? Thanks for the advice!

@dcherian
Copy link
Contributor

Merge master is enough. I just did it to kick off the tests.

@shoyer
Copy link
Member

shoyer commented Aug 27, 2019 via email

@ulijh
Copy link
Contributor Author

ulijh commented Aug 27, 2019

Thanks guys!

@dcherian dcherian merged commit 0a046db into pydata:master Sep 6, 2019
dcherian added a commit to dcherian/xarray that referenced this pull request Sep 19, 2019
@dcherian
Merge branch 'master' into map_blocks_2
599b70a 
* master:
  Fix whats-new date :/
  Revert to dev version
  Release v0.13.0
  auto_combine deprecation to 0.14 (pydata#3314)
  Deprecation: groupby, resample default dim. (pydata#3313)
  Raise error if cmap is list of colors (pydata#3310)
  Refactor concat to use merge for non-concatenated variables (pydata#3239)
  Honor `keep_attrs` in DataArray.quantile (pydata#3305)
  Fix DataArray api doc (pydata#3309)
  Accept int value in head, thin and tail (pydata#3298)
  ignore h5py 2.10.0 warnings and fix invalid_netcdf warning test. (pydata#3301)
  Update why-xarray.rst with clearer expression (pydata#3307)
  Compat and encoding deprecation to 0.14 (pydata#3294)
  Remove deprecated concat kwargs. (pydata#3288)
  allow np-array levels and colors in 2D plots (pydata#3295)
  Remove some deprecations (pydata#3292)
  Make argmin/max work lazy with dask (pydata#3244)
  Add head, tail and thin methods (pydata#3278)
  Updater to testing environment name (pydata#3253)
@ulijh ulijh deleted the lazy_argmax_min branch October 25, 2019 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shoyer shoyer shoyer approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

argmax() causes dask to compute
4 participants
@ulijh @max-sixty @dcherian @shoyer