Optimize AreaDefinition.get_proj_coords when requesting dask arrays #401
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #401 +/- ##
==========================================
+ Coverage 93.81% 93.83% +0.02%
==========================================
Files 65 65
Lines 11076 11118 +42
==========================================
+ Hits 10391 10433 +42
Misses 685 685
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
pnuu
approved these changes
Dec 3, 2021
mraspaud
reviewed
Dec 3, 2021
pyresample/geometry.py
Outdated
Comment on lines
1133
to
1134
| x = np.arange(start_x_idx, end_x_idx, dtype=dtype) * pixel_size_x + pixel_upper_left_x | ||
| y = np.arange(start_y_idx, end_y_idx, dtype=dtype) * -pixel_size_y + pixel_upper_left_y |
There was a problem hiding this comment.
Could this be refactored so that it doesn't duplicate the code in _get_proj_vectors?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is part of my work on the Satpy issue pytroll/satpy#1902. As described in some of the docstrings in this PR, these changes make it so that
get_proj_coords, the method that returns a 2D X and a 2D Y coordinate array for each pixel of an AreaDefinition, generates the data usings amap_blockscall. Previously this data was generated by getting the 1D vectors vianp.arangeand some other math, then passing them tonp.meshgridto turn them into 2D vectors. Those series of steps produce an odd series of tasks for dask when the coordinates are used for future work like converting to lon/lat arrays and then using those lon/lats to generate solar zenith angles (for example). The result is that dask has trouble scheduling the future tasks because it sees the complicated set of tasks and series of linked tasks required to turn two 1D arrays into two 2D arrays.This PR switches to using
map_blocksto generate the 2D coordinate arrays from the scalar properties of the AreaDefinition which essentially shows up as 1 task in the dask graph. As shown in the satpy issue referenced above this reduces memory usage of processing (specifically AHI) by a lot.git diff origin/main **/*py | flake8 --diff