Add optional caching to AreaDefinition.get_area_slices

[djhoese]

While profiling some Satpy computations (ABI full disk -> nearest neighbor resampling) I noticed that a decent amount of time at the beginning of processing was spent outside of dask computations and was using a single core. After some print-statement debugging I discovered it was Satpy's reduce_data functionality and the AreaDefinition.get_area_slices that was taking the most time. The majority of that time is spent in the polygon intersection operation to see if the two areas being used intersect and where.

This PR adds a decorator and a couple configuration settings for caching the results of AreaDefinition.get_area_slices to on-disk JSON files. For my testing case I was seeing about ~10-12 seconds being used for get_area_slices per area definition pair. I was using 2 resolutions of ABI data and one target area so that was ~22 seconds. With this caching enabled that time basically disappears.

This PR is only a proof of concept at this point and I will continue to improve it. I just wanted to get the initial commits up on github for others to see.

• Closes #xxxx
• Tests added
• Tests passed
• Passes git diff origin/main **/*py | flake8 --diff
• Fully documented

[codecov]

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (6a8afc0) 94.11% compared to head (b0a2579) 94.13%.

Files	Patch %	Lines
pyresample/future/geometry/_subset.py	90.00%	8 Missing ⚠️
pyresample/_caching.py	96.10%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #553      +/-   ##
==========================================
+ Coverage   94.11%   94.13%   +0.02%     
==========================================
  Files          82       84       +2     
  Lines       13078    13188     +110     
==========================================
+ Hits        12308    12415     +107     
- Misses        770      773       +3     

Flag	Coverage Δ
unittests	94.13% <94.71%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coveralls]

coverage: 93.72% (+0.03%) from 93.69%
when pulling b0a2579 on djhoese:cache-area-slices
into 6a8afc0 on pytroll:main.

[mraspaud]

Nice with the refactoring! Just a couple of comments, but looks good otherwise.

[mraspaud]

I think we need to add a sentence or two on what kind of improvement we can expect on performance and in which situation

[djhoese]

Oh your comment is reminding me: Do you have a go-to benchmark for the gradient search that you can tell me or run yourself with this PR and with this caching enabled? I think the gradient search is the only other part of pyresample that uses the area slices directly and I don't want to make it unnecessarily slow. That said, if it caches the slices for an area -> area resampling (the only thing gradient search supports right now) then it'd probably make all future operations fast. Especially since iirc satpy doesn't do reduce_data for gradient search.

[djhoese]

Ok I added more information, but I got a little wordy so let me know what you think.

[mraspaud]

@pnuu is using gradient search alot, maybe he can help here?

[mraspaud]

Looks good with the explanation.

[pnuu]

I don't have my usual test script at hand, but can test this on monday. If I remember....

[pnuu]

Timings using Satpy main and this PR. Using gradient_search to load, resample and save 10 composites (some Day/Night, some normal) for FCI L1c to 3868 x 3918 EPSG:3035 area.

reduce_data=True:

• no caching 73.1 s (Dask graph A)
• caching, first run: 72.3 s
• caching, 2nd run: 57.7 s (B)

As a comparison, with reduce_data=False it takes ~59.7 s (C) to run the same script.

Dask graphs:

A

B

C

[mraspaud]

Thanks @pnuu , this looks good!
So I'm merging this.

[djhoese]

Thanks for testing @pnuu!