-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intermittent failures on 32bit architectures #481
Comments
There seems to be also another issue linked to a 32 bit architecture (mipsel this time).
|
So the legacy dask EWA one we've noticed on 64-bit Windows platforms (at least) in our CI. It is on my TODO list to figure that out. For the other failure...very weird. A single pixel is too different and only by:
Any idea how often it fails versus passes? My guess is it would have to be something with the order that dask computes the individual chunks. I guess the easiest thing would be to increase the threshold for the test, but maybe I spend some time figuring out the legacy dask ewa failure first which at least happens in our CI occasionally. |
well, i would say quite often:
|
@avalentino I just merged #482 into main which should fix the intermittent |
@djhoese I run the I have also tried to set the dask scheduler to "sync" for the diff --git a/pyresample/test/test_dask_ewa.py b/pyresample/test/test_dask_ewa.py
index cfb96a7..0ac8286 100644
--- a/pyresample/test/test_dask_ewa.py
+++ b/pyresample/test/test_dask_ewa.py
@@ -333,16 +333,16 @@ class TestDaskEWAResampler:
input_dims=input_dims,
)
swath_data.data = swath_data.data.astype(np.float32)
+ with dask.config.set(scheduler='sync'):
+ resampler = DaskEWAResampler(source_swath, target_area)
+ new_data = resampler.resample(swath_data, rows_per_scan=10,
+ maximum_weight_mode=maximum_weight_mode)
+ new_arr = new_data.compute()
- resampler = DaskEWAResampler(source_swath, target_area)
- new_data = resampler.resample(swath_data, rows_per_scan=10,
- maximum_weight_mode=maximum_weight_mode)
- new_arr = new_data.compute()
-
- legacy_resampler = LegacyDaskEWAResampler(source_swath, target_area)
- legacy_data = legacy_resampler.resample(swath_data, rows_per_scan=10,
- maximum_weight_mode=maximum_weight_mode)
- legacy_arr = legacy_data.compute()
+ legacy_resampler = LegacyDaskEWAResampler(source_swath, target_area)
+ legacy_data = legacy_resampler.resample(swath_data, rows_per_scan=10,
+ maximum_weight_mode=maximum_weight_mode)
+ legacy_arr = legacy_data.compute()
np.testing.assert_allclose(new_arr, legacy_arr) |
Yeah that's basically how I would have done it. I mean this change in result by such a small amount could be anything from numpy to dask to the specific process the code is being run on. Especially since it is one pixel I'm not sure it is worth trying to narrow it down rather than changing the threshold/tolerance on the comparison. |
OK for me to change the tolerance. |
I think this was solved in #482, please tell if it isn't. Closing for now. |
Code Sample, a minimal, complete, and verifiable piece of code
Problem description
On 32bit architectures I'm experiencing intermittent failures of the test
pytesample.test.test_dask_ewa.TestDaskEWAResampler.test_compare_to_legacy
Expected Output
All unittest pass.
Actual Result, Traceback if applicable
Versions of Python, package at hand and relevant dependencies
Python 3.10
PyResample 1.26.0
The text was updated successfully, but these errors were encountered: