Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drone_map test failing with 4 MPI tasks #766

Open
sbiquard opened this issue Jun 14, 2024 · 2 comments
Open

drone_map test failing with 4 MPI tasks #766

sbiquard opened this issue Jun 14, 2024 · 2 comments

Comments

@sbiquard
Copy link

Hello,

While running the MPI tests on my laptop after compiling from source, I noticed the drone_map test was failing (see extract below).
However the CI tests are passing, the difference being that they only run with 2 MPI tasks whereas I used 4...

test_drone_map (toast.tests.ops_pointing_wcs.PointingWCSTest) ... TOAST INFO: Adding patch "small_patch"
TOAST INFO: Rectangular format
TOAST INFO: Creating '/tmp/tmp6lhvedua'
TOAST INFO: Loading schedule from /tmp/tmp6lhvedua/ground_schedule.txt
TOAST INFO: Loaded 23 scans from /tmp/tmp6lhvedua/ground_schedule.txt totaling 4.25 hours.
[3]error Proc 3: Traceback (most recent call last):
(...)
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/ops/pixels_wcs.py", line 413, in _exec
    lnmin, lnmax, ltmin, ltmax = scan_range_lonlat(
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/timing.py", line 81, in df
    return f(*args, **kwargs)
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/pointing_utils.py", line 163, in scan_range_lonlat
    det_lon, det_lat = center_offset_lonlat(
Proc 3:   File "/home/sbiquard/work/software/toastdev/lib/python3.10/site-packages/toast/pointing_utils.py", line 45, in center_offset_lonlat
    raise ValueError(msg)
Proc 3: ValueError: center_offset dimensions (21300, 2) not compatible with 42600 quaternion values

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
@sbiquard
Copy link
Author

(this is on the toast3 branch)

@tskisner
Copy link
Member

I also discovered this bug while making drone maps for SO. It is fixed in #765

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants