Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The atmos_products job runs wgrib2 with too many threads #2206

Closed
DavidHuber-NOAA opened this issue Jan 8, 2024 · 1 comment
Closed

The atmos_products job runs wgrib2 with too many threads #2206

DavidHuber-NOAA opened this issue Jan 8, 2024 · 1 comment
Assignees
@DavidHuber-NOAA
Labels
bug Something isn't working

Comments

@DavidHuber-NOAA
Copy link
Contributor

DavidHuber-NOAA commented Jan 8, 2024 •
edited
Loading

What is wrong?

The atmos_products job is running wgrib2 without specifying -ncpu, which is causing wgrib2 to over-thread when called during MPMD jobs, resulting in OpenMP errors on S4.

What should have happened?

OMP_NUM_THREADS should be specified using the env/<MACHINE>.env file specified thread value NTHREADS_UPP.
-ncpu 1 should be specified inside of the ex script for atmos_products when running on S4 (and perhaps other machines as well).

What machines are impacted?

All or N/A

Steps to reproduce

Run an atmos_products job on S4 and check for OpenMP errors.

Additional information

Reported by @souopgui on S4.

Do you have a proposed solution?

Add export OMP_NUM_THREADS=${OMP_NUM_THREADS:-${NTHREADS_UPP:-1}} to JGLOBAL_ATMOS_UPP.
Add -ncpu 1 to wgrib2 calls inside of MPMD jobs.
@DavidHuber-NOAA DavidHuber-NOAA added bug Something isn't working triage Issues that are triage labels Jan 8, 2024
@DavidHuber-NOAA DavidHuber-NOAA self-assigned this Jan 8, 2024
@DavidHuber-NOAA DavidHuber-NOAA removed the triage Issues that are triage label Jan 8, 2024
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this issue Jan 8, 2024
@DavidHuber-NOAA
Set OMP_NUM_THREADS for upp jobs. NOAA-EMC#2206
4947088
@DavidHuber-NOAA DavidHuber-NOAA changed the title The UPP job does not declare OMP_NUM_THREADS The atmos_products job runs wgrib2 with too many threads Jan 9, 2024
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this issue Jan 9, 2024
@DavidHuber-NOAA
Set -ncpu 1 for MPMD wgrib2 calls NOAA-EMC#2206
84a48af
@souopgui souopgui mentioned this issue Jan 9, 2024
Closed
@DavidHuber-NOAA
Copy link
Contributor Author

Closing as a duplicate of #2211.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DavidHuber-NOAA DavidHuber-NOAA

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DavidHuber-NOAA