Error: out of memory

[Qingfu-Liu]

Description

To Reproduce:

Additional context

Output

[Qingfu-Liu]

Not sure why the text is not showed up in the description section. If you open the "edit" button, you will see the text. I also copy the text here :

I have an error message "Out Of Memory" from the workflow I checked out on May 18/19, 2023.
The workflow I checked out on Hera from: https://github.com/NOAA-EMC/global-workflow
and created the workflow directory and run the suite FV3_GFS_v17_p8
I am able to run through 240 hours forecast from the version I checked out on April 28:
/scratch1/NCEPDEV/global/Qingfu.Liu/git/gfsv17_20230428/sorc/ufs_model.fd]git branch -vva

• (HEAD detached at 0c8e74c) 0c8e74c Hpc stack update (Issue 1465) (Hpc stack update (Issue 1465) #1596)
  develop 3cc95d6 [origin/develop] Add CLM Lake Model and update Flake (Add CLM Lake Model and update Flake #1509)
  remotes/origin/HEAD -> origin/develop
  remotes/origin/develop 3cc95d6 Add CLM Lake Model and update Flake (Add CLM Lake Model and update Flake #1509)
  However, I can only run 12 steps from the version I checked out on May 18/19 and have the "out of memory" problem. The version I checked out on May 18:
  /scratch1/NCEPDEV/global/Qingfu.Liu/git/gfsv17_20230518_org/sorc/ufs_model.fd[1454]git branch -vva
• (HEAD detached at 2247060) 2247060 Update in-line post for RRFS and GFS (Update in-line post for RRFS and GFS #1690)
  develop 2cf2227 [origin/develop] Update build for gfsv17 test to use -D32BIT=ON and set fewer tests for S4 (Update build for gfsv17 test to use -D32BIT=ON and set fewer tests for S4 #1728)
  remotes/origin/HEAD -> origin/develop
  remotes/origin/develop 2cf2227 Update build for gfsv17 test to use -D32BIT=ON and set fewer tests for S4 (Update build for gfsv17 test to use -D32BIT=ON and set fewer tests for S4 #1728)

I am running suite FV3_GFS_v17_p8 on Hera using two different versions. Not sure which commit cause the problem.

Screenshots
: PASS: fcstRUN phase 1, n_atmsteps = 12 time is 1.417832
0: PASS: fcstRUN phase 2, n_atmsteps = 12 time is 0.321013
4608: PROCESS SURFCE done
4: ncells= 5
4: nlives= 12
4: nthresh= 18.0000000000000
4680: slurmstepd: error: Detected 1 oom-kill event(s) in StepId=45100467.0. Some of your processes may have been killed by the cgroup out-of-memory handler.
4608: PROCESS CLDRAD done
srun: error: h33m34: task 4688: Out Of Memory
srun: launch/slurm: _step_signal: Terminating StepId=45100467.0
0: slurmstepd: error: *** STEP 45100467.0 ON h1c01 CANCELLED AT 2023-05-19T16:00:47 ***
4964: forrtl: error (78): process killed (SIGTERM)

output logs
Hera: /scratch1/NCEPDEV/stmp2/Qingfu.Liu/ROTDIRS/gfsv17p8c_p9/logs/2020080100/gfsfcst.log_save

The code is here: /scratch1/NCEPDEV/global/Qingfu.Liu/git/gfsv17_20230518_org

[DeniseWorthen]

@Qingfu-Liu Since you're running into this problem from the workflow, I'd suggest the issue be created there.

[JessicaMeixner-NOAA]

@Qingfu-Liu I'd suggest adding more tasks the write component. That's helped me with memory problems in the workflow in the past. You can also see this issue: NOAA-EMC/global-workflow#1572 which might be of relevance.

[Qingfu-Liu]

@DeniseWorthen @JessicaMeixner-NOAA Thank you very much for the help. I will try more tests by changing the tasks in the workflow

[Qingfu-Liu]

@jessica Meixner - NOAA Federal ***@***.***> Thank you very much. The "out of memory" problem is fixed after increasing the computer node from 140 to 144.

…

On Fri, May 19, 2023 at 1:00 PM Jessica Meixner ***@***.***> wrote: @Qingfu-Liu <https://github.com/Qingfu-Liu> I'd suggest adding more tasks the write component. That's helped me with memory problems in the workflow in the past. You can also see this issue: NOAA-EMC/global-workflow#1572 <NOAA-EMC/global-workflow#1572> which might be of relevance. — Reply to this email directly, view it on GitHub <#1759 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGTS6UVFIS2JBB2UJ2QW7JLXG6RKXANCNFSM6AAAAAAYH7ZYXQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>