Too many grib files?

[benjamin-cash]

I ran into the following in the ocean_prod tasks and was wondering if anyone had any insight into what might have happened here:

*** FATAL ERROR: too many grib files .. 1st=template.global.1p00.gb2 2nd=hour ***

+ oceanice_nc2grib2.sh[1](ocean): postamble oceanice_nc2grib2.sh 1736898739 8
+ preamble.sh[70](ocean): set +x
End oceanice_nc2grib2.sh at 23:52:19 with error code 8 (time elapsed: 00:00:00)
^[[38;5;196m2025-01-14 17:52:19,872 - ERROR    - oceanice_products: FATAL ERROR: Error occurred during execution of /work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh^[[0m
Traceback (most recent call last):
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/pygfs/task/oceanice_products.py", line 233, in netCDF_to_grib2
    exec_cmd(*arguments)
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/executable.py", line 230, in __call__
    raise ProcessError(f"Command exited with status {proc.returncode}:", long_msg)
wxflow.executable.ProcessError: Command exited with status 8:
'/work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh' 'ocean' '1p00' '1994050100' '192-216' '5:15:25:35:45:55:65:75:85:95:105:115:125:135:145:155:165:175:185:195:205:215:226:241:267:309:374:467:594:757:960:1204:1490:1817:2184:2587:3024:3489:3977:4481'
^[[38;5;196m2025-01-14 17:52:19,872 - ERROR    - oceanice_products: FATAL ERROR: Error occurred during execution of /work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh^[[0m
Traceback (most recent call last):
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/pygfs/task/oceanice_products.py", line 233, in netCDF_to_grib2
    exec_cmd(*arguments)
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/executable.py", line 230, in __call__
    raise ProcessError(f"Command exited with status {proc.returncode}:", long_msg)
wxflow.executable.ProcessError: Command exited with status 8:
'/work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh' 'ocean' '1p00' '1994050100' '192-216' '5:15:25:35:45:55:65:75:85:95:105:115:125:135:145:155:165:175:185:195:205:215:226:241:267:309:374:467:594:757:960:1204:1490:1817:2184:2587:3024:3489:3977:4481'
^[[38;21m2025-01-14 17:52:19,894 - INFO     - root        : BEGIN: wxflow.exceptions.__init__^[[0m
^[[38;5;39m2025-01-14 17:52:19,894 - DEBUG    - root        : ( WorkflowException('/work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh'), '/work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh' )^[[0m
^[[38;5;196m2025-01-14 17:52:19,894 - ERROR    - root        : /work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh^[[0m
^[[38;5;196m2025-01-14 17:52:19,894 - ERROR    - root        : /work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh^[[0m
^[[38;21m2025-01-14 17:52:19,894 - INFO     - root        :   END: wxflow.exceptions.__init__^[[0m
^[[38;5;39m2025-01-14 17:52:19,894 - DEBUG    - root        :  returning: None^[[0m
Traceback (most recent call last):
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/pygfs/task/oceanice_products.py", line 233, in netCDF_to_grib2
    exec_cmd(*arguments)
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/executable.py", line 230, in __call__
    raise ProcessError(f"Command exited with status {proc.returncode}:", long_msg)
wxflow.executable.ProcessError: Command exited with status 8:
'/work2/02441/bcash/frontera/global-workflow-SFS/ush/oceanice_nc2grib2.sh' 'ocean' '1p00' '1994050100' '192-216' '5:15:25:35:45:55:65:75:85:95:105:115:125:135:145:155:165:175:185:195:205:215:226:241:267:309:374:467:594:757:960:1204:1490:1817:2184:2587:3024:3489:3977:4481'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/work2/02441/bcash/frontera/global-workflow-SFS/scripts/exglobal_oceanice_products.py", line 52, in <module>
    main()
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/work2/02441/bcash/frontera/global-workflow-SFS/scripts/exglobal_oceanice_products.py", line 42, in main
    oceanice.execute(oceanice_dict, grid)
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/pygfs/task/oceanice_products.py", line 175, in execute
    OceanIceProducts.netCDF_to_grib2(config, product_grid)
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/work2/02441/bcash/frontera/global-workflow-SFS/ush/python/pygfs/task/oceanice_products.py", line 239, in netCDF_to_grib2
    raise WorkflowException(f"{exec_cmd}")
wxflow.exceptions.WorkflowException
+ JGLOBAL_OCEANICE_PRODUCTS[1]: postamble JGLOBAL_OCEANICE_PRODUCTS 1736898681 1
+ preamble.sh[70]: set +x
End JGLOBAL_OCEANICE_PRODUCTS at 23:52:20 with error code 1 (time elapsed: 00:00:59)
postamble oceanice_products.sh 1736898680 $?
+ oceanice_products.sh[1]: postamble oceanice_products.sh 1736898680 1

[DavidHuber-NOAA]

This is a wgrib2 error and I believe this may be the result of an environmental variable being unset/set oddly. Could you share the log file?

[DavidHuber-NOAA]

@mark-a-potts Any ideas on how singularity could better handle complex commands like this?

[benjamin-cash]

Hi @DavidHuber-NOAA - thanks! This looks like it is a bash problem of some kind. In this case my wgrib2 is a wrapper script that calls wgrib2 from within the container. The lines where singularity (apptainer) is running the command are:

cmd=$(basename "$0")
arg="$@"
echo running: singularity exec "${img}" $cmd $arg
apptainer exec   -B /mnt "${img}" $cmd $arg

So that output from that 'running' line is actually a diagnostic, and for some reason the single quotes are getting stripped out of the passed argument. @mark-a-potts any thoughts about where those could be getting lost? From what I understand that "$@" construction should be preserving those quotes, but they are getting lost somewhere.

[benjamin-cash]

I added a few more diagnostics, and the quotes are already stripped out if I just echo "$@".

[benjamin-cash]

@DavidHuber-NOAA @mark-a-potts - I think I've got it. Changed arg="$@" to ("$@") and passed it as an array, which preserved the quotes, ran to completion!

[DavidHuber-NOAA]

Excellent!