Failure to clone UFS_UTILS/ccpp-physics/physics/rte-rrtmgp on Orion

[MichaelLueken]

Expected behavior

On Orion, using ./manage_externals/checkout_externals should clone all of the externals required to build the SRW App.

Current behavior

On Orion, using ./manage_externals/checkout_externals fails while cloning UFS_UTILS with the following message:

Orion-login-1[57] mlueken$ ./manage_externals/checkout_externals 
Processing externals description file : Externals.cfg
Checking status of externals: ufs-weather-model, aqm-utils, rrfs_utl, arl_nexus, ufs_utils, gsi, upp, 
Checking out externals: aqm-utils, ufs_utils, ERROR:root:Command '[u'git', u'submodule', u'update', u'--init', u'--recursive']' returned non-zero exit status 1
ERROR:root:Failed with output:
    Submodule 'ccpp-physics' (https://github.com/NCAR/ccpp-physics.git) registered for path 'ccpp-physics'
    Cloning into 'ccpp-physics'...
    Submodule path 'ccpp-physics': checked out '12c115e992d3a265eaaa67d72fcbdb3a6f21195f'
    Submodule 'physics/rte-rrtmgp' (https://github.com/earth-system-radiation/rte-rrtmgp) registered for path 'physics/rte-rrtmgp'
    Cloning into 'physics/rte-rrtmgp'...
    fatal: reference is not a tree: 7f01618c92409658bddd3afa9acb004c608f6a0d
    Unable to checkout '7f01618c92409658bddd3afa9acb004c608f6a0d' in submodule path 'ccpp-physics/physics/rte-rrtmgp'
    Failed to recurse into submodule path 'ccpp-physics'

ERROR: In directory
    /work/noaa/epic-ps/mlueken/ufs-srweather-app/sorc/UFS_UTILS
Process did not run successfully; returned status 1:
    git submodule update --init --recursive
See above for output from failed command.

ERROR:root:Failed with output:
    Submodule 'ccpp-physics' (https://github.com/NCAR/ccpp-physics.git) registered for path 'ccpp-physics'
    Cloning into 'ccpp-physics'...
    Submodule path 'ccpp-physics': checked out '12c115e992d3a265eaaa67d72fcbdb3a6f21195f'
    Submodule 'physics/rte-rrtmgp' (https://github.com/earth-system-radiation/rte-rrtmgp) registered for path 'physics/rte-rrtmgp'
    Cloning into 'physics/rte-rrtmgp'...
    fatal: reference is not a tree: 7f01618c92409658bddd3afa9acb004c608f6a0d
    Unable to checkout '7f01618c92409658bddd3afa9acb004c608f6a0d' in submodule path 'ccpp-physics/physics/rte-rrtmgp'
    Failed to recurse into submodule path 'ccpp-physics'

ERROR: In directory
    /work/noaa/epic-ps/mlueken/ufs-srweather-app/sorc/UFS_UTILS
Process did not run successfully; returned status 1:
    git submodule update --init --recursive
See above for output from failed command.


ERROR: Failed with output:
    Submodule 'ccpp-physics' (https://github.com/NCAR/ccpp-physics.git) registered for path 'ccpp-physics'
    Cloning into 'ccpp-physics'...
    Submodule path 'ccpp-physics': checked out '12c115e992d3a265eaaa67d72fcbdb3a6f21195f'
    Submodule 'physics/rte-rrtmgp' (https://github.com/earth-system-radiation/rte-rrtmgp) registered for path 'physics/rte-rrtmgp'
    Cloning into 'physics/rte-rrtmgp'...
    fatal: reference is not a tree: 7f01618c92409658bddd3afa9acb004c608f6a0d
    Unable to checkout '7f01618c92409658bddd3afa9acb004c608f6a0d' in submodule path 'ccpp-physics/physics/rte-rrtmgp'
    Failed to recurse into submodule path 'ccpp-physics'

ERROR: In directory
    /work/noaa/epic-ps/mlueken/ufs-srweather-app/sorc/UFS_UTILS
Process did not run successfully; returned status 1:
    git submodule update --init --recursive
See above for output from failed command.

This behavior is seen in the develop branch, as well as forked branches. The last PR that was merged to develop, PR #618, on March 1, did not have this issue:

+ /work/noaa/epic-ps/role-epic-ps/jenkins/workspace/fs-srweather-app_pipeline_PR-618/manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Checking status of externals: ufs-weather-model, aqm-utils, rrfs_utl, arl_nexus, ufs_utils, gsi, upp, 
Checking out externals: aqm-utils, ufs_utils, upp, arl_nexus, gsi, rrfs_utl, ufs-weather-model,

Machines affected

This behavior is only on Orion. It has been noted that is happening for both the automated Jenkins testing, as well as cloning into my own personal directory on the machine.

Steps To Reproduce

1. Clone the SRW App on Orion (git clone git@github.com:ufs-community/ufs-srweather-app.git)
2. Clone the externals (./manage_externals/checkout_externals)
3. See the above noted failure.

[MichaelLueken]

Attempting to recursively clone the UFS_UTILS repo on Orion directly also ends with the same failure:

Orion-login-1[97] mlueken$ git clone --recursive git@github.com:ufs-community/UFS_UTILS.git
Cloning into 'UFS_UTILS'...
remote: Enumerating objects: 41320, done.
remote: Counting objects: 100% (1766/1766), done.
remote: Compressing objects: 100% (585/585), done.
remote: Total 41320 (delta 1207), reused 1630 (delta 1156), pack-reused 39554
Receiving objects: 100% (41320/41320), 70.63 MiB | 24.30 MiB/s, done.
Resolving deltas: 100% (31257/31257), done.
Submodule 'ccpp-physics' (https://github.com/NCAR/ccpp-physics.git) registered for path 'ccpp-physics'
Cloning into 'ccpp-physics'...
remote: Enumerating objects: 27047, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 27047 (delta 20), reused 28 (delta 12), pack-reused 26995
Receiving objects: 100% (27047/27047), 30.26 MiB | 21.24 MiB/s, done.
Resolving deltas: 100% (21293/21293), done.
Submodule path 'ccpp-physics': checked out '12c115e992d3a265eaaa67d72fcbdb3a6f21195f'
Submodule 'physics/rte-rrtmgp' (https://github.com/earth-system-radiation/rte-rrtmgp) registered for path 'physics/rte-rrtmgp'
Cloning into 'physics/rte-rrtmgp'...
remote: Enumerating objects: 5094, done.
remote: Counting objects: 100% (57/57), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 5094 (delta 22), reused 42 (delta 15), pack-reused 5037
Receiving objects: 100% (5094/5094), 116.18 MiB | 55.28 MiB/s, done.
Resolving deltas: 100% (3424/3424), done.
fatal: reference is not a tree: 7f01618c92409658bddd3afa9acb004c608f6a0d
Unable to checkout '7f01618c92409658bddd3afa9acb004c608f6a0d' in submodule path 'ccpp-physics/physics/rte-rrtmgp'
Failed to recurse into submodule path 'ccpp-physics'

[MichaelLueken]

To get around this issue, a user will need to load git/2.28.0 in their environment before running .manage_externals/checkout_externals on Orion. Of course, this isn't being set in the role-epic-ps account, so the Jenkins testing is failing on that machine.

[MichaelLueken]

module load git/2.28.0 has been added to the .bashrc file for the role-epic-ps account. Looking at the latest Jenkins test submission this morning, Jenkins is now properly loading git/2.28.0, allowing UFS_UTILS to be cloned properly. Closing this issue.