Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support coupling on AWS #2859

Merged
merged 78 commits into from
Aug 28, 2024
Merged

Support coupling on AWS #2859

merged 78 commits into from
Aug 28, 2024

Conversation

weihuang-jedi
Copy link
Contributor

Make ATM-OCN-ICE coupling model run on AWS.

Description

This add capability to run UFS atm-ocn-ice coupling on AWS.

Resolves #2858

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? YES
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • Clone and build on AWS
  • run test on AWS

Checklist

  • [x ] Any dependent changes have been merged and published
  • [ x] My code follows the style guidelines of this project
  • [ x] I have performed a self-review of my own code
  • [ x] I have commented my code, particularly in hard-to-understand areas
  • [ x] My changes generate no new warnings
  • [ x] New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

weihuang-jedi and others added 30 commits June 18, 2024 23:05
@weihuang-jedi
compiled OK now
103f2c4
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
916ff6c
@weihuang-jedi
re-test on aws with fewer changes
b0ac406
@weihuang-jedi
make change in tasks.py to avoid error finding libiomp5.so problem
3de972f
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
8308375
@weihuang-jedi
add comments so the reviewers know that these changes are for AWS, an…
bc4c4a8 
…d hope can provide a better way
@weihuang-jedi
Merge branch 'aws-forecast-only' of ssh://github.com/NOAA-EPIC/global…
924aede 
…-workflow-cloud into aws-forecast-only
@weihuang-jedi
add comments so the reviewers know that these changes are for AWS, an…
b724937 
…d hope can provide a better way
@weihuang-jedi
reverse config.resource changes, and memory restriction on AWS
12ab29f
@weihuang-jedi
sync with emc repo
adff250
@weihuang-jedi
move common data to a shared place
2290ea2
@weihuang-jedi
use ICs from s3-bucket
cd2c8e7
@weihuang-jedi
Merge branch 'develop' into aws-forecast-only
4e144e5
@weihuang-jedi
change as suggested by reviewer
46e3ef5
@weihuang-jedi
sync with develop
32f13eb
@weihuang-jedi
sync sorc/ufs_model.fd
a34a4c8
@weihuang-jedi
remove mpmd_opt from APRUN_UFS
44011a3
@weihuang-jedi
mpmd_opt and switch off tracker/genesis default for AWS
965ec80
@weihuang-jedi
add TODO
3ce268e
@weihuang-jedi
remove ncl version on AWS
f03ac78
@weihuang-jedi
Merge remote-tracking branch 'origin/develop' into aws-forecast-only
007a56b
@weihuang-jedi
sync ufs_model
2f6ec6e
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
dba83a7
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
24fe804
@weihuang-jedi
sync and remove gempak from noaacloud
e8a2e0f
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
4013eb1
@weihuang-jedi
update modules hash
a548c7f
@weihuang-jedi
update module hash
d37e646
@weihuang-jedi
use bucket
2a80162
@weihuang-jedi
remove /scratch1, but kept TODO
fa44862
weihuang-jedi and others added 18 commits July 30, 2024 15:48
@weihuang-jedi
merge fro develop
e1a57b4
@weihuang-jedi
Remove ACCOUNT_SERVICE
fe9a457
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-forecast-only
5c6e052
@weihuang-jedi
Merge branch 'develop' of ssh://github.com/NOAA-EPIC/global-workflow-…
1a273ae 
…cloud into aws-atmc768
@weihuang-jedi
update AWSPW.env for coupled model
cc0ae02
@weihuang-jedi
Merge remote-tracking branch 'origin/develop' into aws-coupled-atm-oc…
7bf88dd 
…n-ic3-c192
@weihuang-jedi
Merge remote-tracking branch 'origin/develop' into aws-coupled-atm-oc…
c429225 
…n-ic3-c192
@weihuang-jedi
add C768
c2b765d
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192
199e8be
@weihuang-jedi
make c48s2sw run, but have issue with gfswavepostpnt
e9734a9
@weihuang-jedi
sync
a96dcca
@weihuang-jedi
synce with develop
1920240
@weihuang-jedi
turn off wave on CSPs
0fafa6b
@weihuang-jedi
sync with develop
3256945
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192
aa43bcb
@weihuang-jedi
Merge branch 'NOAA-EMC:develop' into aws-coupled-atm-ocn-ic3-c192
398d40d
@weihuang-jedi
sync with develop
1ca7d1f
@weihuang-jedi
revert change to develop
fd90543
@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title support coupling on AWS Support coupling on AWS Aug 27, 2024
aerorahul
aerorahul reviewed Aug 27, 2024
View reviewed changes
env/AWSPW.env Outdated Show resolved Hide resolved
@weihuang-jedi weihuang-jedi marked this pull request as ready for review August 27, 2024 17:18
aerorahul
aerorahul approved these changes Aug 27, 2024
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.

WalterKolczynski-NOAA
WalterKolczynski-NOAA approved these changes Aug 28, 2024
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fan of manually changing the DO_WAVE switch and creating a mismatch with the APP rather than just running with APP that don't run waves. But fine for a short-term work-around.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 85c7659 into NOAA-EMC:develop Aug 28, 2024
5 checks passed
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Aug 30, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into fix/early_run_configs
c4bba29 
* origin/develop:
  Add 3 and 9 hr increment files to IC staging (NOAA-EMC#2876)
  Add diffusion/diag B for aerosol DA and some other needed changes (NOAA-EMC#2738)
  Correct ocean `MOM.res_#` stage copy (NOAA-EMC#2868)
  Support coupling on AWS (NOAA-EMC#2859)
  Add JEDI ATM lgetkf observer and solver jobs (NOAA-EMC#2833)
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Sep 9, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into feature/ss_180
386a940 
* origin/develop:
  Create JEDI class (NOAA-EMC#2805)
  Restructure the bufr sounding job    (NOAA-EMC#2853)
  Add an archive task to GEFS system to archive files locally (NOAA-EMC#2816)
  Reenable Orion Cycling Support (NOAA-EMC#2877)
  Eliminate race conditions and remove DATAROOT last in cleanup (NOAA-EMC#2893)
  Update aerosol climatology to 2013-2024 mean (NOAA-EMC#2888)
  Add ability to run CI test C96_atm3DVar.yaml to Gaea-C5 (NOAA-EMC#2885)
  Support global-workflow GEFS C48 on Google Cloud (NOAA-EMC#2861)
  Add 3 and 9 hr increment files to IC staging (NOAA-EMC#2876)
  Add diffusion/diag B for aerosol DA and some other needed changes (NOAA-EMC#2738)
  Correct ocean `MOM.res_#` stage copy (NOAA-EMC#2868)
  Support coupling on AWS (NOAA-EMC#2859)
  Add JEDI ATM lgetkf observer and solver jobs (NOAA-EMC#2833)
  Fix gdas build on Gaea and add Gaea to available CI list (NOAA-EMC#2857)
  Support ATM forecast only on Google (NOAA-EMC#2832)
  Add GEFS C48 support on AWS (NOAA-EMC#2818)
  Update omega calculation (NOAA-EMC#2751)
  Add snow DA update and recentering for the EnKF forecasts (NOAA-EMC#2690)
  support ATM forecast only on Azure (NOAA-EMC#2827)
  Convert staging job to python and yaml (NOAA-EMC#2651)
  Fixed test on UNAVAILBLE in python Rocoto check (NOAA-EMC#2842)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul aerorahul approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support coupling on AWS
3 participants
@weihuang-jedi @aerorahul @WalterKolczynski-NOAA