Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace bash env files with modules #238

Merged

Conversation

danielabdi-noaa
Copy link
Collaborator

@danielabdi-noaa danielabdi-noaa commented Apr 12, 2022

DESCRIPTION OF CHANGES:

  • Replaces environment files with modules. This should make build and workflow execution SRW app from both bash and tcsh login shells possible.

  • Removes detect_machine.sh script and forces the user to pass --platform=PLATFORM instead. Auto-detection of machine is error prone and an impediment to new systems, e.g. Gaea

TESTS CONDUCTED:

Tested build on Hera, Jet, Orion, Gaea, WCOSS and Singularity :

./devbuild.sh --platform=hera

Cheyenne, Macosx and Odin need testing


To build directly with cmake, as is the current recommendation, some systems such as Odin/Gaea/Macosx need
to properly initialize Lmod first. This is taken care of automatically if you build with devbuild.sh but has to be done
manually otherwise. Assuming you are using bash login shell:

source etc/lmod-setup.sh gaea
module use modulefiles
module load build_gaea_intel
...

DEPENDENCIES:

None

DOCUMENTATION:

Updated docs/INSTALL and docs/RUNTIME instructions, and BuidRunSRW.rst as well

ISSUE (optional):

None

CONTRIBUTORS (optional):

None

@chan-hoo
Copy link
Collaborator

@danielabdi-noaa, can you explain why you try to remove 'detect_machine.sh' and go back to the platform argument option? I personally like this approach because I don't need to think of which machine I am working on and I just need to type 'devbuild.sh' on the terminal. This approach is also used in the UFS Weather Model. Do you have any specific reasons to remove it?

@RatkoVasic-NOAA
Copy link
Collaborator

@danielabdi-noaa, can you explain why you try to remove 'detect_machine.sh' and go back to the platform argument option? I personally like this approach because I don't need to think of which machine I am working on and I just need to type 'devbuild.sh' on the terminal. This approach is also used in the UFS Weather Model. Do you have any specific reasons to remove it?

We definitely need detect_machine.sh both in UFS Weather Model App and SRW app.

@christopherwharrop-noaa
Copy link
Collaborator

christopherwharrop-noaa commented Apr 12, 2022

@chan-hoo / @RatkoVasic-NOAA - I disagree. Detecting the machine is inherently unportable and unmaintainable and provides no added value. Not to mention that it publishes FQDN of our machines in a public repo. There is absolutely no reason for it because every user always knows what machine they are using and can easily provide that information. Having a lot of fragile code to provide a service that save a human mere seconds, is not worth it. Furthermore, it makes working on additional platforms less intuitive and gives the incorrect impression to users that the system only works in certain places. Replacing automated detection of machine with a human input makes more sense and removes portability issues. Automated detection of the machine is not a scalable, maintainable, portable, or secure, feature and should be removed.

@chan-hoo
Copy link
Collaborator

@christopherwharrop-noaa, I disagree with you. Saving even a few seconds is worth it. I assume that most users of the UFS SRW App/Regional Workflow are working on the tier-1 and tier-2 machines. Using the script can save their precious time. You can find some specific machines in 'ufs-srweather-app/env/' and 'regional_workflow/ush/machine/'. If an anonymous user wants to use the UFS SRW App on a machine that is not listed there, the user should add scripts to these directories. If the user can do this, it will not be hard to add its machine to the list in 'detect_machine.sh'. Why do you think it is not scalable, maintainable, portable, or secure? I do not think that just using one auto-detection script is such a big deal. From this point of view, the UFS SRW App / Regional workflow are not scalable, maintainable, portable, or secure.

@christopherwharrop-noaa
Copy link
Collaborator

"saving even a few seconds is worth it. " - No it isn't. Sacrificing portability and maintainability of the system to avoid typing "hera" or "wcoss" is a very bad trade. The benefit is almost non-existent, while the disadvantages are plentiful.

"I assume that most users of the UFS SRW App/Regional Workflow are working on the tier-1 and tier-2 machines" - This assumption is directly responsible for some very poor design choices which many of us are now trying to reverse. It doesn't matter where "most" users run because it still needs to be functional and portable, even to non-Tier-1 machines. A user should on a non-tier-1 platform should NEVER have to edit code or the build system to make the system work, they should only have to provide a env file that configures the environment.

The auto-detection feature scales linearly with O(N) where N is the number of machines. Unfortunately the number of machines is unlimited, so keeping that up-to-date means adding more and more machines. The ufs-community should not be responsible for maintaining an unbounded growing list of stanza's in that script.

It is unmaintainable because the detection feature's correctness depends on outside entities beyond the control of ufs-community. Names of machines change, and the details may change over time without anyone noticing. Every time a machine changes or is added, the code (not just configuration) in the repository has to change. It is fragile code that depends on magic, hard-coded, constants.

It is fundamentally unportable because the auto-detection feature only works on machines for which someone has added code. The instant any part of the UFS system relies on detection of specific machines, it becomes unportable. Portability means it works everywhere without changing the code. Use of auto-detection of machine fundamentally breaks that. Portability is also a critical requirement for community-based software.

It is insecure because it embeds FQDN into code published in a public repository. While those details are probably easily accessible to would-be attackers via other means, it seems irresponsible to advertise an entire list of hostnames for all of our Tier-1 platforms.

@chan-hoo
Copy link
Collaborator

@christopherwharrop-noaa, I still disagree with you :) However, if other reviewers agree with it, I'll respect their opinions.

Copy link
Collaborator

@christinaholtNOAA christinaholtNOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danielabdi-noaa Thanks for doing this. It is super helpful for those of us running and testing on the cloud. We definitely have reached a state of running on a whole variety of platforms of our own creation that are not the Tier-1 platforms, and this is an important step toward reducing machine dependency for all that work.

It is also important to capture the idea that /bash is not the default for every user, and this transition to module files is a huge help in streamlining the build/deploy process.

I did make a couple of comments below about some issues that might need some follow up tests. Someone who has access to Odin might be able to help test that, and I can help with the Jet/Hera tests. Those wflow_ module files aren't a part of the workflow from what I can tell, so they just need to be tested manually to make sure you get the python environment you need.

env/set_machine.sh Outdated Show resolved Hide resolved
env/wflow_odin Outdated
else
export PATH="/scratch/software/Odin/python/anaconda2/bin:$PATH"
fi
fi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this section would work. It looks fairly bash-like.

It may be useful for someone to install a module to handle this Python installation. There seems to be no way to "unload" it and that can lead to issues on some platforms.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, wflow_odin seem to be the most challenging. TCL language allows to call shell scripts or commands via a puts to stdout so was just hoping it will work. Recent modules versions have sh-to-mod, but couldn't find it on any system to test ...

env/wflow_jet Outdated Show resolved Hide resolved
@christinaholtNOAA
Copy link
Collaborator

@chan-hoo We are fundamentally in a new age of spreading to all sorts of non-Tier-1/Tier-2 machines, and it is unbelievably time consuming to be able to handle these sorts of machine dependencies each time we spin up new cloud resources. The cloud resources, for the most part, are not meant to be "supported platforms" and the system will be much more portable if we can have more flexible ways for the user to specify the platform from the command line.

I believe that being able to specify a machine on the command line is a tiny compromise for enabling smooth portability to platforms that could free up some resources on research platforms.

@chan-hoo
Copy link
Collaborator

@christinaholtNOAA, Thank you for your explanation. This is pretty much convincing to me. Sorry, Chris :) :) I'll agree with this change from now on. I think that the directory 'env' should be renamed to 'modulefiles' because all other external components have their module files in the directory 'modulefiles'. What do you think of it?

@christopherwharrop-noaa
Copy link
Collaborator

@chan-hoo - Renaming env to modulefiles seems reasonable to me.

@danielabdi-noaa
Copy link
Collaborator Author

@chan-hoo @christopherwharrop-noaa Thanks for the suggestion! I have renamed env to modulefiles now since it only contains modulefiles now.

@danielabdi-noaa danielabdi-noaa changed the title Feature/modules Replace bash env files with modules Apr 27, 2022
@danielabdi-noaa danielabdi-noaa marked this pull request as ready for review April 27, 2022 19:15
@danielabdi-noaa danielabdi-noaa merged commit 74d9249 into ufs-community:develop May 1, 2022
gsketefian added a commit to ufs-community/regional_workflow that referenced this pull request May 5, 2022
## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR #725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).
mark-a-potts added a commit to ufs-community/regional_workflow that referenced this pull request May 11, 2022
* Add Gaea as a supported platform for the regional_workflow (#734)

* Updates to port regional workflow to gaea

* Temp change with -v as batch option

* new fixes for gaea/slurm

* Updated time for make lbcs

* added TEST data directory path

* Update gaea.sh

* fixes for PR

* Add more parameters to CSV file containing WE2E test info (#740)

## DESCRIPTION OF CHANGES: 
The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests.  Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description.  In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file.  This additional information consists of the values of the following experiment variables for each test:
```
PREDEF_GRID_NAME
CCPP_PHYS_SUITE
EXTRN_MDL_NAME_ICS
EXTRN_MDL_NAME_LBCS
DATE_FIRST_CYCL
DATE_LAST_CYCL
CYCL_HRS
INCR_CYCL_FREQ
FCST_LEN_HRS
LBC_SPEC_INTVL_HRS
NUM_ENS_MEMBERS
```
In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members).  

## TESTS CONDUCTED: 
The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`.  This created a new CSV file that contained the new fields (columns).  The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct.

## DOCUMENTATION:
The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`.  This PR slightly modifies that documentation to update it.

* Update directory structure of NCO mode (#743)

* update vertical structure of NCO mode

* update sample script for nco

* Fix typo on write component of new RRFS CONUS

* Default CCPP physics option is FV3_GFS_v16 (#746)

* Updated the default CCPP physics option to FV3_GFS_v16

* Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh

Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>

* Adds an alternative python workflow generation path (#698)

* Workflow in python starting to work.

* Use new python_utils package structure.

* Some bug fixes.

* Use uppercase TRUE/FALSE in var_dfns

* Use config.sh by default.

* Minor bug fixes.

* Remove config.yaml

* Update to the latest develop

* Remove quotes from numbers in predef grid.

* Minor bug fix.

* Move validity checker to the bottom of setup

* Add more unit tests.

* Update with python_utils changes.

* Update to latest develop additions (Need to re-run regression test)

* Use set_namelist and fill_jinja_template as python functions.

* Replace sed regex searches with python re.

* Use python realpath.

* Construct settings as dictionary before passing to fill_jinja and set_namelist

* Use yaml for setting predefined grid parameters.

* Use xml parser for ccpp phys suite definition file.

* Remove more run_command calls.

* Simplify some func argument processing.

* Move different config format parsers to same file.

* Use os.path.join for the sake of macosx

* Remove remaining func argument processing via os.environ.

* Minor bug fix in set_extrn_mdl_params.sh

* Add suite defn in test_data.

* Minor fixes on unittest on jet.

* Simplify boolean condition checks.

* Include old in renaming of old directories

* Fix conflicting yaml !join tag for paths and strings.

* Bug fix with setting sfcperst dict.

* Imitate "readlink -m" with os.path.realpath instead of os.readlink

* Don't use /tmp as that is shared by multiple users.

* Bug fix with cron line, maintain quotes around TRUE/FALSE.

* Update to latest develop (untested)

* Bug fix with existing cron line and quotes.

* Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR.

* Update to latest develop

* More updates.

* Bug fix thanks to @willmayfield! Check both starting/ending
characters are brackets for shell variable to be considered an array.

* Make empty EXPT_BASEDIR workable.

* Update to latest develop

* Update in predef grid.

* Check f90nml as well.

Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>

* Fix typo and crontab issue on wcoss dell in workflow python scripts (#750)

* Fix typo and failure on wcoss

* fix new line issue on wcoss dell

* remove capture_output

* Get USER from environment

Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>

* Add new WE2E configs (#748)

## DESCRIPTION OF CHANGES: 
Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. 

In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. 

## TESTS CONDUCTED: 
Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance.

## DEPENDENCIES:
None.

## DOCUMENTATION:
No documentation changes are required.

* Added a fixed WoF grid and the python tool to determine the write component parameters (#733)

* Added a fixed WoF grid and the python tool to determine the write component parameters

* Update set_predef_grid_params.sh

* Renamed file as recommended and removed unused lines

* Modified comment

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>

* Replace env with modulefiles in scripts (#752)

* change env to mod

* update we2e script

* WE2E script improvements for usability (#745)

## DESCRIPTION OF CHANGES: 
* Modifications to `run_WE2E_tests.sh`:
  * Add examples to help/usage statement
* Modifications to `check_expts_status.sh`:
  * Add arguments list that can be processed by `process_args`
  * Add new optional arguments:  `num_log_lines`, `verbose`
  * Include a help/usage message

## TESTS CONDUCTED:
* Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message.
* Ran `check_expts_status.sh --help` from the command line and got the expected help message.
* Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected.
* Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message.
 
## DEPENDENCIES:
PR #[241](ufs-community/ufs-srweather-app#241)

## DOCUMENTATION:
A lot of this PR is documentation in the scripts.  There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app.

* Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (#744)

* Bug fixes (grid size + suppress screen output from module load) (#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR #725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).

* Update default SPP ISEED array in config_defaults.sh to use unique values (#759)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Modify default ISEED values for SPP

* Fix grid in WE2E test

* Update workflow python scripts (#760)

* update python scripts

* Change output file name of run_post to meet NCO standards (#758)

* change output file name

* change variable name

* update python script

* remove duplicates

* add a check for empty variables

* move variable to common area

* clean up unnecessary comments

* update scripts

* remove duplicate

* update python scripts

* fix user-staged dir path issue in python script

* Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (#763)

* Add new var to we2e tests for new grids

* rename we2e tests for custom grid

* remove unnecessary $

Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>
Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com>
Co-authored-by: Yunheng Wang <47898913+ywangwof@users.noreply.github.com>
Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
@natalie-perlin
Copy link
Collaborator

@danielabdi-noaa - a couple of issues found when using the TCL modulefiles build_macos_gnu and wflow_macos for the MacOS would you mind sharing some thoughts whether they could be fixed, were there any similar issues on other platforms?

  • build_macos_gnu: setting LDFLAGS environment variable is not working. This seems to be a known issue (discussed elsewhere) with the TCL modules (#%Modules), because the LDFLAGS uses other env. variable $MPI_ROOT that is only available after loading another module, and not being a system environment variable. As far as I understand, TCL file does not actually load the module, it only contains the instructions. Thus the environment variables of modules being loaded (i.e., in "module load xxxx") are not available within the TCL file when this file is interpreted by Lua. However, the variable LDFLAGS could be successfully set after the module load of build_macos_gnu is completed. My workaround suggestion for the moment is to comment out the line setenv LDFLAGS "-L$env(MPI_ROOT)/lib" , and suggest a user to run this line in the terminal! Maybe more elegant solution could be later, but this works for now.

  • wflow_macos, which supposed to activate python environment "regional_workflow" created earlier, does not seem to work for me. It does OK if I repeat the instructions manually, or when sourcing a similar wflow_macos.sh script in sh/bash - compatible form (instead of loading the module/script).
    Are there any similar issues with other platforms, or why this could be the case with MacOS?. - any comments are welcome! I could include this wflow_macos.sh script along with the original TCL wflow_macos in the repository, as a temporary workaround or at least for the PR to the SRW repo with the changes for the MacOS, until a better solution is found.
    Attached is a file wflow_macos.sh except renamed to wflow_macos.sh.txt to conform to the attachable formats allowed by GitHub
    wflow_macos.sh.txt

@danielabdi-noaa
Copy link
Collaborator Author

@natalie-perlin Thanks for testing it out. I was not able to test modulefiles on macosx because I lack admin privilages on my NOAA mac laptop.

There are similar issues on other systems, for example, one can't load miniconda3 and execute conda activate in the same modulefile because the environment changes made by module load miniconda3 will only be available after the module is loaded. But for macos I think the problem is solvable atleast for wflow_macos. It looks like the problem is that the environment variable set with setenv is not immediately available within the same modulefile, so maybe using a local variable will work instead using something like:

set VENV_t "/Users/username/venv/regional_workflow"
setenv VENV $VENV_t
if { [module-info mode load] } { 
  system "source $VENV_t/bin/activate;"
}

Let me know if that works.

For build_macos_gnu, I think the problem is similar i.e the environment changes that loading the mpi module makes such as MPI_ROOT are not yet available in the modulefile. If you know the path where MPI is installed, you can set LDFLAGS directly with it but that is not an ideal solution. I will look for other solutions for this one ...

@natalie-perlin
Copy link
Collaborator

@danielabdi-noaa - Hi Daniel,
Yes, thank you, I did test using a local variable, it did not help. Only when/if this command "source $VENV/bin/activate" is launched outside the script it does start the python environment.

So maybe this file wflow_macos could be left unused until it is more clear on how to activate the python env. from this module. At the present moment, MacOS users could then be instructed just to load the python env. manually from the command line.

@danielabdi-noaa
Copy link
Collaborator Author

@natalie-perlin I just installed lmod with a local installation of homebrew and figured out the solution for wflow_macos.
Using puts instead of system seem to solve the problem and venv is loaded first time.
On linux systems though, the opposite is true which is why I have been using system instead of puts there.
Using TCL modulefile with Lmod causes problems depending on version numbers

Change

if { [module-info mode load] } { 
   puts "source $env(VENV)/bin/activate;"
}
abdi-lt:~ daniel.abdi$ source homebrew/opt/lmod/init/bash 
abdi-lt:~ daniel.abdi$ module --version

Modules based on Lua: Version 8.7.2  2022-05-04 13:42 -05:00
    by Robert McLay mclay@tacc.utexas.edu

abdi-lt:~ daniel.abdi$ ls modulefiles/
wflow_macos
abdi-lt:~ daniel.abdi$ module use modulefiles
abdi-lt:~ daniel.abdi$ module load wflow_macos
(venv) abdi-lt:~ daniel.abdi$ deactivate
abdi-lt:~ daniel.abdi$ 

@danielabdi-noaa
Copy link
Collaborator Author

The same goes for the system command that sets stack size to unlimited in build_macos_gnu.

puts "ulimit -S -s unlimited;"

natalie-perlin pushed a commit that referenced this pull request May 20, 2022
* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

Co-authored-by: gspetro <gillian.s.petro@gmail.com>
JeffBeck-NOAA pushed a commit that referenced this pull request May 21, 2022
* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

* clarify compiler prereqs

Co-authored-by: gspetro <gillian.s.petro@gmail.com>
@natalie-perlin
Copy link
Collaborator

@danielabdi-noaa - Thank you for your comments, all worked out!
setting env. variable LDFLAGS in terminal/cli, immediately after loading "build_macos_gnu" did solve the issue to compile the code.

$ module load build_macos_gnu
$ export LDFLAGS="-L${MPI_ROOT}/lib"

With the workflow environment activation: using "puts" did solved the issue.

So all is good with the modulefiles! :)

natalie-perlin pushed a commit that referenced this pull request May 26, 2022
* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* create MacOS install/build instructions

* add MacOS Build/Run instructions

* fix MacOS Build/Run details

* add MacOS info directly to Build/Run SRW chapter

* minor details

* minor edits

* update Include-HPCInstall with mac installation docs

* add note re: Terminal.app & bash shell in MacOS section

* remove MacInstall file-contents added to BuildRunSRW

* update hpc-stack submodule to include mac installation info

* add MacOS config details

* add MacOS config & run details

* minor MacOS note

* mention need to verify software library version #'s

* update hpc-stack-mod

* align MacDetails section with PR #238 info

* remove gsed & alter related commands

* update hpc-stack submodule

* typos

* switch from env to module load

Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>

* Update BuildRunSRW.rst

* update hpc-stack module docs & MacOS config.sh

* update machine file instructions

* updates to BuildRun chapter

* fix typo

Co-authored-by: gspetro <gillian.s.petro@gmail.com>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
This was referenced May 27, 2022
christinaholtNOAA referenced this pull request in NOAA-GSL/ufs-srweather-app Jun 7, 2022
* Add support on NSSL/Odin (#227)

* Add support on NSSL/Odin

* Add wlfow_odin.env and modify detect_machine.sh

* update comment

* Detect explicitly odin1/odin2

* Add python module to cheyenne build environments (#232)

* Update SRW Documentation (#212)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

* Update hashes of all components to latest versions (#233)

* Add a Contributor's Guide feature to the docs (#228)

* create contributor's guide

* add guidelines for making good PR

* good pull request edits

* 1st Draft of COntributor's Guide

* minor formatting edits

* Add instructions for srw_common module load

* fix module files guidance

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* add info on documentation requirements

* 1st round of J. Beck's edits

* add textonly issue template + minor typo-type fixes

* minor edits/formatting

* remove pylintrc requirement

* fixed .yaml & other minor

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

* Updates to parameters in config_defaults .rst files (#237)

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

* Added functionality for MacOS X (#242)

* Added functionality for MacOS X

 Functionality for MacOS, updated module list in srw_common

* Update build_macos_gnu.env

* Update srw_common

* Update build_macos_gnu.env

The env/build_mac_gnu.env does not load srw_common module, but instead loads individual HPC-stack modules built locally on the Mac that contain higher-versions of some packages. This avoids conflicts with SRW builds for other platforms.

* Update srw_common

corrected the version of the gftl-shared

* Update build_macos_gnu.env

No need to load libpng module found in srw_common, as this is not being built as a part of the hpc-stack on MacOS X, rather installed system-wide.

Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>

* Add gaea to supported platforms (#236)

* fixes for gaea

* updates for gaea

* tweak for build env

* 2nd tweak for build env

* Fixes for slurm

* another fix for env

* added version for cmake

* Update Externals.cfg

* Update wflow_gaea.env

* tweak

* pulling Externals.cfg explicitly from develop

* temporarily removing externals

* Checked out directly from develop

* Bug fix with singularity env files (#245)

* Replace bash env files with modules  (#238)

* Pass machine name to build scripts.

* Use modules environment instead of shell scripts.

* Leave conda activation to the user.

* Remove set_machine script.

* Rename env to modulefiles

* Minor fix.

* Minor fix

* Take out *module purge* from modufiles and put it in devbuild.sh

* Activate conda directly in signularity modulefile.

* Minor fixes.

* Add Gaea modulefiles.

* Restore odin env files.

* Bug fixes in singularity modulefiles.

* Move activation of Lmod to devbuild.sh

* Don't do 'module purge' on cray systems

* Put Lmod initialization code in separate script.

* Go back to using modulefile for odin.

* Optionally pass machine name to lmod-setup.sh

* Modify odin wflow modulefile.

* Allow unknown platforms in devbuild.sh

* Update documentation.

* Move cmake init out of lmod-setup.sh on odin

* Also update markup language build documentation.

* Lmod setup script for both bash and tcsh login shells.

* Some fixes for tcsh login shell.

* Add singularity platform to lmod-setup

* update hash of regional workflow (#247)

* Update WE2E documentation (#241)

## DESCRIPTION OF CHANGES: 
This updates the documentation for how to use the WE2E testing system.

## TESTS CONDUCTED: 
Compiled the documentation using `sphinx-build` and viewed the resulting html.

## DEPENDENCIES:
PR #[745](ufs-community/regional_workflow#745) in regional_workflow.

## CONTRIBUTORS: 
@gspetro

* fixes for gaea modules (#248)

* Update regional_workflow hash, add shortcuts for common devbuild.sh options (#251)

* Modifications to `CODEOWNERS` file (#252)

## DESCRIPTION OF CHANGES: 
Make the following modifications to the github `CODEOWNERS` file:
1) Add several EPIC staff (@gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA) so they are notified of all PRs and can review them.
2) Remove duplicate entries.
3) Remove users who will no longer be working with the repo (thus far only @jwolff-ncar) .

## TESTS CONDUCTED: 
None.

* Add verification tasks to documentation (#243)


* METplus, model config & forecast params, separator

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* condense VX task table using ##

* update README

* add png and revert hpc-stack commits until PR#240 (mac docs) is approved

* jwolff edits

* add info on run_vx.local

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

* Add NOAA cloud platforms to SRW (#221)

* updates for noaacloud

* working version

* pointing to noaa-epic for testing

* changes for noaacloud

* switched to load-any

* fix for regional_workflow pointer (#260)

* update hash of regional workflow (#261)

* Feature/cheyenne fix (#258)

* Adding a github action to build on cheyenne with intel

* fixing yml

* fixes for missing load-any on cheyenne

* added pio as well

* Update .github/workflows/build.yml

Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>

Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>

* tweaks for build/run on gaea (#254)

* tweaks for build/run on gaea

* fixed path

* Updates for PR

* Check-in Jenkinsfile and unified scripts (#253)

* Add Jenkinsfile that includes a build and test pipeline, which
leverages the unified build and test scripts
* Supported platforms are Cheyenne, Gaea, and Orion
* Supported compilers are GNU and Intel

* Fix for miniconda3 load on Hera (#257)

Pin down the version of miniconda3 on Hera, and do not append to the module path

* Updates to Remaining Chapters (6 & 8-12) of SRW Docs/User's Guide (#255)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

* Update compiler prerequisite in docs (#267)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* edit config intro & platform environment sections

* edit sections on cron & directory parameters, platform & parameters for running without a workflow manager

* edit NCO, file-separator, filename params, add some METplus and model config params

* ConfigWorkflow.rst revisions, added METplus to Components, grid info

* add grid config details

* changes to readme.md

* RTD readme.md edits

* create MacOS install/build instructions

* update task run and grid parameters

* fixed file params & workflow task params

* 1st draft of ConfigParameters.inc

* minor edits

* add stochastic physics var details

* update FVCOM, thread affinity params

* halo_blend, ens, crtm, custom post, subhourly updates

* update HPC-Stack submodule/docs

* remove extra macinstall document

* Rocoto WF tasks & params

* workflow tasks/params, debug, verbose, pre-existing dir, predefined grid

* move Stochastic physics to CCP section; write component edits

* comp'l forecast, grid gen, NOMADS, user-staged files

* METplus, model config & forecast params, separator

* 2nd draft complete

* physics updates

* remove MacInstall empty file

* undo hpc-stack submodule update (save for separate PR)

* undo hpc-stack install doc update (save for separate PR)

* revert hpc-stack submodule update

* revisions to SPP & LSM physics

* minor edits

* update comments in LAM Grid chapter

* update LSM_SPP_EACH_STEP

* revert LSM_SPP_EACH_STEP to original definition

* combine config info into one doc instead of two

* remove ConfigParameters.inc (contents now appear in ConfigWorkflow.rst)

* update hpc-stack docs submodule

* odds & ends

* add VX tables, config info, & Rocoto output tables

* add module use/load statements, fix typos

* varied minor details

* add workflow svg diagram

* edits to rocoto ch

* updates to Rocoto chapter

* fix minor formatting/wording issues

* updates to LAMgrid chapter

* LAM Grid edits

* LAM ch: user-defined grid section

* add UPP Product tables ch 6

* I/O edits & glossary terms

* I/O Pt2

* I/O changes

* include updated images

* update docs to reflect changes in PR #238

* Graphics Ch-1st pass

* minor updates to Graphics

* minor updates to Graphics

* edit ConfigNewPlatform sections 1-4

* ConfigNewPlatform edits

* resolve merge conflicts

* I/O ch edits

* I/O edits

* more I/O edits

* hpc-stack submodule updates

* add HPC-Stack MacOs info

* WE2E edits & tables

* fix typo

* minor grammar/typos

* merge conflict resolution

* merge conflict resolution

* fix grid name

* remove resolved comments

* add compact grids

* file path updates & info for HPC-Stack

* add SRW prereqs to Intro

* change ConfigNewPlatform to a non-container quickstart

* clean up non-container quickstart

* update build options for non-container QS

* update file paths & WE2E

* minor fixes

* update I/O & Gaea file paths

* update error in non-container QS

* add warning for users w/o Rocoto

* add UPP Satellite Product instructions

* Xlink for UPP satellite output info

* clarify compiler prereqs

Co-authored-by: gspetro <gillian.s.petro@gmail.com>

Co-authored-by: Yunheng Wang <47898913+ywangwof@users.noreply.github.com>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>
Co-authored-by: gspetro <gillian.s.petro@gmail.com>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Jesse McFarland <jesse@mcfarland.sh>
christinaholtNOAA added a commit to NOAA-GSL/regional_workflow that referenced this pull request Jun 7, 2022
* Add missing user-defined stochastic physics options; fix stochastic physics seed generation script (ufs-community#704)

## DESCRIPTION OF CHANGES: 
Add missing user-defined options for tendency-based stochastic physics and fix the ensemble-based seed generation script to work regardless of whether stochastic physics is turned on or not.

## TESTS CONDUCTED: 
Tested on Hera using the following WE2E configurations with and without stochastic physics:

config.grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh
config.community_ensemble_2mems.sh

## ISSUE (optional): 
[Issue ufs-community#702](ufs-community#702)

## CONTRIBUTORS (optional): 
Thanks to @mkavulich and @chan-hoo for finding this problem.

* Add namelist option for netCDF4 when running with the 3-km NA domain; update NAM HPSS settings and WE2E tests (ufs-community#707)

* Change to netcdf4 when using the NA 3-km domain

* Update HPSS paths for NAM data

* Update NAM HPSS locations and dates for WE2E tests.

* Remove lines from merge.

* Tweaks to allow compiler and build_env_fn to be specified in the run_WE2E_test.sh script (ufs-community#711)

* Changed 20200304 to 20200303 in ush/mrms_pull_topofhour.py (ufs-community#712)

* Remove unused rocoto directory in ush (ufs-community#720)

* Fix bug for nco we2e tests on Orion; re-organize we2e input data and nco we2e tests (ufs-community#713)

* Update machine script for orion

* Update machine script for wcoss_dell_p3

* Update we2e run script for wcoss and orion

* Reorganize nco we2e tests

* remove machine based logic

* Add symlink for nco inline post test

* Added stand-alone verification scripts (feature/issue_683_standaloneVX) (ufs-community#726)

* Grid-stat and point-stat run scripts.

* Stand-alone scripts for verification.

* Added comments to gridvx scripts.

* Added qsub_job.sh and added comments to provide context on running Vx.

* remove machine base logic (ufs-community#727)

* Allow user-defined file names for input template files (ufs-community#717)

* Allow multiple template names

* parameterize file_TMPL_FN and add a we2e test

* Increase maxtries_task for make_grid/orog/sfc_climo

* Modify file name and description

* Changes to RRFS 3- and 13-km domains, setup.sh script bug fixes, make_ics task modification, and tweaks to stochastic physics namelist settings (ufs-community#721)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Change sotyp_from_climo to "true" based on operational RAP grib2 files.

* Update for changes to stochastic physics namelist options.

* Check for DO_ENSEMBLE="TRUE" when running ensemble verification and turn of VX when running in NCO mode.

* Revert to 3-km domain.

* Remove commented-out GFDL grid for the RRFS_NA_13km domain

* Add RRFS_NA_13km WE2E test

* Changes to comments.

* Adding 25 km tests to Jet/Hera suites. (ufs-community#718)

* Add a small 3km predefined grid over Indianapolis for testing (ufs-community#725)

* Add 3km grid over Indianapolis.  This is about 600km x 600km in extent (200 x 200 grid points).  It is intended for use in the WE2E tests.

* Edit comments.

* Use Python tool for get_extrnl_mdl_file tasks (ufs-community#681)

These changes hook in the Python-based data ingest tool, replacing the previous scripts that handled this work as part of the get_extrn_mdl_file task. No attempt was made in this PR to replace the NOMADS fetching script with the Python utility, but the NOMADS data location has been added to the data_locations.yml file.

The functionality to write the data summary file has also been added to the Python tool to match the capabilities of the existing workflow tools.

* Increase size of RRFS CONUS grid (ufs-community#724)

Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov>
Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov>

* add include-style quality mark options in metplus confs (ufs-community#738)

* Add Gaea as a supported platform for the regional_workflow (ufs-community#734)

* Updates to port regional workflow to gaea

* Temp change with -v as batch option

* new fixes for gaea/slurm

* Updated time for make lbcs

* added TEST data directory path

* Update gaea.sh

* fixes for PR

* Add more parameters to CSV file containing WE2E test info (ufs-community#740)

## DESCRIPTION OF CHANGES: 
The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests.  Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description.  In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file.  This additional information consists of the values of the following experiment variables for each test:
```
PREDEF_GRID_NAME
CCPP_PHYS_SUITE
EXTRN_MDL_NAME_ICS
EXTRN_MDL_NAME_LBCS
DATE_FIRST_CYCL
DATE_LAST_CYCL
CYCL_HRS
INCR_CYCL_FREQ
FCST_LEN_HRS
LBC_SPEC_INTVL_HRS
NUM_ENS_MEMBERS
```
In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members).  

## TESTS CONDUCTED: 
The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`.  This created a new CSV file that contained the new fields (columns).  The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct.

## DOCUMENTATION:
The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`.  This PR slightly modifies that documentation to update it.

* Update directory structure of NCO mode (ufs-community#743)

* update vertical structure of NCO mode

* update sample script for nco

* Fix typo on write component of new RRFS CONUS

* Default CCPP physics option is FV3_GFS_v16 (ufs-community#746)

* Updated the default CCPP physics option to FV3_GFS_v16

* Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh

Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>

* Adds an alternative python workflow generation path (ufs-community#698)

* Workflow in python starting to work.

* Use new python_utils package structure.

* Some bug fixes.

* Use uppercase TRUE/FALSE in var_dfns

* Use config.sh by default.

* Minor bug fixes.

* Remove config.yaml

* Update to the latest develop

* Remove quotes from numbers in predef grid.

* Minor bug fix.

* Move validity checker to the bottom of setup

* Add more unit tests.

* Update with python_utils changes.

* Update to latest develop additions (Need to re-run regression test)

* Use set_namelist and fill_jinja_template as python functions.

* Replace sed regex searches with python re.

* Use python realpath.

* Construct settings as dictionary before passing to fill_jinja and set_namelist

* Use yaml for setting predefined grid parameters.

* Use xml parser for ccpp phys suite definition file.

* Remove more run_command calls.

* Simplify some func argument processing.

* Move different config format parsers to same file.

* Use os.path.join for the sake of macosx

* Remove remaining func argument processing via os.environ.

* Minor bug fix in set_extrn_mdl_params.sh

* Add suite defn in test_data.

* Minor fixes on unittest on jet.

* Simplify boolean condition checks.

* Include old in renaming of old directories

* Fix conflicting yaml !join tag for paths and strings.

* Bug fix with setting sfcperst dict.

* Imitate "readlink -m" with os.path.realpath instead of os.readlink

* Don't use /tmp as that is shared by multiple users.

* Bug fix with cron line, maintain quotes around TRUE/FALSE.

* Update to latest develop (untested)

* Bug fix with existing cron line and quotes.

* Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR.

* Update to latest develop

* More updates.

* Bug fix thanks to @willmayfield! Check both starting/ending
characters are brackets for shell variable to be considered an array.

* Make empty EXPT_BASEDIR workable.

* Update to latest develop

* Update in predef grid.

* Check f90nml as well.

Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>

* Fix typo and crontab issue on wcoss dell in workflow python scripts (ufs-community#750)

* Fix typo and failure on wcoss

* fix new line issue on wcoss dell

* remove capture_output

* Get USER from environment

Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>

* Add new WE2E configs (ufs-community#748)

## DESCRIPTION OF CHANGES: 
Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. 

In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. 

## TESTS CONDUCTED: 
Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance.

## DEPENDENCIES:
None.

## DOCUMENTATION:
No documentation changes are required.

* Added a fixed WoF grid and the python tool to determine the write component parameters (ufs-community#733)

* Added a fixed WoF grid and the python tool to determine the write component parameters

* Update set_predef_grid_params.sh

* Renamed file as recommended and removed unused lines

* Modified comment

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>

* Replace env with modulefiles in scripts (ufs-community#752)

* change env to mod

* update we2e script

* WE2E script improvements for usability (ufs-community#745)

## DESCRIPTION OF CHANGES: 
* Modifications to `run_WE2E_tests.sh`:
  * Add examples to help/usage statement
* Modifications to `check_expts_status.sh`:
  * Add arguments list that can be processed by `process_args`
  * Add new optional arguments:  `num_log_lines`, `verbose`
  * Include a help/usage message

## TESTS CONDUCTED:
* Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message.
* Ran `check_expts_status.sh --help` from the command line and got the expected help message.
* Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected.
* Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message.
 
## DEPENDENCIES:
PR #[241](ufs-community/ufs-srweather-app#241)

## DOCUMENTATION:
A lot of this PR is documentation in the scripts.  There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app.

* Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (ufs-community#744)

* Bug fixes (grid size + suppress screen output from module load) (ufs-community#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).

* Update default SPP ISEED array in config_defaults.sh to use unique values (ufs-community#759)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Modify default ISEED values for SPP

* Fix grid in WE2E test

* Update workflow python scripts (ufs-community#760)

* update python scripts

* Change output file name of run_post to meet NCO standards (ufs-community#758)

* change output file name

* change variable name

* update python script

* remove duplicates

* add a check for empty variables

* move variable to common area

* clean up unnecessary comments

* update scripts

* remove duplicate

* update python scripts

* fix user-staged dir path issue in python script

* Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (ufs-community#763)

* Add new var to we2e tests for new grids

* rename we2e tests for custom grid

* remove unnecessary $

* Modifications to `CODEOWNERS` file (ufs-community#757)

* Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them.

* Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo.

* Adding a python utility for summarizing compute. (ufs-community#769)

Adds a utility that summarizes Rocoto database computational usage information.

* Add github actions for python unittests. (ufs-community#747)

* Add github actions for python unittests.

* Include all python script in ush

* Skip defining QUILTING params when it is set to False

* Update py_workflow

* Update unittest for set_extrn_mdl_params.

* Updates from develop.

Co-authored-by: Daniel Shawul <dshawul@yahoo.com>

* Update sample script for NCO mode (ufs-community#771)

* update config.nco.sh

* Add comment

* Feature/noaacloud (ufs-community#767)

* updates for noaacloud

* working version

* fixes for noaacloud

* added extra modules for post

* removed cheyenne-specific crontab editing section (ufs-community#773)

* Pin down hera miniconda3 module file version. (ufs-community#770)

Pin down the version of miniconda3 on Hera, and do not append to the module path.

* update staged data dir (ufs-community#774)

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com>
Co-authored-by: michelleharrold <harrold@ucar.edu>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: BenjaminBlake-NOAA <52074832+BenjaminBlake-NOAA@users.noreply.github.com>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v72a1.ncep.noaa.gov>
Co-authored-by: Benjamin.Blake EMC <Benjamin.Blake@v71a3.ncep.noaa.gov>
Co-authored-by: chan-hoo <chan-hoo.jeon@noaa.gov>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>
Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com>
Co-authored-by: Yunheng Wang <47898913+ywangwof@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
Co-authored-by: Daniel Shawul <dshawul@yahoo.com>
ywangwof added a commit to ufs-community/regional_workflow that referenced this pull request Jun 9, 2022
* Added workflow for RRFS_v1nssl

* Renamed FV3_RRFS_v1nssl to FV3_WoFS_v0 and added WE2E tests configuraiton files for suite FV3_WoFS_v0

* Added SUBCONUS test for WoFS suite

* water_nc initial value sets to zero for FV3GFS data

* Add Gaea as a supported platform for the regional_workflow (#734)

* Updates to port regional workflow to gaea

* Temp change with -v as batch option

* new fixes for gaea/slurm

* Updated time for make lbcs

* added TEST data directory path

* Update gaea.sh

* fixes for PR

* Add more parameters to CSV file containing WE2E test info (#740)

## DESCRIPTION OF CHANGES: 
The script/function `get_WE2Etest_names_subdirs_descs.sh` (which is called from `run_WE2E_tests.sh` if needed) creates a CSV (Comma-Separated Value) file named `WE2E_test_info.csv` that contains information about the WE2E tests.  Currently, this CSV file contains only 3 columns: the test name, the names of any alternate names for the test, and the test description.  In order to have a more complete summary of the WE2E tests, this PR modifies `get_WE2Etest_names_subdirs_descs.sh` so that additional information is included in the CSV file.  This additional information consists of the values of the following experiment variables for each test:
```
PREDEF_GRID_NAME
CCPP_PHYS_SUITE
EXTRN_MDL_NAME_ICS
EXTRN_MDL_NAME_LBCS
DATE_FIRST_CYCL
DATE_LAST_CYCL
CYCL_HRS
INCR_CYCL_FREQ
FCST_LEN_HRS
LBC_SPEC_INTVL_HRS
NUM_ENS_MEMBERS
```
In addition, the script uses this information to calculate the number of times each test calls the forecast model (e.g. if the test uses 3 different cycle dates, then the forecast model will be called 3 times; if it is an ensemble test for a single cycle, the test will call the forecast model as many times as the number of ensemble members).  

## TESTS CONDUCTED: 
The script `run_WE2E_tests.sh` was called that in turn calls `get_WE2Etest_names_subdirs_descs.sh`.  This created a new CSV file that contained the new fields (columns).  The CSV file was imported into Google Sheets (using "|" as the field/column separator) and looked correct.

## DOCUMENTATION:
The documentation is for the most part already within the `get_WE2Etest_names_subdirs_descs.sh`.  This PR slightly modifies that documentation to update it.

* Update directory structure of NCO mode (#743)

* update vertical structure of NCO mode

* update sample script for nco

* Fix typo on write component of new RRFS CONUS

* Default CCPP physics option is FV3_GFS_v16 (#746)

* Updated the default CCPP physics option to FV3_GFS_v16

* Updated the default CCPP physics option to FV3_GFS_v16 in config_defaults.sh

Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>

* Adds an alternative python workflow generation path (#698)

* Workflow in python starting to work.

* Use new python_utils package structure.

* Some bug fixes.

* Use uppercase TRUE/FALSE in var_dfns

* Use config.sh by default.

* Minor bug fixes.

* Remove config.yaml

* Update to the latest develop

* Remove quotes from numbers in predef grid.

* Minor bug fix.

* Move validity checker to the bottom of setup

* Add more unit tests.

* Update with python_utils changes.

* Update to latest develop additions (Need to re-run regression test)

* Use set_namelist and fill_jinja_template as python functions.

* Replace sed regex searches with python re.

* Use python realpath.

* Construct settings as dictionary before passing to fill_jinja and set_namelist

* Use yaml for setting predefined grid parameters.

* Use xml parser for ccpp phys suite definition file.

* Remove more run_command calls.

* Simplify some func argument processing.

* Move different config format parsers to same file.

* Use os.path.join for the sake of macosx

* Remove remaining func argument processing via os.environ.

* Minor bug fix in set_extrn_mdl_params.sh

* Add suite defn in test_data.

* Minor fixes on unittest on jet.

* Simplify boolean condition checks.

* Include old in renaming of old directories

* Fix conflicting yaml !join tag for paths and strings.

* Bug fix with setting sfcperst dict.

* Imitate "readlink -m" with os.path.realpath instead of os.readlink

* Don't use /tmp as that is shared by multiple users.

* Bug fix with cron line, maintain quotes around TRUE/FALSE.

* Update to latest develop (untested)

* Bug fix with existing cron line and quotes.

* Bug fix with case-sensitive MACHINE name, and empty EXPT_DIR.

* Update to latest develop

* More updates.

* Bug fix thanks to @willmayfield! Check both starting/ending
characters are brackets for shell variable to be considered an array.

* Make empty EXPT_BASEDIR workable.

* Update to latest develop

* Update in predef grid.

* Check f90nml as well.

Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>

* Fix typo and crontab issue on wcoss dell in workflow python scripts (#750)

* Fix typo and failure on wcoss

* fix new line issue on wcoss dell

* remove capture_output

* Get USER from environment

Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>

* Add new WE2E configs (#748)

## DESCRIPTION OF CHANGES: 
Added two new WE2E config files for the Sub-CONUS Indianapolis domain to support the upcoming SRW release. 

In addition, modified the external data used in the `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` to match more common datasets used in the WE2E testing process. 

## TESTS CONDUCTED: 
Successfully ran the new WE2E tests (`config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR.sh`, `config.SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.sh`) and `config.specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS.sh` on NOAA Parallel Works AWS instance.

## DEPENDENCIES:
None.

## DOCUMENTATION:
No documentation changes are required.

* Added a fixed WoF grid and the python tool to determine the write component parameters (#733)

* Added a fixed WoF grid and the python tool to determine the write component parameters

* Update set_predef_grid_params.sh

* Renamed file as recommended and removed unused lines

* Modified comment

Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>

* Replace env with modulefiles in scripts (#752)

* change env to mod

* update we2e script

* WE2E script improvements for usability (#745)

## DESCRIPTION OF CHANGES: 
* Modifications to `run_WE2E_tests.sh`:
  * Add examples to help/usage statement
* Modifications to `check_expts_status.sh`:
  * Add arguments list that can be processed by `process_args`
  * Add new optional arguments:  `num_log_lines`, `verbose`
  * Include a help/usage message

## TESTS CONDUCTED:
* Ran `run_WE2E_tests.sh --help` from the command line and got the expected help message.
* Ran `check_expts_status.sh --help` from the command line and got the expected help message.
* Used `run_WE2E_tests.sh` to run a set of 2 WE2E tests -- works as expected.
* Used `check_expts_status` to check on the status of the 2 tests run above and got the expected status message.
 
## DEPENDENCIES:
PR #[241](ufs-community/ufs-srweather-app#241)

## DOCUMENTATION:
A lot of this PR is documentation in the scripts.  There is an accompanying documentation PR #[241](ufs-community/ufs-srweather-app#241) into ufs-srweather-app.

* Standardize static data across Tier-1 platforms; fix and improve IC and LBC data retrieval (#744)

* Bug fixes (grid size + suppress screen output from module load) (#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR #725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community/ufs-srweather-app#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).

* Update default SPP ISEED array in config_defaults.sh to use unique values (#759)

* Modify RRFS North America 3- and 13-km domain configuration and WE2E test.

* Modify default ISEED values for SPP

* Fix grid in WE2E test

* Update workflow python scripts (#760)

* update python scripts

* Change output file name of run_post to meet NCO standards (#758)

* change output file name

* change variable name

* update python script

* remove duplicates

* add a check for empty variables

* move variable to common area

* clean up unnecessary comments

* update scripts

* remove duplicate

* update python scripts

* fix user-staged dir path issue in python script

* Add POST_OUTPUT_DOMAIN_NAME to WE2E tests for new grids (#763)

* Add new var to we2e tests for new grids

* rename we2e tests for custom grid

* remove unnecessary $

* Modifications to `CODEOWNERS` file (#757)

* Add @gspetro-NOAA, @natalie-perlin, and @EdwardSnyder-NOAA to CODEOWNERS so they are notified of all PRs and can review them.

* Remove duplicates in CODEOWNERS; remove users who will no longer be working with the repo.

* Adding a python utility for summarizing compute. (#769)

Adds a utility that summarizes Rocoto database computational usage information.

* Add github actions for python unittests. (#747)

* Add github actions for python unittests.

* Include all python script in ush

* Skip defining QUILTING params when it is set to False

* Update py_workflow

* Update unittest for set_extrn_mdl_params.

* Updates from develop.

Co-authored-by: Daniel Shawul <dshawul@yahoo.com>

* Update sample script for NCO mode (#771)

* update config.nco.sh

* Add comment

* Feature/noaacloud (#767)

* updates for noaacloud

* working version

* fixes for noaacloud

* added extra modules for post

* removed cheyenne-specific crontab editing section (#773)

* Pin down hera miniconda3 module file version. (#770)

Pin down the version of miniconda3 on Hera, and do not append to the module path.

* update staged data dir (#774)

* Changed to smaller SUBCONUS_Ind_3km domain

Co-authored-by: Mark Potts <33099090+mark-a-potts@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Daniel Abdi <dabdi@Orion-login-2.HPC.MsState.Edu>
Co-authored-by: Daniel Abdi <daniel.abdi@noaa.gov>
Co-authored-by: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com>
Co-authored-by: JeffBeck-NOAA <55201531+JeffBeck-NOAA@users.noreply.github.com>
Co-authored-by: WYH@MBP <yunheng.wang@4374-ywang.winstorm.nssl>
Co-authored-by: Michael Kavulich <kavulich@ucar.edu>
Co-authored-by: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com>
Co-authored-by: Daniel Shawul <dshawul@yahoo.com>
mark-a-potts added a commit that referenced this pull request Jun 12, 2022
* Fix artifact creation for branches with / in name (#275)

The slash in branch names such as release/.* and feature/.* is
interpreted as a directory separator. This change replaces the "/"
character with a "_".

* Include documentation for building/running SRW App on Mac (#240)

* updated docs

* added git submodule

* fix formatting

* added new submodule commits

* fixed ref links

* finished Intro

* finish Components & Intro edits

* edited Rocoto workflow section of Quickstart

* added minor hpc submodule commits

* Updates to Rocoto Workflow in Quick Start

* add to HPC-stack intro

* submodule updates

* added submodule docs edits

* hpc-stack updates & formatting fixes

* hpc-stack intro edits

* bibtex attempted fix

* add hpc-stack module edits

* update sphinxcontrib version

* add .readthedocs.yaml file

* update .readthedocs.yaml file

* update .readthedocs.yaml file

* update conf.py

* updates .readthedocs.yaml with submodules

* updates .readthedocs.yaml with submodules

* submodule updates

* submodule updates

* minor Intro edits

* minor Intro edits

* minor Intro edits

* submodule updates

* fixed typos in QS

* QS updates

* QS updates

* QS updates

* updates to InputOutput and QS

* fix I/O doc typos

* pull updates to hpc-stack docs

* pull updates to hpc-stack docs

* fix table wrapping

* updates to QS for cloud

* fix QS export statements

* fix QS export statements

* QS edits on bind, config

* add bullet points to notes

* running without rocoto

* add HPC-Stack submodule w/docs

* split QS into container/non-container approaches

* added filepath changes for running in container on Orion, et al.

* edits to overview and container QS

* moved CodeReposAndDirs.rst info to the Introduction & deleted file

* continued edits to SRWAppOverview

* combine overview w/non-container docs

* finish merging non-container guide & SRWOverview, rename/remove files, update FAQ

* minor edits for Intro & QS

* updates to BuildRun doc through 3.8.1

* edits to Build/Run and Components

* remove .gitignore

* fix Ch 3 title, 4 supported platform levels note

* fix typos, add term links

* other minor fixes/suggestions implemented

* updated Intro based on feedback; changed SRW to SRW App throughout

* update comment to Intro citation

* add user-defined vertical levels to future work

* Add instructions for srw_common module load

* fix typo

* update Intro & BuildRunSRW based on Mark's feedback

* minor intro updates

* 1st round of jwolff's edits

* 2nd round of jwolff updates

* update QS intro

* fix minor physics details

* update citation and physics suite name

* add compute node allocation info to QS

* add authoritative hpc-stack docs to Intro

* create MacOS install/build instructions

* add MacOS Build/Run instructions

* fix MacOS Build/Run details

* add MacOS info directly to Build/Run SRW chapter

* minor details

* minor edits

* update Include-HPCInstall with mac installation docs

* add note re: Terminal.app & bash shell in MacOS section

* remove MacInstall file-contents added to BuildRunSRW

* update hpc-stack submodule to include mac installation info

* add MacOS config details

* add MacOS config & run details

* minor MacOS note

* mention need to verify software library version #'s

* update hpc-stack-mod

* align MacDetails section with PR #238 info

* remove gsed & alter related commands

* update hpc-stack submodule

* typos

* switch from env to module load

Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>

* Update BuildRunSRW.rst

* update hpc-stack module docs & MacOS config.sh

* update machine file instructions

* updates to BuildRun chapter

* fix typo

Co-authored-by: gspetro <gillian.s.petro@gmail.com>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>

* Update documentation for CSV file containing WE2E test info (#278)

* Edits to documentation to match latest in code.

* Edits to documentation to match latest in code.

* Minor changes to documentation.

* Port SRW App to WCOSS2 (#270)

* port to wcoss2

* update scripts

* ensure platform name from variance

* update scripts

* remove wcoss2 from lmod-setup

* Fix --ccpp option in devbuild.sh (#280)

* fix ccpp issue

* update script

* Print usage if machine name is not passed to `lmod-setup.sh/csh`. (#262)

* Print usage() message if machine name is not passed.

* Bug fix with macos modulefiles.

* Back to original hashes (#287)

* Updated the Introduction, build for MacOS (#281)

* Updated the Introduction, build for MacOS

* Update Introduction.rst

Some comments removed

* Update Introduction.rst

Updated data storage requirements for the SRW

* Update documentation for python plotting scripts (#289)

## DESCRIPTION OF CHANGES: 
This PR updates the RST documentation files to accompany changes in PR #[783](ufs-community/regional_workflow#783) in `regional_workflow`.

## DEPENDENCIES:
PR #[783](ufs-community/regional_workflow#783) in `regional_workflow`.

## ISSUE (optional): 
Partially resolves issue #[781](ufs-community/regional_workflow#781) (still need to introduce tests in `regional_workflow` that run the plotting scripts).

## CONTRIBUTORS: 
@mark-a-potts

* Updating for testing with updated ufs weather model

* added version to wflow for jet

* switched to epic repo for testing

* Updates to modules for cheyenne

* updated module path

* changed hash to branch for testing

* updated to use ufs-community version of WM

* Pointing to PR-branch of regional_workflow for testing

* Updating module files for cheyenne for testing

* Added png explicitly under gaea.

* Update build_cheyenne_intel

Corrected loading system modules (load intel/2022.1, mpt/2.25), updated package versions if newer modules are built.

* Updated pointer to model for testing

* fixed repos

* updated to point at my regional_workflow branch

* updated regional_workflow to release/public-v2

Co-authored-by: Jesse McFarland <jesse@mcfarland.sh>
Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>
Co-authored-by: gspetro <gillian.s.petro@gmail.com>
Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com>
Co-authored-by: gsketefian <31046882+gsketefian@users.noreply.github.com>
Co-authored-by: Chan-Hoo.Jeon-NOAA <60152248+chan-hoo@users.noreply.github.com>
Co-authored-by: danielabdi-noaa <52012304+danielabdi-noaa@users.noreply.github.com>
Co-authored-by: Natalie Perlin <68030316+natalie-perlin@users.noreply.github.com>
Co-authored-by: Mark Potts <mpotts@redlineperf.com>
mkavulich pushed a commit to mkavulich/ufs-srweather-app that referenced this pull request Aug 26, 2022
…community#756)

## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR ufs-community#725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](ufs-community#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).
mkavulich pushed a commit that referenced this pull request Sep 8, 2022
## DESCRIPTION OF CHANGES: 
1) Adjust y-direction size of write-component grid of `SUBCONUS_Ind_3km` predefined grid from 195 to 197 (this was just an oversight in PR #725 ).
2) Redirect output of module load in launch script (`launch_FV3LAM_wflow.sh`) to `/dev/null` to avoid unwanted screen output (which was introduced in PR #[238](#238) in ufs-srweather-app and is about how to load the `regional_workflow` environment and is not relevant in this context).

## TESTS CONDUCTED: 
1) Plotted the `SUBCONUS_Ind_3km` grid to ensure it has correct size (it does).
2) Manually ran `launch_FV3LAM_wflow.sh` from the command line to verify that screen output is suppressed (it is).
SamuelTrahanNOAA pushed a commit to SamuelTrahanNOAA/ufs-srweather-app that referenced this pull request Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants