Add DART input.nml to template in WRF Tutorial #464

braczka · 2023-03-13T17:16:36Z

Description:

Simple 1 line documentation change that repairs broken link to WRF DART tutorial input.nml.template file. Resolves problems during gen_retro_icbc.csh and init_ensemble_var.csh steps.

Fixes issue

Related to github issue #295 .

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

This fixes broken link to WRF tutorial input.nml.template file during gen_retro_icbc.csh and init_ensemble_var.csh steps

hkershaw-brown

Hi Brett,

This fix seems to fix the problem of the missing input.nml. I'm not sure why there is all this scripting but these 3 input files have to be copied by hand. There is definitely still a lot that could be done to make the tutorial better.

init_ensemble_var.csh has a couple of places where input.nml.template is copied or linked:

DART/models/wrf/shell_scripts/init_ensemble_var.csh

Line 53 in e17959d

${COPY} ${TEMPLATE_DIR}/input.nml.template input.nml

DART/models/wrf/shell_scripts/init_ensemble_var.csh

Line 68 in e17959d

    
           ${LINK} ${TEMPLATE_DIR}/input.nml.template ${RUN_DIR}/advance_temp${n}/input.nml

and this note:

DART/models/wrf/shell_scripts/init_ensemble_var.csh

Lines 64 to 65 in e17959d

    
           # TJH why does the run_dir/*/input.nml come from the template_dir and not the rundir? 
        
           # TJH furthermore, template_dir/input.nml.template and rundir/input.nml are identical. SIMPLIFY.

I'm not sure if these scripts are used outside the tutorial or not, so I can't comment on whether these copies/links are expected to work.

Cheers,
Helen

hkershaw-brown · 2023-03-14T12:37:28Z

models/wrf/tutorial/README.rst

@@ -236,6 +236,7 @@ bash  ``export BASE_DIR=<path_to_your_working_directory>``

       cp $DART_DIR/models/wrf/tutorial/template/namelist.input.meso   $BASE_DIR/template/.
       cp $DART_DIR/models/wrf/tutorial/template/namelist.wps.template $BASE_DIR/template/.
+       cp $DART_DIR/models/wrf/tutorial/template/input.nml.template    $BASE_DIR/template/.


input.nml.template is a DART namelist. The comment above talks about WRF namelists. It would be good to be clear about namelists for running WRF vs namelist for running DART executables.

Latest commit 911dc4e, addresses this.

Commits 08aba5e and 63a5748 eliminate the need for this line in WRF Tutorial instructions.

braczka · 2023-03-14T14:12:43Z

@hkershaw-brown This is just a simple band-aid fix that gets the tutorial to run further. I am not suggesting things are ideal -- definitely is room for a refactor. I am still trying to actually complete the tutorial in its entirety just to get more perspective. I will comment more later today -- Cheyenne has been quite overloaded lately so it has been difficult for me to get even short jobs through. In general, the integration of multiple jobs of WRF interacting tightly with shell scripts that arrange the output files makes troubleshooting very difficult. These are all in one scripts designed for an advanced user. Feel like the tutorial could be more step-wise so user can check that each step was completed correctly before shell scripts start looking for files that may/ may not be there, then fails. The pure number of scripts, pbs jobs, log files, diagnostic files has to be overwhelming to a new user. It seems the default tutorial could be brought down from 50 to 5 ensemble members, and not clear to me why ./gen_retroc_icbc.sh needs to run from 201742700 through 201743000 to generate icbc, when the assimilation runs from only 201742700 to 201742712. Could be streamlined, without effecting functionality, I would think.

hkershaw-brown · 2023-03-14T14:16:46Z

totally agree! Great work smashing through the tutorial!

braczka · 2023-03-16T14:16:02Z

The excessive linking as @hkershaw-brown points out above makes it very difficult/confusing to make meaningful changes to the DART input.nml file since it is in many different places.

${TEMPLATE_DIR}/input.nml.template
${RUN_DIR}/advance_temp${n}/input.nml
${RUN_DIR}/input.nml
${WORK_DIR}/input.nml

Also settings may be updated during changes to param.csh and running the ./setup.csh. This is especially annoying when updating the ensemble member number for the tutorial -- which is an especially useful update during troubleshooting. Looking to streamline this.

Redirecting ls of linked files provides bad syntax to diagnostic_obs.csh

There are several template model_mod.f90 in models/template This fix picks a particular model_mod.f90 for each developer test. Fixes #408 developer test quickbuilds Added LOCATION back into developer test quickbuilds

If first time through driver.csh start_member and done_member diagnostics could exist causing errors with file transfers

braczka · 2023-04-03T18:36:37Z

models/wrf/shell_scripts/driver.csh

I had major trouble running tutorial without this check to make sure there were no pre-existing start_member and done_member diagnostic files that existed during first run through of driver.csh. There was a tendency for these diagnostic files to exist already from prior execution of init_ensemble_var.csh. This led to file transfers to begin within driver.csh before the WRF integrations were complete.

braczka · 2023-04-03T18:39:17Z

models/wrf/tutorial/README.rst

+running the *driver.csh* script in Step 5, using a smaller ensemble 
+(e.g. < 20 members) can lead to spurious updates during the analysis step,
+causing the WRF simulation to fail. 
+


The tutorial can run OK with as little as 20 ensemble members. Wanted to avoid confusion with new users that using less than recommended 50 members, although good for quick troubleshooting, introduces problems if not done carefully.

braczka · 2023-04-03T18:40:52Z

models/wrf/tutorial/README.rst

@@ -457,7 +459,7 @@ changes:
 +====================+======================================+=========================================================================================================================================================================================================================================================+
 | driver.csh         | datefnl = 2017042712                 | Change to the final target date; here the final date is already set correctly for this tutorial.                                                                                                                                                        |
 +--------------------+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| gen_retro_icbc.csh | datefnl = 2017043000                 | This is the final date to create WRF initial/boundary conditions for. This is set to the last date that files are included in the tutorial.                                                                                                             |
+| gen_retro_icbc.csh | datefnl = 2017042712                 | Set to the final target date of the tutorial.  However, it is possible (not necessary) to create WRF initial/boundary conditions to 2017043000. This is the latest date that files are included in the tutorial.                                        |


No need to run gen_retro_icbc.csh all the way to 2017043000 when tutorial only runs to 2017042712. This helps streamline troubleshooting.

Nice catch here

…te/threed_model_mod.f90 for readability

Eliminating need to transfer DART input.nml to template directory within tutorial. Support editing of rundir input.nml only.

WRF shell scripts now generate links to DART input.nml within run dir, not template directory. Corrects missing updates from setup.csh.

braczka · 2023-04-04T19:52:09Z

models/wrf/shell_scripts/setup.csh

@@ -89,7 +89,7 @@ ${COPY} ${SHELL_SCRIPTS_DIR}/add_bank_perts.ncl    ${RUN_DIR} || exit 8
 ${COPY} ${SHELL_SCRIPTS_DIR}/new_advance_model.csh ${RUN_DIR} || exit 9

 # Edit input.nml.template so that its ens_size is set to the same value as $NUM_ENS in param.csh
-sed "s/ens_size.*/ens_size                 =$NUM_ENS,/g" ${DART_DIR}/models/wrf/tutorial/template/input.nml.template > ${RUN_DIR}/input.nml || exit 8
+sed "s/ens_size.*/ens_size                 =  $NUM_ENS,/g" ${DART_DIR}/models/wrf/tutorial/template/input.nml.template > ${RUN_DIR}/input.nml || exit 8


The setup.csh script was designed to apply ensemble member updates to the run directory input.nml, thus all WRF shell scripting (e.g. gen_retro_icbc.csh, init_ensemble_var.csh, new_advance_model.csh etc) should link or copy the run directory input.nml, not the input.nml located in the template directory. This is now consistent with the WRF tutorial instructions.

braczka · 2023-04-04T19:55:45Z

models/wrf/shell_scripts/driver.csh

+                ${RUN_DIR}/done_member_*
+   endif
+
+


If first time through the driver.csh, there could be lingering start_member or done_member files from the prior init_ensemble_var.csh run. Need to get rid of these or the driver.csh script starts transferring files that do not exist because WRF integration is not yet completed.

@mgharamti I know you are super busy, but as a regular WRF user, would be worth for you to look this over, especially commit 63a5748 . This commit changes some of the wrf shell scripting which extends outside the tutorial. I think it is OK and just a more simple way of linking the DART input.nml information to the WRF case. No rush on this. I think this is about as much as I want to change for foreseeable future. Thanks @mjs2369 for looking at this too. If everything looks OK please include in next DART release.

@braczka I've looked over these changes and I agree that the WRF shell scripting needs to link ${RUN_DIR}/input.nml as opposed to ${TEMPLATE_DIR}/input.nml.template . I think this is ready to merge.

But there's no rush on this release (the bug-fix that Helen implemented is developer code only), so I also think that we should wait for @mgharamti to take a look if he has the time. If not, we can just go ahead and release sometime next week.

mjs2369 · 2023-04-11T16:59:50Z

@braczka I've been reading through the WRF tutorial, and I noticed that in the step before where the user runs gen_retro_icbc.csh and init_ensemble_var.csh, here , the user has to edit some of the variables' values in driver.csh, gen_retro_icbc.csh, and gen_pert_bank.csh. I'm just wondering as to why we have the users make these changes and we don't go ahead and make these changes in main?

This is NOT relevant for the release, but just a curiosity question.

mjs2369

Approved.

Bundle this PR with #466 for release.

braczka · 2023-04-11T17:57:30Z

@mjs2369 Technically the only variable that needs to be updated in the tutorial is the path to paramfile because this is unique for each user's environment. The datefnl variable is now hardwired into the default scripting, and should work as is, however, the user could run the tutorial for longer than the default 2017042712, all the way to 2017043000, if they wish. The changes for gen_pert_bank.csh are just there for reference. The tutorial already includes a perturbation bank.

Add DART input.nml to template in WRF Tutorial

2bab64d

This fixes broken link to WRF tutorial input.nml.template file during gen_retro_icbc.csh and init_ensemble_var.csh steps

braczka added Documentation Improvements or additions to documentation Minor Minor functionality, non-critical data; easy fix labels Mar 13, 2023

braczka requested a review from hkershaw-brown March 13, 2023 17:16

hkershaw-brown reviewed Mar 14, 2023

View reviewed changes

braczka and others added 4 commits March 16, 2023 10:19

Updated obs_seq.final flist for WRF_DART Tutorial

aa686ac

Redirecting ls of linked files provides bad syntax to diagnostic_obs.csh

bug-fix: developer test quickbuilds

92ce866

There are several template model_mod.f90 in models/template This fix picks a particular model_mod.f90 for each developer test. Fixes #408 developer test quickbuilds Added LOCATION back into developer test quickbuilds

Remove WRF diagnostics before next integration

859bf4d

If first time through driver.csh start_member and done_member diagnostics could exist causing errors with file transfers

WRF-DART Tutorial documentation edits

911dc4e

braczka commented Apr 3, 2023

View reviewed changes

mjs2369 self-requested a review April 4, 2023 15:28

mjs2369 and others added 4 commits April 4, 2023 09:46

Changing EXTRA=/models/template/model_mod.f90 to EXTRA=/models/templa…

c354a65

…te/threed_model_mod.f90 for readability

Fixing EXTRA var in random_seq as well

07170e4

WRF tutorial eliminates DART template input.nml

08aba5e

Eliminating need to transfer DART input.nml to template directory within tutorial. Support editing of rundir input.nml only.

WRF shell scripts switch to rundir input.nml

63a5748

WRF shell scripts now generate links to DART input.nml within run dir, not template directory. Corrects missing updates from setup.csh.

braczka commented Apr 4, 2023

View reviewed changes

braczka requested a review from mgharamti April 4, 2023 20:00

mjs2369 added 2 commits April 11, 2023 11:16

Merge branch 'devel-tests-quickbuild' into WRF_tutorial

2f6ed97

bump conf.py and update CHANGELOG

282c2ca

mjs2369 approved these changes Apr 11, 2023

View reviewed changes

mjs2369 merged commit e9bb19a into main Apr 11, 2023

mjs2369 deleted the WRF_tutorial branch April 11, 2023 17:33

mjs2369 mentioned this pull request Apr 11, 2023

bug-fix: developer test quickbuilds #466

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DART input.nml to template in WRF Tutorial #464

Add DART input.nml to template in WRF Tutorial #464

braczka commented Mar 13, 2023 •

edited by mjs2369

Loading

hkershaw-brown left a comment •

edited

Loading

hkershaw-brown Mar 14, 2023

braczka Apr 3, 2023

braczka Apr 4, 2023

braczka commented Mar 14, 2023

hkershaw-brown commented Mar 14, 2023

braczka commented Mar 16, 2023

braczka Apr 3, 2023

braczka Apr 3, 2023

braczka Apr 3, 2023

mjs2369 Apr 5, 2023

braczka Apr 4, 2023

braczka Apr 4, 2023

braczka Apr 4, 2023

mjs2369 Apr 5, 2023

mjs2369 commented Apr 11, 2023 •

edited

Loading

mjs2369 left a comment

braczka commented Apr 11, 2023

	# TJH why does the run_dir/*/input.nml come from the template_dir and not the rundir?
	# TJH furthermore, template_dir/input.nml.template and rundir/input.nml are identical. SIMPLIFY.

Add DART input.nml to template in WRF Tutorial #464

Add DART input.nml to template in WRF Tutorial #464

Conversation

braczka commented Mar 13, 2023 • edited by mjs2369 Loading

Description:

Fixes issue

Types of changes

Documentation changes needed?

Checklist for merging

Checklist for release

hkershaw-brown left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braczka commented Mar 14, 2023

hkershaw-brown commented Mar 14, 2023

braczka commented Mar 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjs2369 commented Apr 11, 2023 • edited Loading

mjs2369 left a comment

Choose a reason for hiding this comment

braczka commented Apr 11, 2023

braczka commented Mar 13, 2023 •

edited by mjs2369

Loading

hkershaw-brown left a comment •

edited

Loading

mjs2369 commented Apr 11, 2023 •

edited

Loading