NoahMP as a submodule in WRF #1653
Conversation
TYPE: enhancement KEYWORDS: NoahMP, submodule SOURCE: Cenlin He (RAL/NCAR), internal DESCRIPTION OF CHANGES: This is an initial effort to include an external stand-alone repository into the WRF system using git "submodules". Here is the recipe, should we be interested in doing this again. Also this serves as a "perhaps not that way again" warning. Time will tell. The code does not build after this initial step. We are just preparing the soil. 1. Remove the existing WRF-versions of the NoahMP source code ``` cd WRF/phys git rm module_sf_noahmp_glacier.F git rm module_sf_noahmp_groundwater.F git rm module_sf_noahmpdrv.F git rm module_sf_noahmplsm.F ``` ... and the single table over which NoahMP has dominion ``` cd WRF/run git rm MPTABLE.TBL ``` 2. Add the NoahMP repository as a submodule ``` cd WRF/phys git submodule add https://github.com/NCAR/noahmp noahmp ``` This generates the new file WRF/.gitmodules 3. Along with the rest of the staged files, include the SHA1 of the NoahMP repository as part of the what the WRF model keeps permanently. ``` cd WRF/phys/noahmp git checkout release-v4.4-WRF cd .. git add noahmp ``` Right now, this is SHA1 aed82e17e95b5 from NoahMP NCAR/noahmp@aed82e1 LIST OF MODIFIED FILES: A .gitmodules D phys/module_sf_noahmp_glacier.F D phys/module_sf_noahmp_groundwater.F D phys/module_sf_noahmpdrv.F D phys/module_sf_noahmplsm.F A phys/noahmp D run/MPTABLE.TBL The code builds after this step, which includes the mods to the build system (the variously impacted Makefiles). LIST OF MODIFIED FILES: M Makefile M clean M phys/Makefile In the top-level WRF Makefile, the `physics` target now includes a pre-step to first build the NoahMP submodule files. If the NoahMP source files exist (the *.F files) nothing happens. If the files do not exist, then the rule is `git submodule update --init --recursive` to bring in the NoahMP repository, and then manually link the files into the original WRF locations. If the source files did not exist, assume that the NoahMP table files were also missing, so put those in the run directory. Since these files are not really part of the WRF repo anymore, we remove them when asked for a super-duper cleaning: `clean -aa`. The NoahMP files are linked into the physics directory. This emulates the existing paradigm of putting all of the physics source into a single location. This minimizes the infrastructure modifications, and becomes almost transparent to a casual user. Care will need to be taken so that no files associated with NoahMP are accidentally re-incorporated back into WRF. TESTS CONDUCTED: 1. The code builds locally. When there are no NoahMP files, they are available with a git submodule command, and subsequent linking. 2. With print statements in the Makefiles, when existing NoahMP files are detected, the submodule step is bypassed. 3. Jenkins - all tests are pass. RELEASE NOTE: The NoahMP external repository is now a submodule for WRF.
In the physics Makefile, before any files source files are processed, check to see if the NoahMP source files exist (the *.F files). If so, then proceed as normally (they must already be linked in). If the files do not exist, then `git submodule update --init --recursive` to bring in the NoahMP repository, and then manually link the files into the original WRF locations. If the source files did not exist, assume that the NoahMP table file (MPTABLE.TBL) is also missing, so put that file in the run directory. Since these files are not really part of the WRF repo anymore, we remove them when asked for a super-duper cleaning: `clean -aa`. An "if" test in the phys/Makefile detects if the NoahMP source files exist. If not, then the git submodule command is used. The important item is that the top-level Makefile has a submodules target that is processed for each build. M Makefile M clean M phys/Makefile
|
@davegill What does 'git checkout release-v4.4-WRF' do in step 3? I also tried to do 'clean -aa' from top WRF directory, it seems it does not copy configure.wrf to configure.wrf.backup. Also somewhere we need instructions for those who gets the tar file from us (or github), and how they may get the noahmp code if they don't use git, and/or don't have internet when they compile. |
This is the "version" of the NoahMP code that looked like what we wanted. There was also the master branch that I could have chosen. Basically, we issue the submodule, cd into the NoahMP directory, checkout the branch we want, then in the WRF repo we "git add noahmp".
The "-aa" is a clean everything possible option. So, "clean -a" would leave a configure.wrf.backup, but the "-aa" option would not. This seems OK behavior.
The only restriction is "no access to github". I think the best that we could do would be to trap this event, and report that the build cannot proceed without the NoahMP directory. |
Make the EPS args to MP driver optional, then all is well. M phys/module_microphysics_driver.F
|
Using this PR instead of NoahMP as a submodule in WRF #1646 |
|
@davegill @weiwangncar @dudhia Did we decide to go with this solution for the v4.4 release (i.e., only use MPTABLE.TBL from NoahMP github repo)? If so, I am fine with the current solution for now. We could discuss more in the future. Just want to make sure that for the other two tables, SOILPARM.TBL and GENPARM.TBL in this current WRF repo are the same as those in the NoahMP repo (i.e., WRF is already using the updated SOILPARM.TBL and GENPARM.TBL for v4.4 release). |
|
@cenlinhe @weiwangncar @dudhia |
|
Sounds good! |
davegill
changed the title
modified: Makefile
modified: Makefile
|
@weiwangncar @dudhia |
|
@CenLin @weiwangncar @dudhia |
TYPE: enhancement KEYWORDS: NoahMP, submodule SOURCE: Cenlin He (RAL/NCAR), internal DESCRIPTION OF CHANGES: This is an initial effort to include an external stand-alone repository into the WRF system using git "submodules". Here is the recipe, should we be interested in doing this again. The code does not build after this initial step, as we are removing mandatory source files that are required for the build of the physics directory. 1. Remove the existing WRF-versions of the NoahMP source code ``` cd WRF/phys git rm module_sf_noahmp_glacier.F git rm module_sf_noahmp_groundwater.F git rm module_sf_noahmpdrv.F git rm module_sf_noahmplsm.F ``` ... and the single table over which NoahMP has dominion ``` cd WRF/run git rm MPTABLE.TBL ``` 2. Add the NoahMP repository as a submodule ``` cd WRF/phys git submodule add https://github.com/NCAR/noahmp noahmp ``` This generates the new file WRF/.gitmodules 3. Along with the rest of the staged files, include the SHA1 of the NoahMP repository as part of the what the WRF model keeps permanently. ``` cd WRF/phys/noahmp git checkout release-v4.4-WRF cd .. git add noahmp ``` Right now, this is SHA1 aed82e17e95b5 from NoahMP NCAR/noahmp@aed82e1 LIST OF MODIFIED FILES: A .gitmodules D phys/module_sf_noahmp_glacier.F D phys/module_sf_noahmp_groundwater.F D phys/module_sf_noahmpdrv.F D phys/module_sf_noahmplsm.F A phys/noahmp D run/MPTABLE.TBL The code builds after this step, which includes the mods to the build system (the variously impacted Makefiles). LIST OF MODIFIED FILES: M Makefile M clean M phys/Makefile In the top-level WRF Makefile, the `physics` target now includes a pre-step to first build the NoahMP submodule files. If the NoahMP source files exist (the *.F files). * If so, then proceed as normally (they must already be linked in). * If the files do not exist, then `git submodule update --init --recursive` to bring in the NoahMP repository, and then manually link the files into the original WRF locations. Also, if the source files did not exist, assume that the NoahMP table file (MPTABLE.TBL) is also missing, so put that file in the run directory. Since these files are not really part of the WRF repo anymore, we remove them when asked for a super-duper cleaning: `clean -aa`. Please note that usage of the `clean -aa` script removes the backup configure.wrf file. An "if" test in the phys/Makefile detects if the NoahMP source files exist. If not, then the git submodule command is used. The important item is that the top-level Makefile has a submodules target that is processed for each build. The NoahMP files are linked into the physics directory. This emulates the existing paradigm of putting all of the physics source into a single location. This minimizes the infrastructure modifications, and becomes almost transparent to a casual user. Care will need to be taken so that no files associated with NoahMP are accidentally re-incorporated back into WRF. Considerations: The introduction of an external repository is a complication. We choose to avoid modifying the WRF build system, meaning that the entire physics directory must be WHOLE for a successful build. This is handled by copying files from the NoahMP repository directory location to the WRF repository directory location. The infrastructure has been put in place in the WRF build to copy the files to the expected WRF locations. Therefore, starting from a clean clone and without a preceding build, a user will not find the NoahMP source files in the physics directory. The NoahMP repository could be manually included as a submodule with the following command: ``` cd WRF git submodule update --init --recursive ``` However, again, until a build occurs, the files are not moved from the NoahMP repository directory structure into the proper and expected locations in the WRF directory structure. TESTS CONDUCTED: 1. The code builds locally. When there are no NoahMP files, they are available with a git submodule command, and subsequent linking. 2. With print statements in the Makefiles, when existing NoahMP files are detected, the submodule step is bypassed. 3. If the required files are not available, an error condition is raised, and the compilation stops: ``` ------------------------------------------------------------------------------ Error Error Error NoahMP submodule files not populating WRF directories ------------------------------------------------------------------------------ ``` 4. When the NoahMP files are available, a message is printed to the build log: ``` ------------------------------------------------------------------------------ NoahMP submodule files populating WRF directories ------------------------------------------------------------------------------ ``` 5. Jenkins - all tests are pass. RELEASE NOTE: The NoahMP external repository is now a submodule for WRF.
TYPE: enhancement
KEYWORDS: NoahMP, submodule
SOURCE: Cenlin He (RAL/NCAR), internal
DESCRIPTION OF CHANGES:
This is an initial effort to include an external stand-alone repository into the
WRF system using git "submodules".
Here is the recipe, should we be interested in doing this again.
The code does not build after this initial step, as we are removing mandatory source files that are
required for the build of the physics directory.
... and the single table over which NoahMP has dominion
This generates the new file WRF/.gitmodules
as part of the what the WRF model keeps permanently.
Right now, this is SHA1 aed82e17e95b5 from NoahMP
NCAR/noahmp@aed82e1
LIST OF MODIFIED FILES:
A .gitmodules
D phys/module_sf_noahmp_glacier.F
D phys/module_sf_noahmp_groundwater.F
D phys/module_sf_noahmpdrv.F
D phys/module_sf_noahmplsm.F
A phys/noahmp
D run/MPTABLE.TBL
The code builds after this step, which includes the mods to the build system (the variously impacted Makefiles).
LIST OF MODIFIED FILES:
M Makefile
M clean
M phys/Makefile
In the top-level WRF Makefile, the
physicstarget now includes a pre-stepto first build the NoahMP submodule files. If the NoahMP source files exist
(the *.F files).
git submodule update --init --recursiveto bring inthe NoahMP repository, and then manually link the files into the
original WRF locations. Also, if the source files did not exist,
assume that the NoahMP table file (MPTABLE.TBL) is also missing,
so put that file in the run directory.
Since these files are not really part of the WRF repo anymore, we remove
them when asked for a super-duper cleaning:
clean -aa. Please note thatusage of the
clean -aascript removes the backup configure.wrf file.An "if" test in the phys/Makefile detects if the NoahMP source files
exist. If not, then the git submodule command is used. The important
item is that the top-level Makefile has a submodules target that is processed
for each build.
The NoahMP files are linked into the physics directory. This emulates the existing paradigm
of putting all of the physics source into a single location. This minimizes the infrastructure
modifications, and becomes almost transparent to a casual user. Care will need to be taken
so that no files associated with NoahMP are accidentally re-incorporated back into WRF.
Considerations:
The introduction of an external repository is a complication. We choose to avoid modifying the WRF build
system, meaning that the entire physics directory must be WHOLE for a successful build. This is handled
by copying files from the NoahMP repository directory location to the WRF repository directory location.
The infrastructure has been put in place in the WRF build to copy the files to the expected WRF locations.
Therefore, starting from a clean clone and without a preceding build, a user will not find the NoahMP source
files in the physics directory. The NoahMP repository could be manually included as a submodule with the
following command:
However, again, until a build occurs, the files are not moved from the NoahMP repository directory structure into
the proper and expected locations in the WRF directory structure.
TESTS CONDUCTED:
command, and subsequent linking.
step is bypassed.
RELEASE NOTE: The NoahMP external repository is now a submodule for WRF.