Customizing runtime settings

To run a CIME case, you will run the script case.submit after you have modified env_run.xml for your particular needs.

The env_run.xml file contains variables which may be modified at the initialization of a model run and during the course of that model run. These variables comprise coupler namelist settings for the model stop time, model restart frequency, coupler history frequency and a flag to determine if the run should be flagged as a continuation run. In general, you only need to set the variables $STOP_OPTION and $STOP_N. The other coupler settings will then be given consistent and reasonable default values. These default settings guarantee that restart files are produced at the end of the model run.

In the following, we focus on the handling of run control (e.g. length of run, continuing a run) and output data. We also give a more detailed description of CIME restarts.

Controlling starting, stopping and restarting a run

The case initialization type is set in env_run.xml. A CIME run can be initialized in one of three ways; startup, branch, or hybrid.

startup

In a startup run (the default), all components are initialized using baseline states. These baseline states are set independently by each component and can include the use of restart files, initial files, external observed data files, or internal initialization (i.e., a "cold start"). In a startup run, the coupler sends the start date to the components at initialization. In addition, the coupler does not need an input data file. In a startup initialization, the ocean model does not start until the second ocean coupling step.

branch

In a branch run, all components are initialized using a consistent set of restart files from a previous run (determined by the $RUN_REFCASE and $RUN_REFDATE variables in env_run.xml). The case name is generally changed for a branch run, although it does not have to be. In a branch run, setting $RUN_STARTDATE is ignored because the model components obtain the start date from their restart datasets. Therefore, the start date cannot be changed for a branch run. This is the same mechanism that is used for performing a restart run (where $CONTINUE_RUN is set to TRUE in the env_run.xml file).

Branch runs are typically used when sensitivity or parameter studies are required, or when settings for history file output streams need to be modified while still maintaining bit-for-bit reproducibility. Under this scenario, the new case is able to produce an exact bit-for-bit restart in the same manner as a continuation run if no source code or component namelist inputs are modified. All models use restart files to perform this type of run. $RUN_REFCASE and $RUN_REFDATE are required for branch runs.

To set up a branch run, locate the restart tar file or restart directory for $RUN_REFCASE and $RUN_REFDATE from a previous run, then place those files in the $RUNDIR directory. See setting up a branch run for an example.

hybrid

A hybrid run indicates that CESM is initialized more like a startup, but uses initialization datasets from a previous case. This is somewhat analogous to a branch run with relaxed restart constraints. A hybrid run allows users to bring together combinations of initial/restart files from a previous case (specified by $RUN_REFCASE) at a given model output date (specified by $RUN_REFDATE). Unlike a branch run, the starting date of a hybrid run (specified by $RUN_STARTDATE) can be modified relative to the reference case. In a hybrid run, the model does not continue in a bit-for-bit fashion with respect to the reference case. The resulting climate, however, should be continuous provided that no model source code or namelists are changed in the hybrid run. In a hybrid initialization, the ocean model does not start until the second ocean coupling step, and the coupler does a "cold start" without a restart file.

The variable $RUN_TYPE determines the initialization type. This setting is only important for the initial run of a production run when the $CONTINUE_RUN variable is set to FALSE. After the initial run, the $CONTINUE_RUN variable is set to TRUE, and the model restarts exactly using input files in a case, date, and bit-for-bit continuous fashion. The variable $RUN_TYPE is the start date (in yyyy-mm-dd format) either a startup or hybrid run. If the run is targeted to be a hybrid or branch run, you must also specify values for $RUN_REFCASE and $RUN_REFDATE. All run startup variables are discussed in run start control variables.

Before a job is submitted to the batch system, you need to first check that the batch submission lines in env_batch.xml are appropriate. These lines should be checked and modified accordingly for appropriate account numbers, time limits, and stdout/stderr file names. You should then modify env_run.xml to determine the key run-time settings. See controlling run termination, controlling run restarts, and performing model restarts for more details. A brief note on restarting runs. When you first begin a branch, hybrid or startup run, CONTINUE_RUN must be set to FALSE. When you successfully run and get a restart file, you will need to change CONTINUE_RUN to TRUE for the remainder of your run. See performing model restarts for more details. Setting the RESUBMIT option to a value > 0 will cause the CONTINUE_RUN variable to be automatically set to TRUE upon completion of the initial run.

By default,

STOP_OPTION = ndays
STOP_N = 5
STOP_DATE = -999

The default setting is only appropriate for initial testing. Before a longer run is started, update the stop times based on the case throughput and batch queue limits. For example, if the model runs 5 model years/day, set RESUBMIT=30, STOP_OPTION= nyears, and STOP_N= 5. The model will then run in five year increments, and stop after 30 submissions.

Customizing component-specific namelist settings

In your $CASEROOT directory, the subdirectory $CASEROOT/Buildconf contains files to create the component namelists, build the component libraries and create the model executable. Buildconf/$component.buildexe.csh creates the component libraries and Buildconf/$component.buildnml.csh creates the component namelists. A new feature in the CESM1.1 and CESM1.2 release series is that ALL CESM components now use a component-specific build-namelist utility (similar to that of CAM, CLM and CICE in the CESM1.0 series) to generate their respective model namelists. In addition, CAM, CLM and CICE have an associated configure utility that sets up compile time configuration options and is also called from the corresponding Buildconf/*.buildnml.csh (e.g. Buildconf/cam.buildnml.csh).

In the CESM1.2 series, user specific component namelist changes should only be made only by editing the ``$CASEROOT/user_nl_xxx`` files OR by changing xml variables in `env_run.xml <http://www.cesm.ucar.edu/models/cesm2.0/external-link-here>`_ or `env_build.xml <http://www.cesm.ucar.edu/models/cesm2.0/external-link-here>`_. A full discussion of how to change the namelist variables for each component is provided below. You can preview the case component namelists by running preview_namelists in your $CASEROOT. Calling p``review_namelists`` results in the creation of component namelists (e.g. atm_in, lnd_in, .etc) in $CASEROOT/CaseDocs/. A complete documentation of all model component namelists for CESM1.2 releases is now available. The namelist files created in the CaseDocs/ are there only for user reference and SHOULD NOT BE EDITED since they are overwritten every time preview_namelists, $CASE.run and $CASE.build are called. In CESM1.2, (like CESM1.1 but unlike CESM1.0) the only files that you should modify are in $CASEROOT. No files in Buildconf/ should be changed. The following represents a summary of controlling and modifying component-specific run-time settings:

DRV

In CESM1.2, driver namelist are in two groups - those that are set directly from xml variables in env_case.xml, env_mach_pes.xml and env_run.xml, and those that are set by the driver build-namelist utility ($CCSMROOT/models/drv/bld/build-namelist) for the target compset and resolution. Except for the following driver namelist variables (see below), driver namelist variables that are in env_run.xml can be changed either by changing the xml variable OR by adding the correct key-word value pair at the end of user_nl_cpl, where any changes in user_nl_cpl will take precedence over values set in the xml file. For example, to change eps_frac to 1.0e-15, add the following line to the end of the user_nl_cpl, "eps_frac = 1.0e-15". To see the result of this modification to user_nl_cpl call preview_namelists and verify that this new value appears in CaseDocs/drv_in.

The following namelist variables MAY NOT be changed in ``user_nl_cpl`` -
but must be changed in the appropriate ``$CASEROOT`` xml file.  
XXX  refers to ATM,LND,ICE,OCN,ROF,GLC,WAV
======================================
drv namelist   => xml variable 
variable       
======================================
case_name      => CASE                
username       => CCSMUSER         
hostname       => MACH                
model_version  => CCSM_REPOTAG  
start_type     => RUN_TYPE          
start_ymd      => RUN_STARTDATE  
start_tod      => START_TOD
XXX_cpl_dt     => XXX_NCPL          
XXX_ntasks     => NTASKS_XXX
XXX_nthreads   => NTHRDS_XXX
XXX_rootpe     => ROOTPE_XXX
XXX_pestride   => PSTRID_XXX
XXX_layout     => NINST_XXX_LAYOUT

CAM

CAM's configure and build-namelist utilities are called by Buildconf/cam.buildnml.csh. CAM_CONFIG_OPTS, CAM_NAMELIST_OPTS and CAM_NML_USECASE are used to set compset variables (e.g., "-phys cam5" for CAM_CONFIG_OPTS) and in general should not be modified for supported compsets. For a complete documentation of namelist settings, see CAM namelist variables. To modify CAM namelist settings, you should add the appropriate keyword/value pair at the end of the $CASEROOT/user_nl_cam file (see the documentation for each file at the top of that file). For example, to change the solar constant to 1363.27, modify the user_nl_cam file to contain the following line at the end "solar_const=1363.27". To see the result of adding this, call preview_namelists and verify that this new value appears in CaseDocs/atm_in.

CLM

CLM's configure and build-namelist utilities are called by Buildconf/clm.buildnml.csh. CLM_CONFIG_OPTS and CLM_NML_USE_CASE are used to set compset specific variables and in general should not be modified for supported compsets. For a complete documentation of namelist settings, see CLM namelist variables. To modify CLM namelist settings, you should add the appropriate keyword/value pair at the end of the $CASEROOT/user_nl_clm file (see the documentation for each file at the top of that file). To see the result of your change, call preview_namelists and verify that the changes appear correctly in CaseDocs/lnd_in.

RTM

RTM's build-namelist utility is called by Buildconf/rtm.buildnml.csh. For a complete documentation of namelist settings, see RTM namelist variables. To modify RTM namelist settings, you should add the appropriate keyword/value pair at the end of the $CASEROOT/user_nl_rtm file (see the documentation for each file at the top of that file). To see the result of your change, call preview_namelists and verify that the changes appear correctly in CaseDocs/rof_in.

CICE

CICE's configure and build-namelist utilities are now called by Buildconf/cice.buildnml.csh. Note that CICE_CONFIG_OPTS, and CICE_NAMELIST_OPTS are used to set compset specific variables and in general should not be modified for supported compsets. For a complete documentation of namelist settings, see CICE namelist variables. To modify CICE namelist settings, you should add the appropriate keyword/value pair at the end of the $CASEROOT/user_nl_cice file (see the documentation for each file at the top of that file). To see the result of your change, call preview_namelists and verify that the changes appear correctly in CaseDocs/ice_in.

In addition, cesm_setup creates CICE's compile time block decomposition variables in env_build.xml as follows:

./cesm_setup
   ⇓
Buildconf/cice.buildnml.csh and $NTASKS_ICE and $NTHRDS_ICE
   ⇓
env_build.xml variables CICE_BLCKX, CICE_BLCKY, CICE_MXBLCKS, CICE_DECOMPTYPE 
CPP variables in cice.buildexe.csh

POP2

See POP2 namelist variables for a complete description of the POP2 run-time namelist variables. Note that OCN_COUPLING, OCN_ICE_FORCING, OCN_TRANSIENT are normally utilized ONLY to set compset specific variables and should not be edited. For a complete documentation of namelist settings, see CICE namelist variables. To modify POP2 namelist settings, you should add the appropriate keyword/value pair at the end of the $CASEROOT/user_nl_pop2 file (see the documentation for each file at the top of that file). To see the result of your change, call preview_namelists and verify that the changes appear correctly in CaseDocs/ocn_in.

In addition, cesm_setup also generates POP2's compile time compile time block decomposition variables in env_build.xml as follows:

./cesm_setup  
   ⇓
Buildconf/pop2.buildnml.csh and $NTASKS_OCN and $NTHRDS_OCN
   ⇓
env_build.xml variables POP2_BLCKX, POP2_BLCKY, POP2_MXBLCKS, POP2_DECOMPTYPE 
CPP variables in pop2.buildexe.csh

CISM

See CISM namelist variables for a complete description of the CISM run-time namelist variables. This includes variables that appear both in cism_in and in cism.config. To modify any of these settings, you should add the appropriate keyword/value pair at the end of the user_nl_cism file (see the documentation for each file at the top of that file). To see the result of your change, call preview_namelists and verify that the changes appear correctly in CaseDocs/cism_in and CaseDocs/cism.config.

There are also some run-time settings set via env_run.xml, as documented in CISM run time variables - in particular, the model resolution, set via CISM_GRID. The value of CISM_GRID determines the default value of a number of other namelist parameters.

DATM

DATM is discussed in detail in Data Model's User's Guide. DATM is normally used to provide observational forcing data (or forcing data produced by a previous run using active components) to drive CLM (I compset), POP2 (C compset), and POP2/CICE (G compset). As a result, DATM variable settings are specific to the compset that will be targeted.

DATM can be user configured in three different ways.

You can set DATM run-time variables my modifying control settings for CLM and CPLHIST forcing.

You can edit user_nl_datm to change namelist settings namelists settings by adding all user specific namelist changes in the form of "namelist_var = new_namelist_value". Note that any namelist variable from shr_strdata_nml and datm_nml can be modified below using the this syntax. Use preview_namelists to view (not modify) the output namelist in CaseDocs.

You can modify the contents of a DATM stream txt file. To do this:

• use preview_namelists to obtain the contents of the stream txt files in CaseDocs
• place a copy of this file in $CASEROOT with the string "*user*_" prepended
• Make sure you change the permissions of the file to be writeabl (chmod 644)
• Modify the user_datm.streams.txt.* file.

As an example, if the stream txt file in CaseDocs/ is datm.streams.txt.CORE2_NYF.GISS, the modified copy in $CASEROOT should be user_datm.streams.txt.CORE2_NYF.GISS. After calling preview_namelists again, you should see your new modifications appear in CaseDocs/datm.streams.txt.CORE2_NYF.GISS.

DOCN

DOCN is discussed in detail in Data Model's User's Guide.

DOCN running in prescribed mode assumes that the only field in the input stream is SST and also that SST is in Celsius and must be converted to Kelvin. All other fields are set to zero except for ocean salinity, which is set to a constant reference salinity value. Normally the ice fraction data (used for prescribed CICE) is found in the same data files that provide SST data to the data ocean model since SST and ice fraction data are derived from the same observational data sets and are consistent with each other. For DOCN prescribed mode, default yearly climatological datasets are provided for various model resolutions. For multi-year runs requiring AMIP datasets of sst/ice_cov fields, you need to set the variables for DOCN_SSTDATA_FILENAME, DOCN_SSTDATA_YEAR_START, and DOCN_SSTDATA_YEAR_END. CICE in prescribed mode also uses these values.

DOCN running as a slab ocean model is used (in conjunction with CICE running in prognostic mode) in all E compsets. SOM ("slab ocean model") mode is a prognostic mode. This mode computes a prognostic sea surface temperature and a freeze/melt potential (surface Q-flux) used by the sea ice model. This calculation requires an external SOM forcing data file that includes ocean mixed layer depths and bottom-of-the-slab Q-fluxes. Scientifically appropriate bottom-of-the-slab Q-fluxes are normally ocean resolution dependent and are derived from the ocean model output of a fully coupled run. Note that while this mode runs out of the box, the default SOM forcing file is not scientifically appropriate and is provided for testing and development purposes only. Users must create scientifically appropriate data for their particular application. A tool is available to derive valid SOM forcing.

DOCN can be user-customized in three ways.

You can set DOCN run-time variables.

You can edit user_nl_docn to change namelist settings by adding all user specific namelist changes in the form of "namelist_var = new_namelist_value". Note that any namelist variable from shr_strdata_nml and datm_nml can be modified below using the this syntax. Use preview_namelists to view (not modify) the output namelist in CaseDocs.

You can modify the contents of a DOCN stream txt file. To do this:

• use preview_namelists to obtain the contents of the stream txt files in CaseDocs/
• place a copy of this file in $CASEROOT with the string "*user*_" prepended
• Make sure you change the permissions of the file to be writeable (chmod 644)
• Modify the user_docn.streams.txt.* file.

As an example, if the stream text file in CaseDocs/ is doc.stream.txt.prescribed, the modified copy in $CASEROOT should be user_docn.streams.txt.prescribed. After changing this file and calling preview_namelists again, you should see your new modifications appear in CaseDocs/docn.streams.txt.prescribed.

DICE

DICE is discussed in detail in Data Model's User's Guide.

DICE can be user-customized in three ways.

You can set DICE run-time variables.

You can edit user_nl_dice to change namelist settings by adding all user specific namelist changes in the form of "namelist_var = new_namelist_value". Note that any namelist variable from shr_strdata_nml and datm_nml can be modified below using the this syntax. Use preview_namelists to view (not modify) the output namelist in CaseDocs/.

You can modify the contents of a DICE stream txt file. To do this:

• use preview_namelists to obtain the contents of the stream txt files in CaseDocs/
• place a copy of this file in $CASEROOT with the string "*user*_" prepended
• Make sure you change the permissions of the file to be writeable (chmod 644)
• Modify the user_dice.streams.txt.* file.

DLND

DLND is discussed in detail in Data Model's User's Guide. The data land model is different from the other data models because it can run as a purely data-land model (reading in coupler history data for atm/land fluxes and land albedos produced by a previous run), or to read in model output from CLM to send to CISM.

DLND can be user-customized in three ways:

You can set DLND run-time variables.

You can edit user_nl_dlnd OR user_nl_dsno depending on the component set, to change namelist settings namelists settings by adding all user specific namelist changes in the form of "namelist_var = new_namelist_value". Note that any namelist variable from shr_strdata_nml and dlnd_nml or dsno_nml can be modified below using the this syntax. Use preview_namelists to view (not modify) the output namelist in CaseDocs/.

You can modify the contents of a DLND stream txt file. To do this:

• use preview_namelists to obtain the contents of the stream txt files in CaseDocs/
• place a copy of this file in $CASEROOT with the string "*user*_" prepended
• Make sure you change the permissions of the file to be writeable (chmod 644)
• Modify the user_dlnd.streams.txt.* file.

DROF

DROF is discussed in Data Model's User's Guide. The data river runoff model reads in runoff data and sends it back to the coupler. In general, the data river runoff model is only used to provide runoff forcing data to POP2 when running C or G compsets

DROF can be user-customized in three ways:

You can set DROF run-time variables.

You can edit user_nl_drof to change namelist settings namelists settings by adding all user specific namelist changes in the form of "namelist_var = new_namelist_value". Note that any namelist variable from shr_strdata_nml and drof_nml can be modified using the this syntax. Use preview_namelists to view (not modify) the output namelist in CaseDocs/.

You can modify the contents of a DROF stream txt file. To do this:

• use preview_namelists to obtain the contents of the stream txt files in CaseDocs/
• place a copy of this file in $CASEROOT with the string "*user*_" prepended
• Make sure you change the permissions of the file to be writeable (chmod 644)
• Modify the user_drof.streams.txt.* file.

Controlling output data

During a model run, each CESM component produces its own output datasets consisting of history, restart and output log files. Component history files and restart files are in netCDF format. Restart files are used to either exactly restart the model or to serve as initial conditions for other model cases.

Archiving is a phase of a CESM model run where the generated output data is moved from $RUNDIR to a local disk area (short-term archiving) and subsequently to a long-term storage system (long-term archiving). It has no impact on the production run except to clean up disk space and help manage user quotas. Although short-term and long-term archiving are implemented independently in the scripts, there is a dependence between the two since the short-term archiver must be turned on in order for the long-term archiver to be activated. In env_run.xml, several variables control the behavior of short and long-term archiving. See short and long term archiving for a description of output data control variables. Several important points need to be made about both short and long term archiving:

• By default, short-term archiving is enabled and long-term archiving is disabled.
• All output data is initially written to $RUNDIR.
• Unless a user explicitly turns off short-term archiving, files will be moved to $DOUT_S_ROOT at the end of a successful model run.
• If long-term archiving is enabled, files will be moved to $DOUT_L_MSROOT by $CASE.l_archive, which is run as a separate batch job after the successful completion of a model run.
• Users should generally turn off short term-archiving when developing new CESM code.
• If long-term archiving is not enabled, users must monitor quotas and usage in the $DOUT_S_ROOT/ directory and should manually clean up these areas on a frequent basis.

Standard output generated from each CESM component is saved in a "log file" for each component in $RUNDIR. Each time the model is run, a single coordinated datestamp is incorporated in the filenames of all output log files associated with that run. This common datestamp is generated by the run script and is of the form YYMMDD-hhmmss, where YYMMDD are the Year, Month, Day and hhmmss are the hour, minute and second that the run began (e.g. ocn.log.040526-082714). Log files are also copied to a user specified directory using the variable $LOGDIR in env_run.xml. The default is a 'logs' subdirectory beneath the case directory.

By default, each component also periodically writes history files (usually monthly) in netCDF format and also writes netCDF or binary restart files in the $RUNDIR directory. The history and log files are controlled independently by each component. History output control (i.e. output fields and frequency) is set in the Buildconf/$component.buildnml.csh files.

The raw history data does not lend itself well to easy time-series analysis. For example, CAM writes one or more large netCDF history file(s) at each requested output period. While this behavior is optimal for model execution, it makes it difficult to analyze time series of individual variables without having to access the entire data volume. Thus, the raw data from major model integrations is usually postprocessed into more user-friendly configurations, such as single files containing long time-series of each output fields, and made available to the community.

As an example, for the following example settings:

DOUT_S = TRUE
DOUT_S_ROOT = /ptmp/$user/archive
DOUT_L_MS = TRUE
DOUT_L_MSROOT /USER/csm/$CASE

the run will automatically submit the $CASE.l_archive to the queue upon its completion to archive the data. The system is not bulletproof, and you will want to verify at regular intervals that the archived data is complete, particularly during long running jobs.