New coupled baseline #349

PeterHjortLauritzen · 2023-07-24T22:41:02Z

Description of the run

B1850 coupled simulation.

Atmosphere: same as #345 except

clubb_gamma_coef = 0.30->0.35
clubb_gamma_coefb = 0.3 -> 0.35

Why?

Need new baseline.
latest F-case baseline has RESTOM=1.169, SWCF=-47.551 and LWCF=23.98. By increasing clubb_gamma_coef we expect SWCF to be closer to -45W/m^2 and increasing an RESTOM.

Suffix in the casename

Suffix

Namelist modifications

Namelist

Source modifications

SourceMods

Sandbox

Requested sandbox or tag

Contact info

@cecilehannay @adamrher @JulioTBacmeister @PeterHjortLauritzen

Any other relevant information

Enter relevant info

The text was updated successfully, but these errors were encountered:

adamrher · 2023-07-25T16:47:50Z

I think we should use the new land spun-up inic for this run @olyson @wwieder. Note also that the cloud tunings in this new coupled sim are substantially different from that used to generate the off-line land spinup, so the biases may look quite different from the offline generated climo.

olyson · 2023-07-25T17:04:22Z

I agree. The new initial conditions are here:

/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc

cecilehannay · 2023-07-26T20:01:16Z

Based on offline discussions with @PeterHjortLauritzen, @adamrher, and @cacraigucar:

we will use gamma_coef=0.33 and 0.35. The concern is that 0.35 would be an overkill in term of RESTOM

clubb_gamma_coef = 0.3 ->0.33
clubb_gamma_coefb = 0.3 -> 0.33

we will use a new cesm tag cesm2_3_alpha16b. @fischer-ncar will create this tag tomorrow
We will include some mods and/or user_nl_cam from @fvitt for a SOAE bug described at Need to adjust SAOE emissions factors ESCOMP/CAM#860

jedwards4b · 2023-07-26T20:29:33Z

Is this to be run on derecho? If so let me know when set up and I can do some load balancing.

cecilehannay · 2023-07-26T20:59:47Z

Yes, this run will be on derecho. It would be wonderful if you can provide a pe layout.

cecilehannay · 2023-07-27T22:27:31Z

We are getting ready to start a new coupled run with cesm2_3_alpha16b

The preliminary case directory is:

/glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033

The setting is as follow:

ATM

@PeterHjortLauritzen, @adamrher and @JulioTBacmeister

we will be matching: #353

LND

@olyson, @wwieder and @slevis-lmwg

I used SourceMods from

/glade/u/home/oleson/run_hist_1850_files/SPINUP/casefiles_cecile/SourceMods/ctsm5.1.dev120/src.clm

and user_nl_clm

use_init_interp = .true.
fsurdat = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_ne30np4.pg3_SSP5-8.5_78pfts_CMIP6_1850-2100_c230227.nc'
flanduse_timeseries = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/landuse.timeseries_ne30np4.pg3_SSP5-8.5_78_CMIP6_1850-2100_c230227.nc'
finidat='/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc'

OCN

@gustavo-marques and @klindsay28:

I added a diag_table in:

/glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033/SourceMods/src.mom/diag_table

Do I need anything else?

ICE

@dabail10:

Should I use a special ice_ic?

pe_layout

@jedwards4b:

Do you want me to run one month so you can provide a pe_layout?

olyson · 2023-07-27T22:59:15Z

LND: Please remove the SnowHydrologyMod.F90 SourceMod, it's not longer needed.
Please remove flanduse_timeseries = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/landuse.timeseries_ne30np4.pg3_SSP5-8.5_78_CMIP6_1850-2100_c230227.nc'. Since this is an 1850 run, it's not needed.
I believe you can remove use_init_interp=.true., because the initial file you are using is generated from a coupler history run that used the new surface dataset. But let me know when it fails :)

dabail10 · 2023-07-27T23:03:03Z

Not sure about sea ice. If this is the new MOM grid, we don't have a sea ice restart. Maybe Gustavo's G case?

gustavo-marques · 2023-07-27T23:46:05Z

We do have a restart from a G case with the new ocean grid. Please add the following to user_nl_cice.

ice_ic = '/glade/p/cesmdata/cseg/inputdata/cesm2_init/gmom.e23.GJRAv4.TL319_t061_zstar_N65.tx2_3v2.001/0061-01-01/gmom.e23.GJRAv4.TL319_t061_zstar_N65.tx2_3v2.001.cice.r.0061-01-01-00000.nc'

The MOM6 diag_table is outdated. We will give you a new one tomorrow.

jedwards4b · 2023-07-28T13:23:06Z

@olyson I think that the use_init_interp=.true. is still needed:

dec2305.hsn.de.hpc.ucar.edu 1:  check_dim_size ERROR: mismatch of input dimension        15920 
dec2305.hsn.de.hpc.ucar.edu 1:   with expected value        15962  for variable gridcell
dec2305.hsn.de.hpc.ucar.edu 1: Did you mean to set use_init_interp = .true. in user_nl_clm?
dec2305.hsn.de.hpc.ucar.edu 1: (Setting use_init_interp = .true. is needed when doing a
dec2305.hsn.de.hpc.ucar.edu 1: transient run using an initial conditions file from a non-transient run,
dec2305.hsn.de.hpc.ucar.edu 1: or a non-transient run using an initial conditions file from a transient run,
dec2305.hsn.de.hpc.ucar.edu 1: or when running a resolution or configuration that differs from the initial conditions.)
dec2305.hsn.de.hpc.ucar.edu 1:  ERROR: 
dec2305.hsn.de.hpc.ucar.edu 1:  ERROR in /glade/work/hannay/cesm_tags/cesm2_3_alpha16b/components/clm/src/main/
dec2305.hsn.de.hpc.ucar.edu 1:  ncdio_pio.F90.in at line 409

jedwards4b · 2023-07-28T13:24:38Z

Also from cam:

FLDLST: CLD_MISR in fincl(116, 1) not found
 FLDLST: FISCCP1_COSP in fincl(117, 1) not found
 FLDLST: CLD_CAL in fincl(118, 1) not found
 FLDLST: CLD_MISR in fincl(119, 1) not found
 FLDLST: CLDTOT_CAL in fincl(120, 1) not found
 FLDLST: CLDHGH_CAL in fincl(121, 1) not found
 FLDLST: CLDMED_CAL in fincl(122, 1) not found
 FLDLST: CLDLOW_CAL in fincl(123, 1) not found
 FLDLST: CLMODIS in fincl(124, 1) not found
 FLDLST: Uzm in fincl(1, 5) only available with FV dycore
 FLDLST: Vzm in fincl(2, 5) only available with FV dycore
 FLDLST: Wzm in fincl(3, 5) only available with FV dycore
 FLDLST: THzm in fincl(4, 5) only available with FV dycore
 FLDLST: VTHzm in fincl(5, 5) only available with FV dycore
 FLDLST: WTHzm in fincl(6, 5) only available with FV dycore
 FLDLST: UVzm in fincl(7, 5) only available with FV dycore
 FLDLST: UWzm in fincl(8, 5) only available with FV dycore
 ERROR: FLDLST: 9 errors found, see log

jedwards4b · 2023-07-28T13:33:31Z

Adding ./xmlchange --append CAM_CONFIG_OPTS=-cosp solved the problem with cam FLDLST.

jedwards4b · 2023-07-28T13:47:50Z

I ran this setup in :
/glade/derecho/scratch/jedwards/PFS.ne30pg3_t232.BLT1850_v0c.derecho_intel.20230728_065830_p4afpe
I think that it's pretty well balanced as is.

Overall Metrics:                                                                                                                                                                                                 
    Model Cost:            8247.37   pe-hrs/simulated_year                                                                                                                                                         
    Model Throughput:         6.33   simulated_years/day                                                                                                                                                           ```

cecilehannay · 2023-07-28T14:16:35Z

The variables below require to as COSP to the CAM_CONFIG_OPT

FLDLST: CLD_MISR in fincl(116, 1) not found
 FLDLST: FISCCP1_COSP in fincl(117, 1) not found
 FLDLST: CLD_CAL in fincl(118, 1) not found
 FLDLST: CLD_MISR in fincl(119, 1) not found
 FLDLST: CLDTOT_CAL in fincl(120, 1) not found
 FLDLST: CLDHGH_CAL in fincl(121, 1) not found
 FLDLST: CLDMED_CAL in fincl(122, 1) not found
 FLDLST: CLDLOW_CAL in fincl(123, 1) not found
 FLDLST: CLMODIS in fincl(124, 1) not found

But the other variables are unrelated to COSP. These are the variables for the TEM diags.
I agree that this is a bug that we get this warning (I always get this) and it should be fixed.

 FLDLST: Uzm in fincl(1, 5) only available with FV dycore
 FLDLST: Vzm in fincl(2, 5) only available with FV dycore
 FLDLST: Wzm in fincl(3, 5) only available with FV dycore
 FLDLST: THzm in fincl(4, 5) only available with FV dycore
 FLDLST: VTHzm in fincl(5, 5) only available with FV dycore
 FLDLST: WTHzm in fincl(6, 5) only available with FV dycore
 FLDLST: UVzm in fincl(7, 5) only available with FV dycore
 FLDLST: UWzm in fincl(8, 5) only available with FV dycore

olyson · 2023-07-28T14:16:54Z

Thanks @jedwards4b . I guess the use_init_interp is needed because of the mesh mask difference. The Fcase that provided the coupler history files to force the land spinup, and the land spinup itself used:

/glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc

while the fully coupled simulation here uses:

/glade/p/cesmdata/inputdata/share/meshes/tx2_3v2_230415_ESMFmesh.nc

When the first is used, it produces 15920 land gridcells. When the second is used, it produces 15962 gridcells.

cecilehannay · 2023-07-28T15:29:01Z

@gustavo-marques and @klindsay28

I see that out of the box I get:

MEKE_GEOMETRIC_ALPHA = 0.07000000000000001

In previous runs, we set:

MEKE_GEOMETRIC_ALPHA = 0.10

Which one do I need?

gustavo-marques · 2023-07-28T15:32:27Z

Please use MEKE_GEOMETRIC_ALPHA = 0.10

adamrher · 2023-07-28T15:36:36Z

@olyson are these mesh files the "maskfile" variable? I guess I didn't realize until now that we probably need to change our grid aliases used in the F compsets to use the MOM6 mask for the coastlines.

olyson · 2023-07-28T15:49:11Z

I looked at the mesh_mask variable in nuopc.runconfig, e.g.,

mesh_mask = /glade/p/cesmdata/cseg/inputdata/share/meshes/gx1v7_151008_ESMFmesh.nc

It looks like that is what it is using to create landfrac and landmask, e.g., from the land log:

Computing land fraction and land mask by mapping mask from mesh_mask file

lnd_set_decomp_and_domain: writing landmask and landfrac data to landfrac.nc

Successfully wrote land fraction/mask status file ./init_generated_files/ctsm_landfrac.status
decomp precompute numg,nclumps,seglen1,avg_seglen,nsegspc= 15920
1800 T 1.000000 8.844444
Surface Grid Characteristics
longitude points = 48600
latitude points = 1
total number of land gridcells = 15920
Decomposition Characteristics
clumps per process = 1

That didn't occur to me either.

dabail10 · 2023-07-28T16:21:53Z

Why is the land/ocean mask gx1v7? Should it not be the 2/3 degree tripole grid?

adamrher · 2023-07-28T16:26:34Z

Why is the land/ocean mask gx1v7? Should it not be the 2/3 degree tripole grid?

This just refers to the CLM spinup, not the coupled run. As I said, we didn't add a grid alias for the F-cases that uses the 2/3deg mask. We should probably open an issue. Do we even have the actual new 2/3deg grid in the latest ccs_config?

gustavo-marques · 2023-07-28T16:33:28Z

Yes, the new 2/3 deg grid was added to ccs_config via ESMCI/ccs_config_cesm#105

adamrher · 2023-07-28T16:42:53Z

OK thx. Opened an issue: ESMCI/ccs_config_cesm#112

gustavo-marques · 2023-07-28T17:27:21Z

@cecilehannay, here is the updated diag_table for MOM:

/glade/scratch/gmarques/for_cecile/diag_table

cecilehannay · 2023-07-28T20:50:07Z

I have still having some issues to start the coupled run: /glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16b.BLT1850.ne30_t232.033
Hopefully, this can be solved before the week-end.

In the mean time, I included the change you suggested above.

klindsay28 · 2023-07-28T22:24:46Z

@cecilehannay, please try the following:
copy $CASEROOT/CaseDocs/input.nml to $CASEROOT/SourceMods/src.mom
edit the copy in SourceMods and add the line

    max_num_axis_sets = 26

to diag_manager_nml. This suggestion is based on the last line of ocn.log:

FATAL from PE     0: diag_axis_mod::diag_axis_init: num_axis_sets (26) exceeds max_num_axis_sets (25).  Increase max_num_axis_sets via diag_manager_nml.

This is an FMS namelist, so it is not something you would set from user_nl_mom.

cecilehannay · 2023-07-29T00:19:04Z

Thanks. I added:

  max_num_axis_sets = 26

and submitted

Right now, the queues on derecho seem to be hanging. But hopefully, the job will start later today.

cecilehannay · 2023-07-29T13:42:29Z

It set as recommended in the message error @klindsay28 identified:

max_num_axis_sets = 26

But the model is now requesting to set it to 27.

FATAL from PE     1: diag_axis_mod::diag_axis_init: num_axis_sets (27) exceeds max_num_axis_sets (26).  Increase max_num_axis_sets via diag_manager_nml.

I could keep increasing but I am not sure what this variable does and whether it is an ok strategy.

cecilehannay · 2023-07-29T14:06:23Z

It is an issue with:

SourceMods/src.mom/diag_table

as when I remove this mods, it is running fine. (I just tried that).

Should I keep increasing:

max_num_axis_sets

klindsay28 · 2023-07-29T14:08:36Z

I see that @gustavo-marques has it set to 50 in some of his recent cases, so I suggest going with that.

max_num_axis_sets is an FMS variable used to allocate internal memory for diagnostic axis information. I'm not quite sure why the modified diag_table requires larger values, but it looks like FMS is using it to allocate more space for strings, so it's okay to use larger values. That is, we're not dramatically increasing memory usage by increasing this variable.

cecilehannay · 2023-07-29T14:09:57Z

Thanks for the explanation. I will set it to 50 (instead of increasing by 1 at a time).

cecilehannay · 2023-07-29T14:40:10Z

It went further but is now complaining about

max_output_fields =          300

I could increase that too. Any recommendations of a good value?

klindsay28 · 2023-07-29T14:44:31Z

Could you look at one of Gustavo’s recent cases? I’m away from my computer at the moment.

…

On Sat, Jul 29, 2023 at 8:40 AM Cecile Hannay ***@***.***> wrote: It went further but is now complaining about max_output_fields = 300 I could increase that too. Any recommendations of a good value? — Reply to this email directly, view it on GitHub <#349 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADWZPO55DPRKYTAHEZKGKL3XSUOFLANCNFSM6AAAAAA2WHAIDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

gustavo-marques · 2023-07-29T15:56:31Z

@cecilehannay, please copy the following into $CASEROOT/SourceMods/src.mom:

/glade/scratch/gmarques/for_cecile/input.nml

cecilehannay · 2023-07-29T17:50:24Z

And now it is the max_axes that is exceeded.

FATAL from PE   169: diag_axis_mod::diag_axis_init: max_axes exceeded, increase via diag_manager_nml

The error message doesn't give suggestion. I set

max_axes = 50

We'll see.

cecilehannay · 2023-07-29T18:28:52Z

The model crashed with

max_axes = 50

I set to:

max_axes = 500

It seems to be running now.

klindsay28 · 2023-07-30T00:11:20Z

Thanks for the update. Sorry for the hassle.

cecilehannay · 2023-08-03T22:10:32Z

See #356

PeterHjortLauritzen added the Run request Label for CAM developpers who requested a new development simulation label Jul 24, 2023

PeterHjortLauritzen mentioned this issue Jul 24, 2023

FLTHIST version of new coupled baseline #350

Closed

cecilehannay mentioned this issue Jul 31, 2023

b.e23_alpha16b.BLT1850.ne30_t232.033 #356

Open

cecilehannay closed this as completed Aug 3, 2023

New coupled baseline #349

New coupled baseline #349

Comments

PeterHjortLauritzen commented Jul 24, 2023

Description of the run

Suffix in the casename

Namelist modifications

Source modifications

Sandbox

Contact info

Any other relevant information

adamrher commented Jul 25, 2023

olyson commented Jul 25, 2023

cecilehannay commented Jul 26, 2023

jedwards4b commented Jul 26, 2023

cecilehannay commented Jul 26, 2023

cecilehannay commented Jul 27, 2023

ATM

LND

OCN

ICE

pe_layout

olyson commented Jul 27, 2023

dabail10 commented Jul 27, 2023

gustavo-marques commented Jul 27, 2023

jedwards4b commented Jul 28, 2023

jedwards4b commented Jul 28, 2023

jedwards4b commented Jul 28, 2023

jedwards4b commented Jul 28, 2023

cecilehannay commented Jul 28, 2023 • edited Loading

olyson commented Jul 28, 2023

cecilehannay commented Jul 28, 2023

gustavo-marques commented Jul 28, 2023

adamrher commented Jul 28, 2023

olyson commented Jul 28, 2023

dabail10 commented Jul 28, 2023

adamrher commented Jul 28, 2023 • edited Loading

gustavo-marques commented Jul 28, 2023

adamrher commented Jul 28, 2023

gustavo-marques commented Jul 28, 2023

cecilehannay commented Jul 28, 2023

klindsay28 commented Jul 28, 2023

cecilehannay commented Jul 29, 2023

cecilehannay commented Jul 29, 2023

cecilehannay commented Jul 29, 2023

klindsay28 commented Jul 29, 2023 • edited Loading

cecilehannay commented Jul 29, 2023

cecilehannay commented Jul 29, 2023

klindsay28 commented Jul 29, 2023 via email

gustavo-marques commented Jul 29, 2023

cecilehannay commented Jul 29, 2023 • edited Loading

cecilehannay commented Jul 29, 2023

klindsay28 commented Jul 30, 2023

cecilehannay commented Aug 3, 2023

cecilehannay commented Jul 28, 2023 •

edited

Loading

adamrher commented Jul 28, 2023 •

edited

Loading

klindsay28 commented Jul 29, 2023 •

edited

Loading

cecilehannay commented Jul 29, 2023 •

edited

Loading