Carbon Imbalance with CN runs

[peterdschwartz]

Discovered that GridCBalanceCheck doesn't use absolute value when error checking so negative imbalances can propagate through the Land Model. Line is here

Running e3sm_land_developer test suite after changing to absolute value shows tests with CN enabled all FAIL with small ( ~1.e-07) imbalances. Similar but independent error to keep in mind is Issue #6052

[thorntonpe]

Well, that's not good. Does the list of failing tests include CNP, or just CN tests?

[thorntonpe]

Great catch @peterdschwartz! We will need to dig in on this one.

[peterdschwartz]

@thorntonpe Includes CNP tests as well. FATES is also affected.

[peterdschwartz]

The single point tests like SMS_Ly2_P1x1_D.1x1_smallvilleIA.IELMCNCROP.chrysalis_intel.elm-lulcc_sville still pass. First thing that I need to verify is that the initialization optimizations I did aren't the cause the error. If not, I'll look into creating a single point run that can replicate this error.

[glemieux]

@thorntonpe Includes CNP tests as well. FATES is also affected.

FYI @rgknox

[peterdschwartz]

Good news! My recent changes to col_cf_init for EcosystemDynMod are the cause of the error for at least one test. Hopefully once I fix that, the other tests are fixed as well.

[peterdschwartz]

Got rid of carbon imbalances from all but FATES configs. The FATES issue isn't from my recent PR as checking out b633fd5f2680a0ee434213d5b4d39b363d18ef81 and fixing the mistake with GridCBalanceCheck produces those imbalances

[glemieux]

Thanks for the heads up @peterdschwartz. Could I grab your branch fix to check this out and see about addressing the FATES issue?

[rgknox]

same, could you post here please?

[peterdschwartz]

Take a look at this branch: https://github.com/peterdschwartz/E3SM/commits/peterdschwartz/lnd/fix-balance-errors/

[glemieux]

Just a quick note that I was able to replicate the issue.

[glemieux]

I ran a couple of tests with fates that write out some diagnostic statements after cherry-picking the top commit from @peterdschwartz's branch into a copy of the fates landuse branch from PR #5760. I also reinstated the ColCBalanceCheck for fates noted in #6120. The test case makes it past the column carbon check just fine as the balance error is on the order of E-15.

From the diagnostic output, it looks like the grc_cinputs are zero, which I think is the problem. The grc_conputs are only comprised of er values (i.e. everything else like fire is zero), which I think makes sense for fates. As such, the balance error is simply (0 - grc_cinputs)*dt.

Based on the way the column balance check is set up, in which we only use litter fall and er for the inputs and outputs respectively, I think we may need something similar for the gridcell level check with fates runs. I think this would simply involve calling c2g on the col_cf%litfall variable and creating a similar check as in the column carbon balance error. How does that sounds @rgknox and @peterdschwartz?

[peterdschwartz]

@glemieux
I made some changes based on your thoughts and the one FATES test does PASS now. I don't understand why FATES needs to set different carbon inputs, but here are the changes I made below. I'll try the other fates tests as well.

@@ -940,13 +948,21 @@ contains
       call c2g(bounds, col_som_c_leached(bounds%begc:bounds%endc), grc_som_c_leached(bounds%begg:bounds%endg), &
                c2l_scale_type = 'unity', l2g_scale_type = 'unity')
       call c2g(bounds, col_som_c_yield(bounds%begc:bounds%endc), grc_som_c_yield(bounds%begg:bounds%endg), &
-               c2l_scale_type = 'unity', l2g_scale_type = 'unity')
+               c2l_scale_type = 'unity', l2g_scale_type = 'unity')

+      if(use_fates) then
+        call c2g(bounds, col_cf%litfall(bounds%begc:bounds%endc), grc_cinputs(bounds%begg:bounds%endg), &
+               c2l_scale_type = 'unity', l2g_scale_type = 'unity')
+      end if
+
       dt = real( get_step_size(), r8 )
       nstep = get_nstep()

       do g = bounds%begg, bounds%endg
-         grc_cinputs(g) = grc_gpp(g) + grc_dwt_seedc_to_leaf(g) + grc_dwt_seedc_to_deadstem(g)
+
+         if(.not. use_fates) then
+           grc_cinputs(g) = grc_gpp(g) + grc_dwt_seedc_to_leaf(g) + grc_dwt_seedc_to_deadstem(g)
+         end if

          grc_coutputs(g) = grc_er(g) + grc_fire_closs(g) + grc_hrv_xsmrpool_to_atm(g) + &
               grc_prod1c_loss(g) + grc_prod10c_loss(g) + grc_prod100c_loss(g) - grc_som_c_leached(g) + &
@@ -981,6 +997,14 @@ contains
             write(iulog,*)'endcb               = ', end_totc(g)
             write(iulog,*)'delta store         = ', end_totc(g) - beg_totc(g)

+            write(iulog,*)'Delta totpftc       = ', end_totpftc(g) - beg_totpftc(g)
+            write(iulog,*)'Delta cwdc          = ', end_cwdc(g) - beg_cwdc(g)
+            write(iulog,*)'Delta totlitc       = ', end_totlitc(g) - beg_totlitc(g)
+            write(iulog,*)'Delta totsomc       = ', end_totsomc(g) - beg_totsomc(g)
+            write(iulog,*)'Delta totprodc      = ', end_totprodc(g) - beg_totprodc(g)
+            write(iulog,*)'Delta ctrunc        = ', end_ctrunc(g) - beg_ctrunc(g)
+            write(iulog,*)'Delta crop deficit  = ', end_cropseedc_deficit(g) - beg_cropseedc_deficit(g)
+

[peterdschwartz]

Ok, no more FAILs nor DIFFs for my branch. I'll rebase to run the newly added tests, and if that's good, make a PR with this fix and a few others.