How to fix metadata (as a preprocess) or disable 'cmor_check_metadata'?

[mpmoine]

Dear Support,

When using the rls radiative variable form the CNRM-CM6-1 model (in fact, our model 😉), we are facing a problem with the sign of this field. When we ran the model in 2018-2019, in the CMOR table Emon (and the Data Request as well), the sign was requested as positive up. But in more recent tables (the one ESMValTool is using to check conformances) it is requested as positive down.

Consequently we have errors when executing recipes using this variable, for instance recipe_radiation_budget.yml:

File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/cmor/fix.py", line 124, in fix_metadata
cube = checker(cube).check_metadata()
File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/cmor/check.py", line 188, in check_metadata
self.report_errors()
File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/cmor/check.py", line 244, in report_errors
raise CMORCheckError(msg)
esmvalcore.cmor.check.CMORCheckError: There were errors in variable rls:
rls: positive should be down, not up
in cube:
surface_net_downward_longwave_flux / (W m-2) (time: 1980; latitude: 128; longitude: 256)

We could fix this by hand, but it is not a satisfying solution, since all CNRM-CERFACS production is like that. Thus my question is: is there any mean to add a customised fix at the ESMValTool preprocessing step?

Many thanks,
Marie-Pierre and Hannah (who is doing an internship on ESMValTool at Cerfacs 🙂)

[HannahGaenslen]

Hi,

For completeness, here is the full log file and the exact recipe we've been using.

main_log_debug.txt
recipe_radiation_budget.txt

Cheers,
Hannah

[mpmoine]

While reading the ESMValCore doc, we see that the esmvalcore.cmor.fix.fix_data function is designed to fix known errors on data arrays and esmvalcore.cmor.fix.fix_metadata for fixing errors on metadata.
So would be fine if we could use that, but we are wondering what does the following sentense from the doc exactly means:

"This method collects all the relevant fixes for a given variable"

How are the relevant fix identified? is it internal to ESMValTool of taken from the ES-DOC Errata Service?

Thanks! 🍺
Marie-Pierre.

[mpmoine]

In fact we checked CNRM-CM6-1 data and the conclusion is that the rls field follows correctly the convention positive down, so there is no need to change the sign of rls. It is all negative values, both for HadGEM3 model (for which the recipe works) and CNRM-CM6-1 model. The only difference is:

• for HadGEM3, rls has no attribute positive (In the log we have only the WARNING : “attribute positive not present” when applying to HadGEM3 data)
• for CNRM-CM6-1 , rls has an attribute positive: up (it's erroneous) and this generates the error:
  esmvalcore.cmor.check.CMORCheckError: There were errors in variable rls: - rls: positive should be down, not up)

Trying with CNRM-CM6-1 data after a manual removing of this positive attribute...

[mpmoine]

Result: when removing the positive attribute it works! 👍

So, what I want to do is either deactivate the cmor_check_metadata, or preprocess systematically my input data. Would be suitable if I could do that with ESMValTool, but I cannot find a way to apply my own preprocessing in the doc...

Thanks

[rswamina]

Hi @mpmoine. It is likely someone will respond to you next week as most of our developers are at the EGU meeting this week.
I am tagging @zklaus who may know who to assign this to.

[mpmoine]

Thanks @rswamina. Thanks for the ping! 👍 No worry, it is not so urgent... And yes, I know Klaus 😉

[remi-kazeroni]

Hi @mpmoine, thanks for your request. In this case, I think it would be best to add a fix for the rls variable of the CNRM-CM6-1 model and fix the metadata on-the-fly. This way you would not need to hack the data beforehand. Dataset fixes can be applied to individual variables, mips or all model variables. In your case, it seems enough to add a Rls class to the cnrm-cm6-1 fixes. You can find an example of such metadata fix here. Such fixes need to be done in ESMValCore and are enabled if you use a development version of the Core. It would be great if you could share such fixes with the community by opening a PR in the Core to contribute them. One requirement would be to add some tests but we could help you with that if you want.

[valeriupredoi]

@mpmoine additionally, if you are looking for pirating the run and turning off completely the cmor checks, you can use the --check_level flag:

    --check_level=CHECK_LEVEL
        Type: Optional[]
        Default: None
        Configure the sensitivity of the CMOR check. Possible values are: `ignore` (all errors will be reported as warnings), `relaxed` (only fail if there are critical errors), default (fail if there are any errors), strict (fail if there are any warnings).

in your esmvaltool run command 🏴‍☠️

[HannahGaenslen]

Thank you for this simple solution! With --check_level relaxed it worked very well.