Providing a new obs dataset for ESMValTool?

[Kate-Willett]

Hi,

Please can I have some advice.

I've been wondering how I can make my global in situ surface humidity dataset easier to use. Its gridded (coarsely, 5x5) monthly means and anomalies from 1973 to present, updated every year. I have a version on CEDA (land: https://catalogue.ceda.ac.uk/uuid/062942e96a6e4567b2bc47045be910a7, marine: https://catalogue.ceda.ac.uk/uuid/54d3408edd9e41bca226924754619812, blended: https://catalogue.ceda.ac.uk/uuid/563cb665bc6e43f99b355a9bb8134317). This was intended to be a CMOR compliant format but that was a few years ago so I would like to know if it actually is or what level of modifications might be needed, if people think that would be useful.

I've been told to add this to get the appropriate people's attention: @ESMValGroup/esmvaltool-developmentteam

Thanks

Kate Willett

[remi-kazeroni]

Hi @Kate-Willett, thanks for your question and your consideration in using ESMValTool for your dataset. It looks to me that the data structure is not too different from the one of HadCRUT5 for which we have support, i.e. a "CMORizer" script that makes a dataset fully CMOR-compliant and which can then be used in ESMValTool recipes. A CMORizer is basically made of 2 files:

• a config file that contains information about the original dataset filename, a list of variables available, ... See HadCRUT5.yml as an example.
• a formatter script that reads the original dataset, applies various fixes (coordinates, units, ...) to produce CMOR-compliant data and saves the data with the correct filename (to be latter readable by ESMValTool). See hadcrut5.py as an example.

To get an idea of the modifications needed to process the dataset with ESMValTool, you could try to develop a prototype for a CMORizer, based on the HadCRUT5 example, and apply it to see which remaining CMOR issues should be addressed if any. We have a tutorial lesson for this task. The main steps could be summarized as follows, assuming that you have access to a development version of ESMValTool:

1. Put the original (raw) data in a ~/RAWOBS/Tier2/HadISDH directory and add that RAWOBS directory to your config-user file (see run-the-existing-cmorizer-script)
2. Create a basic formatter script and cmor config file to load the original dataset and save it with the correct filename (see create-a-new-cmorizer-script-and-a-corresponding-config-file).
3. Run the formatter script with esmvaltool data format --config_file <your_config_user.yml> HadISDH. Get the CMORized data produced in ~/esmvaltool_output/data_formatting_YYYYMMDD_HHMMSS/Tier2/HadISDH/OBS*.nc, put a copy in your ~/OBS/Tier2/HadISDH directory and add that OBS directory to your config-user file.
4. Create a simple test recipe to be run with ESMValTool to check which additional CMOR fixes would be needed. The recipe should look like this one. (See make-a-test-recipe) for explanations. The recipe can then be run with esmvaltool run --config_file <your_config_user.yml> --log_level debug recipe_check_hadisdh.yml.

If you have any questions or doubts about this, please let me know. If you put your prototype of CMORizer in a branch in this repository, I'd be happy to take a look and help finalising it.

EDIT: The tutorial episode on CMORization is now up-to-date.

[Kate-Willett]

Hi Rémi, Many thanks for such a detailed response. That sounds doable to me. It will take me a while to find some time to do that but your message lays out the steps very clearly. It will be very useful to understand how different the current format of HadISDH is in terms of any modifications I can make at the development level when I produce the annual updates. I'll see how I get on. Kate From: Rémi Kazeroni ***@***.***> Sent: 11 August 2022 16:12 To: ESMValGroup/ESMValTool ***@***.***> Cc: Willett, Kate ***@***.***>; Mention ***@***.***> Subject: Re: [ESMValGroup/ESMValTool] Providing a new obs dataset for ESMValTool? (Discussion #2765) This email was received from an external source. Always check sender details, links & attachments. Hi @Kate-Willett<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FKate-Willett&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275253847166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j%2B5WUeqKIYzE2gvF85povbVQWRsCYFKf9dhGumwtnt0%3D&reserved=0>, thanks for your question and your consideration in using ESMValTool for your dataset. It looks to me that the data structure is not too different from the one of HadCRUT5<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcrudata.uea.ac.uk%2Fcru%2Fdata%2Ftemperature%2F&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275253847166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Qtgu3KDZIazUL3jK%2BhPgbmTQuVQhfZwWlJoShPx%2BKR0%3D&reserved=0> for which we have support, i.e. a "CMORizer" script that makes a dataset fully CMOR-compliant and which can then be used in ESMValTool recipes. A CMORizer is basically made of 2 files: * a config file<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool%2Fblob%2Fmain%2Fesmvaltool%2Fcmorizers%2Fdata%2Fcmor_config%2F&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=86mN05uHeJVl0guG20aPlaJrO6uYIMjMsgSNU2efoOU%3D&reserved=0> that contains information about the original dataset filename, a list of variables available, ... See HadCRUT5.yml<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool%2Fblob%2Fmain%2Fesmvaltool%2Fcmorizers%2Fdata%2Fcmor_config%2FHadCRUT5.yml&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZtddtzPjO1DvgIykB2XwLw3TNTxO8KFRuUbUKgqRnqM%3D&reserved=0> as an example. * a formatter script<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool%2Fblob%2Fmain%2Fesmvaltool%2Fcmorizers%2Fdata%2Fformatters%2Fdatasets%2F&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Rr7tIeeMs4ZZWJQSwapoMENpkTKtgSCa9iPObkpcpag%3D&reserved=0> that reads the original dataset, applies various fixes (coordinates, units, ...) to produce CMOR-compliant data and saves the data with the correct filename (to be latter readable by ESMValTool). See hadcrut5.py<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool%2Fblob%2Fmain%2Fesmvaltool%2Fcmorizers%2Fdata%2Fformatters%2Fdatasets%2Fhadcrut5.py&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mZR1XPP3vKnGjl8zG3uwvlHZORGqYXmUBfRDQBhEWiI%3D&reserved=0> as an example. To get an idea of the modifications needed to process the dataset with ESMValTool, you could try to develop a prototype for a CMORizer, based on the HadCRUT5 example, and apply it to see which remaining CMOR issues should be addressed if any. We have a tutorial lesson<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fesmvalgroup.github.io%2FESMValTool_Tutorial%2F09-cmorization%2Findex.html&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gxkSh8Ui%2FtHPp5ZytF1YZjoHsklA1vlbRY%2Bo65lLYQI%3D&reserved=0> for this task but this is unfortunately outdated (in fact, just the commands to run the CMORizer script) but I have an updated version ready here<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool_Tutorial%2Fblob%2F0546b8f788db85ef58dd0a24fa5a81da8a84f6a5%2F_episodes%2F09-cmorization.md&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0X6oVoVRpXTz9ZG7dCGpHBEXQ7mnhE7cA4hwBat5UDI%3D&reserved=0>. The main steps could be summarized as follows, assuming that you have access to a development version<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.esmvaltool.org%2Fen%2Flatest%2Fquickstart%2Finstallation.html%23install-from-source&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=22kf43iblAdSwJtnV%2BP4NykewCDwSxrbSRO8AUituA0%3D&reserved=0> of ESMValTool: 1. Put the original (raw) data in a ~/RAWOBS/Tier2/HadISDH directory and add that RAWOBS directory to your config-user file (see here<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool_Tutorial%2Fblob%2F0546b8f788db85ef58dd0a24fa5a81da8a84f6a5%2F_episodes%2F09-cmorization.md%23run-the-existing-cmorizer-script&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=axB8Wff3nT2Z5bS56bwQrWfesrtXa9hG9d7jv2%2BrI4Y%3D&reserved=0>) 2. Create a basic formatter script and cmor config file to load the original dataset and save it with the correct filename (see here<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool_Tutorial%2Fblob%2F0546b8f788db85ef58dd0a24fa5a81da8a84f6a5%2F_episodes%2F09-cmorization.md%23create-a-new-cmorizer-script-and-a-corresponding-config-file&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dfX3Jw%2BPr%2BQ7ww6ksllm%2FY0B9NKqvnFzGxZolxS7IHs%3D&reserved=0>). 3. Run the formatter script with esmvaltool data format --config_file <your_config_user.yml> HadISDH. Get the CMORized data produced in ~/esmvaltool_output/data_formatting_YYYYMMDD_HHMMSS/Tier2/HadISDH/OBS*.nc, put a copy in your ~/OBS/Tier2/HadISDH directory and add that OBS directory to your config-user file. 4. Create a simple test recipe to be run with ESMValTool to check which additional CMOR fixes would be needed. The recipe should look like this one<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool_Tutorial%2Fblob%2F0546b8f788db85ef58dd0a24fa5a81da8a84f6a5%2Ffiles%2Frecipe_check_fluxcom.yml&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lLt5hlp1e0OOB4b9JDHyVrv85uP0wbu%2BK9GM61BA79E%3D&reserved=0>. See here<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool_Tutorial%2Fblob%2F0546b8f788db85ef58dd0a24fa5a81da8a84f6a5%2F_episodes%2F09-cmorization.md%23make-a-test-recipe&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wVaLrAtb%2FRdy5DRqH8uv2MJrWPzBwx7Z0kQ1iYDX43o%3D&reserved=0> for explanations. The recipe can then be run with esmvaltool run --config_file <your_config_user.yml> --log_level debug recipe_check_hadisdh.yml. If you have any questions or doubts about this, please let me know. If you put your prototype of CMORizer in a branch in this repository, I'd be happy to take a look and help finalising it. - Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FESMValGroup%2FESMValTool%2Fdiscussions%2F2765%23discussioncomment-3376645&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fpJgNVBSSGb6Md6Jdx7laFq6iReSU4eh%2F0AgLjBrzAY%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fnotifications%2Funsubscribe-auth%2FADILUC4T2GNM6TU47MYDXGDVYUJ37ANCNFSM555LROGA&data=05%7C01%7Ckate.willett%40metoffice.gov.uk%7Cb9eaccbcff204a16044c08da7babd851%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C637958275254003362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8I3vNGVfPe%2FCXrNEudfOcDMD1PTgXv%2F64%2Fwr672kP1o%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>