Replace recipe_era5.yml with recipe_daily_era5.yml

[SarahAlidoost]

Description

• Closes Including recipe_era5.yml under the folder cmorizer is misleading the users. #1909
• Link to documentation:
  • Updated section "Obtaining input data"
  • New recipe description

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Code is written according to the code quality guidelines
• 🛠 Documentation is available
• 🛠 Tests run successfully
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

• 🧪 Recipe runs successfully
• 🧪 Recipe is well documented

New or updated data reformatting script

• 🛠 Documentation is available
• 🛠 The dataset has been added to the CMOR check recipe

---

To help with the number of pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[bouweandela]

Thanks, this is starting to look really good. I have a few more comments on the content of the documentation, but I think the structure is much improved now, especially if we get more supported datasets in the future.

[bouweandela]

@esmvalbot Please run cmorizers/recipe_daily_era5.yml

[esmvalbot]

Since @bouweandela asked, ESMValBot will run recipe cmorizers/recipe_daily_era5.yml as soon as possible, output will be generated here

[esmvalbot]

ESMValBot is sorry to report it failed to run recipe cmorizers/recipe_daily_era5.yml: exit is 1, output has been generated here

[bouweandela]

It looks like the bot ran out of memory trying to run the recipe.
The broken tests on CircleCI are not related to this pull request.

[bouweandela]

The recipe also runs out of memory on a 64 GB RAM machine, even with max_parallel_tasks: 1 in config-user.yml.

[SarahAlidoost]

The recipe also runs out of memory on a 64 GB RAM machine, even with max_parallel_tasks: 1 in config-user.yml.

The recipe (including all variables) run with max_parallel_tasks: 1 was successful on Cartesius. here is more info from the run directory:
main_log_debug.txt
resource_usage.txt
main_log.txt

and more info from run/daily/rename directory:
log.txt
resource_usage.txt

and the environment file:
environment.txt

[remi-kazeroni]

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

Also this PR will add the MSWEP dataset to the list of supported datasets. For datasets for which we provide a specific cmorizer script, download instructions can usually be found in the script header. This is different for MSWEP for which I couldn't reproduce the steps to download. Could we work on that in this PR or should I open a specific issue for that? Thanks!

[bouweandela]

Also this PR will add the MSWEP dataset to the list of supported datasets. For datasets for which we provide a specific cmorizer script, download instructions can usually be found in the script header. This is different for MSWEP for which I couldn't reproduce the steps to download. Could we work on that in this PR or should I open a specific issue for that? Thanks!

It would be great if you could open a specific issue for MSWEP, because this pull request is really about ERA5. We tried to reorganize the documentation a bit so things are more logically organized, but let's not let this pull request grow too large.

[bouweandela]

The memory problems occurred for me with dask version 2021.7.1 (more than 64GB required). With version 2.30.0 of dask I was able to run the recipe successfully on Mistral (max memory usage 13GB). This might be something to look out for with other recipes too, we may need to pin the dask version..

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

@SarahAlidoost Could you please have a look at this? Once the lists are consistent, this can be merged.

[SarahAlidoost]

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

thanks for your comment. The hourly variables in recipe_daily_era5.yml match all the hourly variables in the documentation, except ptype, rlns, rlus, and rsns, rsus. I think this is OK because this recipe only includes the variables for which we know how to convert hourly data to daily data.

Also, now the monthly and hourly variables in recipe_check_obs.yml match the variables in the documentation, except the hourly/monthly variables rlns, rsns and the hourly ones rlus, rsus. These were added in this PR. There is a cutom table for rlns and rsns.
To add these variables to recipe_check_obs.yml, we need the mip info:

	var	mip
hourly/monthly	rlns	?
hourly/monthly	rsns	?
hourly	rlus	?
hourly	rsus	?

[SarahAlidoost]

The memory problems occurred for me with dask version 2021.7.1 (more than 64GB required). With version 2.30.0 of dask I was able to run the recipe successfully on Mistral (max memory usage 13GB). This might be something to look out for with other recipes too, we may need to pin the dask version..

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

@SarahAlidoost Could you please have a look at this? Once the lists are consistent, this can be merged.

Please see my comment.

[bouweandela]

To add these variables to recipe_check_obs.yml, we need the mip info:

@SarahAlidoost I think you can just use the usual Amon for monthly and E1hr for the hourly variables. For the latter it might be good to set the frequency too in the recipe, since these are radiation variables they are probably accumulated and the frequency should be 1hr. @lukasbrunner Since you added these variables in ESMValGroup/ESMValCore#1052, please correct me if I'm wrong.

[bouweandela]

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

@remi-kazeroni Has your comment been addressed now?

[remi-kazeroni]

If this the right place for that, would it be possible to make the list of ERA5 variables for which cmorization is supported consistent between the recipes (recipe_check_obs.yml, recipe_daily_era5.yml) and the documentation?

@remi-kazeroni Has your comment been addressed now?

@bouweandela yes, that looks fine for me now. Thanks @SarahAlidoost for taking my comment into account.