Improve reading facets from ESGF search results

[bouweandela]

Description

Improve the way facets are read from the ESGF search results and use them to create the local directory structure. This results in more cases where the facets are available and fixes an issue with the obs4MIPs directory where a . in the dataset name was interpreted as a / when creating the local file path.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

To try out the improved facets, run

from esmvalcore.esgf import find_files

files = find_files(
    short_name='tas',
    mip='Amon',
    project='CMIP5',
    dataset='*',
    institute='*',
    ensemble='r1i1p1',
    exp='historical',
    timerange='2000/2000',
)

for file in files:
    print(file)
    print(file.facets)

and note that with this pull request all datasets have many facets.

Closes #1873

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #1920 (9874227) into main (d61d200) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1920      +/-   ##
==========================================
+ Coverage   92.11%   92.12%   +0.01%     
==========================================
  Files         234      234              
  Lines       12130    12151      +21     
==========================================
+ Hits        11173    11194      +21     
  Misses        957      957              

Impacted Files	Coverage Δ
esmvalcore/esgf/facets.py	100.00% <ø> (ø)
esmvalcore/esgf/_download.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[valeriupredoi]

looking good, bud (although am no specialist on esgf searches but I do very much appreciate the amount of testing added here!) 🍺

[remi-kazeroni]

@ESMValGroup/tech-reviewers, would one of you have time to do a final check and merge this, please? It would be great to have this merged soon as it also helps resolving issues in #1609. Thanks 👍

[schlunma]

Thanks for this @bouweandela, I have two additional minor comments on this.

Could you also please briefly summarize what the issue was and how this PR fixes that? To be honest, I don't really understand what's going on here, and you are adding a significant amount of code here. I think the linked issue (#1873) is only relevant for a small part of the changes.

[bouweandela]

Could you also please briefly summarize what the issue was and how this PR fixes that?

@schlunma The main issue is that the facets on ESGF are often incorrect. I tried to work around this by reading them from the dataset_id (e.g. CMIP6.CMIP.NCAR.CESM2.historical.r4i1p1f1.Amon.tas.gn.v20190308) using the template dataset_id_template_ that defines it (e.g. %(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s). This is the code that is currently in the main branch.

Unfortunately, this does not work in many cases because either the dataset_id_template_ is incorrect or there is a dot in the dataset name (often the case for obs4MIPs) and this is the character that separates the facets in the dataset_id. This results in ESGFFiles with empty or very few facets. The new approach is to read the facets from the json response of the ESGF query and then try to correct them by reading them from the dataset_id (assuming at least that is correct). This is the problem that prevented you from using these facets to create datasets in the recipe 589.yml.txt from this comment #1609 (comment).

If you run the code in the description of the pull request with the current main branch and then with this branch and look at the difference, you can see the improvement.

[schlunma]

Nice, got it, thanks for the explanation @bouweandela! I tested the code from the PR description, and can confirm that the facets look much better with this PR!

Will merge after the tests have passed. Cheers!!