Read values of extra_facets from ESGF?

[bouweandela]

It is possible to automatically generate some of the files in the esmvalcore/_config/extra_facets directory with the information provided on ESGF. I tried this and it generates a nice list of institutes that seems more complete than what we have for CMIP3 (some entries missing) and CORDEX (we have no file at all). However, for CMIP5 the institute names on ESGF appear to differ from the institute names used in the filenames in some cases, so there the file we have is more correct than what is generated by just looking at the facets on ESGF (maybe a sort of mapping is needed here too, as for the dataset names in CMIP5 have the same issue).

Here is an example script that pulls the institutes from ESGF:

import yaml
from esmvalcore.esgf.facets import DATASET_MAP, FACETS
from pyesgf.search import SearchConnection

REVERSE_DATASET_MAP = {
    project: {value: key
              for key, value in DATASET_MAP[project].items()}
    for project in DATASET_MAP
}


def generate_institute_yml(project):
    conn = SearchConnection('http://esgf-index1.ceda.ac.uk/esg-search',
                            distrib=True)
    dataset_facet = FACETS[project]['dataset']
    ctx = conn.new_context(project=project, facets=dataset_facet)
    datasets = list(ctx.facet_counts[dataset_facet])

    result = {}
    for dataset in datasets:
        query = {'project': project, dataset_facet: dataset}
        ctx = conn.new_context(**query, facets='institute')
        institutes = list(ctx.facet_counts['institute'])
        dataset_name = REVERSE_DATASET_MAP[project].get(dataset, dataset)
        result[dataset_name] = {'*': {'*': {'institute': institutes}}}
    with open(f"{project.lower()}-institutes.yml", 'w',
              encoding='utf8') as file:
        yaml.dump(result, file, sort_keys=True)


if __name__ == '__main__':
    for project in 'CMIP3', 'CMIP5', 'CORDEX':
        generate_institute_yml(project)

and here is a script for comparing the resulting files:

import sys
from pathlib import Path

import yaml


def compare(file1, file2):
    content1 = yaml.safe_load(file1.read_text())
    content2 = yaml.safe_load(file2.read_text())
    for dataset in content1:
        if dataset not in content2:
            print(f"{dataset} missing from {file1}")
    for dataset in content2:
        if dataset not in content1:
            print(f"{dataset} missing from {file2}")
    for dataset in content1:
        if dataset in content2:
            institutes1 = content1[dataset]['*']['*']['institute']
            institutes2 = content2[dataset]['*']['*']['institute']
            if sorted(institutes1) != sorted(institutes2):
                print(
                    f"institutes are different for {dataset}: {institutes1} != {institutes2}"
                )


if __name__ == '__main__':
    compare(Path(sys.argv[1]), Path(sys.argv[2]))