Custom user-defined location for custom CMOR tables

doc/develop/fixing_data.rst

@@ -342,6 +342,17 @@ This section describes how to add support for additional native datasets.
 You can choose to host this new data source either under a dedicated project or
 under project ``native6``.
 
+.. _add_new_fix_native_datasets_config:
+
+Configuration
+-------------
+
+An example of a configuration in ``config-developer.yml`` for projects used for
+native datasets is given :ref:`here <configure_native_models>`.
+Make sure to use the option ``cmor_strict: false`` for these projects if you
+want to make use of :ref:`custom_cmor_tables`.
+This allows reading arbitrary variables from native datasets.
+
 .. _add_new_fix_native_datasets_locate_data:
 
 Locate data

doc/quickstart/configure.rst

@@ -510,24 +510,83 @@ related to CMOR table settings available:
 * ``cmor_type``: can be ``CMIP5`` if the CMOR table is in the same format as the
   CMIP5 table or ``CMIP6`` if the table is in the same format as the CMIP6 table.
 * ``cmor_strict``: if this is set to ``false``, the CMOR table will be
-  extended with variables from the ``esmvalcore/cmor/tables/custom`` directory
-  and it is possible to use variables with a ``mip`` which is different from
-  the MIP table in which they are defined.
+  extended with variables from the :ref:`custom_cmor_tables` (by default loaded
+  from the ``esmvalcore/cmor/tables/custom`` directory) and it is possible to
+  use variables with a ``mip`` which is different from the MIP table in which
+  they are defined.
 * ``cmor_path``: path to the CMOR table.
   Relative paths are with respect to `esmvalcore/cmor/tables`_.
   Defaults to the value provided in ``cmor_type`` written in lower case.
 * ``cmor_default_table_prefix``: Prefix that needs to be added to the ``mip``
   to get the name of the file containing the ``mip`` table.
   Defaults to the value provided in ``cmor_type``.
 
+.. _custom_cmor_tables:
+
+Custom CMOR tables
+------------------
+
+As mentioned in the previous section, the CMOR tables of projects that use
+``cmor_strict: false`` will be extended with custom CMOR tables.
+By default, these are loaded from `esmvalcore/cmor/tables/custom
+<https://github.com/ESMValGroup/ESMValCore/tree/main/esmvalcore/cmor/tables/custom>`_.
+However, by using the special project ``custom`` in the
+``config-developer.yml`` file with the option ``cmor_path``, a custom location
+for these custom CMOR tables can be specified:
+
+.. code-block:: yaml
+
+   custom:
+     cmor_path: ~/my/own/custom_tables
+
+This path can be given as relative path (relative to `esmvalcore/cmor/tables`_)
+or as absolute path.
+Other options given for this special table will be ignored.
+
+Custom tables in this directory need to follow the naming convention
+``CMOR_{short_name}.dat`` and need to be given in CMIP5 format.
+
+Example for the file ``CMOR_asr.dat``:
+
+.. code-block::
+
+   SOURCE: CMIP5
+   !============
+   variable_entry:    asr
+   !============
+   modeling_realm:    atmos
+   !----------------------------------
+   ! Variable attributes:
+   !----------------------------------
+   standard_name:
+   units:             W m-2
+   cell_methods:      time: mean
+   cell_measures:     area: areacella
+   long_name:         Absorbed shortwave radiation
+   !----------------------------------
+   ! Additional variable information:
+   !----------------------------------
+   dimensions:        longitude latitude time
+   type:              real
+   positive:          down
+   !----------------------------------
+   !
+
+It is also possible to use a special coordinates file ``CMOR_coordinates.dat``.
+If this is not present in the custom directory, the one from the default
+directory (`esmvalcore/cmor/tables/custom/CMOR_coordinates.dat
+<https://github.com/ESMValGroup/ESMValCore/tree/main/esmvalcore/cmor/tables/custom/CMOR_coordinates.dat>`_)
+is used.
+
+
 .. _filterwarnings_config-developer:
 
 Filter preprocessor warnings
 ----------------------------
 
 It is possible to ignore specific warnings of the preprocessor for a given
 ``project``.
-This is particularly useful for native models which do not follow the CMOR
+This is particularly useful for native datasets which do not follow the CMOR
 standard by default and consequently produce a lot of warnings when handled by
 Iris.
 This can be configured in the ``config-developer.yml`` file for some steps of
@@ -590,6 +649,15 @@ Example:
 A detailed description on how to add support for further native datasets is
 given :ref:`here <add_new_fix_native_datasets>`.
 
+.. hint::
+
+   When using native datasets, it might be helpful to specify a custom location
+   for the :ref:`custom_cmor_tables`.
+   This allows reading arbitrary variables from native datasets.
+   Note that this requires the option ``cmor_strict: false`` in the
+   :ref:`project configuration <configure_native_models>` used for the native
+   model output.
+
 
 .. _config-ref:
 

doc/quickstart/find_data.rst

@@ -88,6 +88,15 @@ project, e.g., ``ICON`` (mostly native models).
 A detailed description of how to include new native datasets is given
 :ref:`here <add_new_fix_native_datasets>`.
 
+.. hint::
+
+   When using native datasets, it might be helpful to specify a custom location
+   for the :ref:`custom_cmor_tables`.
+   This allows reading arbitrary variables from native datasets.
+   Note that this requires the option ``cmor_strict: false`` in the
+   :ref:`project configuration <configure_native_models>` used for the native
+   model output.
+
 .. _read_native_obs:
 
 Supported native reanalysis/observational datasets

esmvalcore/cmor/table.py

@@ -55,11 +55,22 @@ def read_cmor_tables(cfg_developer=None):
     with open(var_alt_names_file, 'r') as yfile:
         alt_names = yaml.safe_load(yfile)
 
-    custom = CustomInfo()
     CMOR_TABLES.clear()
+
+    # Try to infer location for custom tables from config-developer.yml file,
+    # if not possible, use default location
+    custom_path = None
+    if 'custom' in cfg_developer:
+        custom_path = cfg_developer['custom'].get('cmor_path')
+    if custom_path is not None:
+        custom_path = os.path.expandvars(os.path.expanduser(custom_path))
+    custom = CustomInfo(custom_path)
     CMOR_TABLES['custom'] = custom
+
     install_dir = os.path.dirname(os.path.realpath(__file__))
     for table in cfg_developer:
+        if table == 'custom':
+            continue
         CMOR_TABLES[table] = _read_table(cfg_developer, table, install_dir,
                                          custom, alt_names)
 
@@ -815,18 +826,40 @@ class CustomInfo(CMIP5Info):
         Full path to the table or name for the table if it is present in
         ESMValTool repository
     """
+
     def __init__(self, cmor_tables_path=None):
         cwd = os.path.dirname(os.path.realpath(__file__))
-        self._cmor_folder = os.path.join(cwd, 'tables', 'custom')
+        default_cmor_folder = os.path.join(cwd, 'tables', 'custom')
+
+        # Get custom location of CMOR tables if possible
+        if cmor_tables_path is None:
+            self._cmor_folder = default_cmor_folder
+        else:
+            self._cmor_folder = self._get_cmor_path(cmor_tables_path)
+        if not os.path.isdir(self._cmor_folder):
+            raise ValueError(f"Custom CMOR tables path {self._cmor_folder} is "
+                             f"not a directory")
+
         self.tables = {}
         self.var_to_freq = {}
         table = TableInfo()
         table.name = 'custom'
         self.tables[table.name] = table
-        self._coordinates_file = os.path.join(
+
+        # Try to read coordinates from custom location, use default location if
+        # not possible
+        coordinates_file = os.path.join(
             self._cmor_folder,
             'CMOR_coordinates.dat',
         )
+        if os.path.isfile(coordinates_file):
+            self._coordinates_file = coordinates_file
+        else:
+            self._coordinates_file = os.path.join(
+                default_cmor_folder,
+                'CMOR_coordinates.dat',
+            )
+
         self.coords = {}
         self._read_table_file(self._coordinates_file, self.tables['custom'])
         for dat_file in glob.glob(os.path.join(self._cmor_folder, '*.dat')):

esmvalcore/config-developer.yml

@@ -2,8 +2,8 @@
 # Developer's configuration file for the ESMValTool
 ###############################################################################
 # This file retains the project- and machine-dependent directory and file name
-# definitions of the input and output data
-# Each dictionary is structured as follows
+# definitions of the input and output data.
+# Each dictionary is structured as follows:
 #
 # PROJECT:
 #   input_dir:
@@ -14,7 +14,16 @@
 #   input_file:
 #   output_file:
 #
-# Only the default drs is mandatory, the others are optional
+# Only the default drs is mandatory, the others are optional.
+#
+# In addition, an entry for the custom tables can be given. For this, only the
+# option 'cmor_path' is considered, which specifies the directory from which
+# custom CMOR tables are loaded. 'cmor_path' can be a relative path (relative
+# to ESMValCore/esmvalcore/cmor/tables) or an absolute path. By default, uses
+# ESMValCore/esmvalcore/cmor/tables/custom.
+#
+# custom:
+#   cmor_path: ~/my/own/custom_tables
 ###############################################################################
 ---
 

tests/integration/cmor/test_read_cmor_tables.py

@@ -5,6 +5,17 @@
 from esmvalcore.cmor.table import __file__ as root
 from esmvalcore.cmor.table import read_cmor_tables
 
+CUSTOM_CFG_DEVELOPER = {
+    'custom': {'cmor_path': Path(root).parent / 'tables' / 'custom'},
+    'CMIP6': {
+        'cmor_strict': True,
+        'input_dir': {'default': '/'},
+        'input_file': '*.nc',
+        'output_file': 'out.nc',
+        'cmor_type': 'CMIP6',
+    },
+}
+
 
 def test_read_cmor_tables():
     """Test that the function `read_cmor_tables` loads the tables correctly."""
@@ -33,3 +44,24 @@ def test_read_cmor_tables():
     table = CMOR_TABLES[project]
     assert Path(table._cmor_folder) == table_path / 'obs4mips' / 'Tables'
     assert table.strict is False
+
+
+def test_read_custom_cmor_tables():
+    """Test reading of custom CMOR tables."""
+    read_cmor_tables(CUSTOM_CFG_DEVELOPER)
+
+    assert len(CMOR_TABLES) == 2
+    assert 'CMIP6' in CMOR_TABLES
+    assert 'custom' in CMOR_TABLES
+
+    custom_table = CMOR_TABLES['custom']
+    assert (Path(custom_table._cmor_folder) ==
+            Path(root).parent / 'tables' / 'custom')
+    assert (Path(custom_table._coordinates_file) ==
+            Path(root).parent / 'tables' / 'custom' / 'CMOR_coordinates.dat')
+
+    cmip6_table = CMOR_TABLES['CMIP6']
+    assert cmip6_table.default is custom_table
+
+    # Restore default tables
+    read_cmor_tables(read_config_developer_file())

tests/integration/cmor/test_table.py

@@ -347,12 +347,45 @@ def setUpClass(cls):
         """
         cls.variables_info = CustomInfo()
 
+    def test_custom_tables_default_location(self):
+        """Test constructor with default tables location."""
+        custom_info = CustomInfo()
+        expected_cmor_folder = os.path.join(
+            os.path.dirname(esmvalcore.cmor.__file__),
+            'tables',
+            'custom',
+        )
+        expected_coordinate_file = os.path.join(
+            os.path.dirname(esmvalcore.cmor.__file__),
+            'tables',
+            'custom',
+            'CMOR_coordinates.dat',
+        )
+        self.assertEqual(custom_info._cmor_folder, expected_cmor_folder)
+        self.assertEqual(custom_info._coordinates_file,
+                         expected_coordinate_file)
+
     def test_custom_tables_location(self):
         """Test constructor with custom tables location."""
         cmor_path = os.path.dirname(os.path.realpath(esmvalcore.cmor.__file__))
         cmor_tables_path = os.path.join(cmor_path, 'tables', 'cmip5')
         cmor_tables_path = os.path.abspath(cmor_tables_path)
-        CustomInfo(cmor_tables_path)
+        custom_info = CustomInfo(cmor_tables_path)
+        self.assertEqual(custom_info._cmor_folder, cmor_tables_path)
+
+        expected_coordinate_file = os.path.join(
+            os.path.dirname(esmvalcore.cmor.__file__),
+            'tables',
+            'custom',
+            'CMOR_coordinates.dat',
+        )
+        self.assertEqual(custom_info._coordinates_file,
+                         expected_coordinate_file)
+
+    def test_custom_tables_invalid_location(self):
+        """Test constructor with invalid custom tables location."""
+        with self.assertRaises(ValueError):
+            CustomInfo('this_file_does_not_exist.dat')
 
     def test_get_variable_netcre(self):
         """Get tas variable."""