Transfer Excel data I/O from message_ix

.appveyor.yml

@@ -4,6 +4,8 @@
 environment:
   # Use the JDK installed on AppVeyor images
   JAVA_HOME: C:\Program Files\Java\jdk13
+  # Always display verbose exceptions in JDBCBackend
+  IXMP_JDBC_EXCEPTION_VERBOSE: '1'
   matrix:
   - PYTHON_VERSION: "3.6"
   - PYTHON_VERSION: "3.7"

.travis.yml

@@ -1,20 +1,23 @@
 # Continuous integration configuration for Travis
+# NB use https://config.travis-ci.com/explore to validate changes
 
 dist: xenial
 
 language: r
 
 r: release
 
-matrix:
-  include:
-  - os: linux
-    env: PYENV=py37
-  - os: osx
-    env: PYENV=py37
-# turn these on once travis support gets a little better, see pyam for example
-#  - os: windows
-#    env: PYENV=py37
+# - Build against Python 3.7
+# - Always display verbose exceptions in JDBCBackend
+env:
+  PYENV=py37
+  IXMP_JDBC_EXCEPTION_VERBOSE=1
+
+os:
+- linux
+- osx
+# TODO turn this on once Travis Windows support improves
+# - windows
 
 r_packages:
 - IRkernel

RELEASE_NOTES.rst

@@ -4,6 +4,7 @@ Next release
 All changes
 -----------
 
+- `#286 <https://github.com/iiasa/ixmp/pull/286>`_: Add :meth:`.Scenario.to_excel` and :meth:`.read_excel`; this functionality is transferred to ixmp from :mod:`message_ix`.
 - `#270 <https://github.com/iiasa/ixmp/pull/270>`_: Include all tests in the ixmp package.
 - `#212 <https://github.com/iiasa/ixmp/pull/212>`_: Add :meth:`Model.initialize` API to help populate new Scenarios according to a model scheme.
 - `#267 <https://github.com/iiasa/ixmp/pull/267>`_: Apply units to reported quantities.

ci/conda-requirements.txt

@@ -8,7 +8,7 @@ numpydoc
 pandas
 pint
 pytest-cov
-pytest>=3.9
+pytest>=5
 PyYAML
 sparse
 sphinx

doc/source/api-backend.rst

@@ -36,6 +36,10 @@ Provided backends
 
    .. tip:: Modifying an item by adding or deleting elements invalidates its cache.
 
+   JDBCBackend has the following limitations:
+
+   - The `comment` argument to :meth:`Platform.add_unit` is limited to 64 characters.
+
 .. automethod:: ixmp.backend.jdbc.start_jvm
 
 Backend API
@@ -86,6 +90,7 @@ Backend API
 
       close_db
       get_auth
+      get_log_level
       get_nodes
       get_scenarios
       get_units

doc/source/api-python.rst

@@ -72,6 +72,7 @@ TimeSeries
       is_default
       last_update
       preload_timeseries
+      read_file
       remove_geodata
       remove_timeseries
       run_id
@@ -144,6 +145,7 @@ Scenario
       load_scenario_data
       par
       par_list
+      read_excel
       remove_par
       remove_set
       remove_solution
@@ -152,6 +154,7 @@ Scenario
       set_list
       set_meta
       solve
+      to_excel
       var
       var_list
 

doc/source/api.rst

@@ -10,6 +10,7 @@ On separate pages:
 
    api-python
    api-backend
+   file-io
    api-model
    reporting
 

doc/source/file-io.rst

@@ -0,0 +1,78 @@
+File formats and input/output
+*****************************
+
+In addition to the data management features provided by :doc:`api-backend`, ixmp is able to write and read :class:`TimeSeries` and :class:`Scenario` data to and from files.
+This page describes those options and formats.
+
+Time series data
+================
+
+Time series data can be:
+
+- Read using :meth:`.import_timeseries`, or the CLI command ``ixmp import timeseries FILE`` for a single TimeSeries object.
+- Written using :meth:`.export_timeseries_data` for multiple TimeSeries objects at once.
+
+Both CSV and Excel files in the IAMC time-series format are supported.
+
+.. _excel-data-format:
+
+Scenario/model data
+===================
+
+Scenario data can be read from/written to Microsoft Excel files using :meth:`.Scenario.read_excel` and :meth:`.to_excel`, and the CLI commands ``ixmp import scenario FILE`` and ``ixmp export FILE``.
+The files have the following structure:
+
+- One sheet named 'ix_type_mapping' with two columns:
+
+  - 'item': the name of an ixmp item.
+  - 'ix_type': the item's type as a length-3 string: 'set', 'par', 'var', or 'equ'.
+
+- One sheet per item.
+- Sets:
+
+  - Sheets for index sets have one column, with a header cell that is the set name.
+  - Sheets for one-dimensional indexed sets have one column, with a header cell that is the index set name.
+  - Sheets for multi-dimensional indexed sets have multiple columns.
+  - Sets with no elements are represented by empty sheets.
+
+- Parameters, variables, and equations:
+
+  - Sheets have zero (for scalar items) or more columns with headers that are the index *names* (not necessarily sets; see below) for those dimensions.
+  - Parameter sheets have 'value' and 'unit' columns.
+  - Variable and equation sheets have 'lvl' and 'mrg' columns.
+  - Items with no elements are not included in the file.
+
+Limitations
+-----------
+
+Reading variables and equations
+   The ixmp API provides no way to set the data of variables and equations, because these are considered model solution data.
+
+   Thus, while :meth:`.to_excel` will write files containing variable and equation data, :meth:`.read_excel` can not add these to a Scenario, and only emits log messages indicating that they are ignored.
+
+Multiple dimensions indexed by the same set
+   :meth:`.read_excel` provides the `init_items` argument to create new sets and parameters when reading a file.
+   However, the file format does not capture information needed to reconstruct the original data in all cases.
+
+   For example::
+
+      scenario.init_set('foo')
+      scenario.add_set('foo', ['a', 'b', 'c'])
+      scenario.init_par(name='bar', idx_sets=['foo'])
+      scenario.init_par(
+          name='baz',
+          idx_sets=['foo', 'foo'],
+          idx_names=['foo', 'another_dimension'])
+      scenario.to_excel('file.xlsx')
+
+   :file:`file.xlsx` will contain sheets named 'bar' and 'baz'.
+   The sheet 'bar' will have column headers 'foo', 'value', and 'unit', which are adequate to reconstruct the parameter.
+   However, the sheet 'baz' will have column headers 'foo' and 'another_dimension'; this information does not allow ixmp to infer that 'another_dimension' is indexed by 'foo'.
+
+   To work around this limitation, initialize 'baz' with the correct dimensions before reading its data::
+
+      new_scenario.init_par(
+          name='baz',
+          idx_sets=['foo', 'foo'],
+          idx_names=['foo', 'another_dimension'])
+      new_scenario.read_excel('file.xlsx', init_items=True)

ixmp/backend/base.py

@@ -4,6 +4,7 @@
 
 from ixmp.core import TimeSeries, Scenario
 from . import ItemType
+from .io import ts_read_file, s_read_excel, s_write_excel
 
 
 class Backend(ABC):
@@ -177,9 +178,30 @@ def open_db(self):
     def set_log_level(self, level):
         """OPTIONAL: Set logging level for the backend and other code.
 
+        The default implementation has no effect.
+
         Parameters
         ----------
         level : int or Python logging level
+
+        See also
+        --------
+        get_log_level
+        """
+
+    def get_log_level(self):
+        """OPTIONAL: Get logging level for the backend and other code.
+
+        The default implementation has no effect.
+
+        Returns
+        -------
+        str
+            Name of a :py:ref:`Python logging level <levels>`.
+
+        See also
+        --------
+        set_log_level
         """
 
     @abstractmethod
@@ -232,6 +254,12 @@ def read_file(self, path, item_type: ItemType, **kwargs):
         the `path` and `item_type` methods. For all other combinations, it
         **must** raise :class:`NotImplementedError`.
 
+        The default implementation supports:
+
+        - `path` ending in '.xlsx', `item_type` is ItemType.MODEL: read a
+          single Scenario given by kwargs['filters']['scenario'] from file
+          using :meth:`pandas.DataFrame.read_excel`.
+
         Parameters
         ----------
         path : os.PathLike
@@ -260,8 +288,13 @@ def read_file(self, path, item_type: ItemType, **kwargs):
         --------
         write_file
         """
-        # TODO move message_ix.core.read_excel here
-        raise NotImplementedError
+        s, filters = self._handle_rw_filters(kwargs.pop('filters', {}))
+        if path.suffix in ('.csv', '.xlsx') and item_type is ItemType.TS and s:
+            ts_read_file(s, path, **kwargs)
+        elif path.suffix == '.xlsx' and item_type is ItemType.MODEL and s:
+            s_read_excel(self, s, path, **kwargs)
+        else:
+            raise NotImplementedError
 
     def write_file(self, path, item_type: ItemType, **kwargs):
         """OPTIONAL: Write Platform, TimeSeries, or Scenario data to file.
@@ -270,6 +303,12 @@ def write_file(self, path, item_type: ItemType, **kwargs):
         the `path` and `item_type` methods. For all other combinations, it
         **must** raise :class:`NotImplementedError`.
 
+        The default implementation supports:
+
+        - `path` ending in '.xlsx', `item_type` is ItemType.MODEL: write a
+          single Scenario given by kwargs['filters']['scenario'] to file using
+          :meth:`pandas.DataFrame.to_excel`.
+
         Parameters
         ----------
         path : os.PathLike
@@ -289,8 +328,11 @@ def write_file(self, path, item_type: ItemType, **kwargs):
         --------
         read_file
         """
-        # TODO move message_ix.core.to_excel here
-        raise NotImplementedError
+        s, filters = self._handle_rw_filters(kwargs.pop('filters', {}))
+        if path.suffix == '.xlsx' and item_type is ItemType.MODEL and s:
+            s_write_excel(self, s, path)
+        else:
+            raise NotImplementedError
 
     @staticmethod
     def _handle_rw_filters(filters: dict):
@@ -355,6 +397,12 @@ def get(self, ts: TimeSeries, version):
         -------
         None
 
+        Raises
+        ------
+        ValueError
+            If :attr:`~.TimeSeries.model` or :attr:`~.TimeSeries.scenario` does
+            not exist on the Platform.
+
         See also
         --------
         ts_set_as_default
@@ -742,14 +790,19 @@ def item_get_elements(self, s: Scenario, type, name, filters=None):
             When *type* is 'set' and *name* an index set (not indexed by other
             sets).
         dict
-            When *type* is 'equ', 'par', or 'set' and *name* is scalar (zero-
+            When *type* is 'equ', 'par', or 'var' and *name* is scalar (zero-
             dimensional). The value has the keys 'value' and 'unit' (for 'par')
             or 'lvl' and 'mrg' (for 'equ' or 'var').
         pandas.DataFrame
             For mapping sets, or all 1+-dimensional values. The dataframe has
             one column per index name with dimension values; plus the columns
             'value' and 'unit' (for 'par') or 'lvl' and 'mrg' (for 'equ' or
             'var').
+
+        Raises
+        ------
+        KeyError
+            If *name* does not exist in *s*.
         """
 
     @abstractmethod