Feature/merge time series schemas (#1127)

* refactor: sync arg name Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: sync internal property name Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: flatten elif block Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: allow string deserialization and allow setting a default source unit for interpreting values without a unit Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: copy convert method Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: copy _serialize method Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: adapt _serialize method Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: adapt docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: update error message Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: update type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: merge schemas Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: reduce number of blank lines Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: deprecate old classes Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: duplicate import Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: document status quo Signed-off-by: F.N. Claessen <felix@seita.nl> * style: black Signed-off-by: F.N. Claessen <felix@seita.nl> * style: flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * feature: allow interpreting float values as quantities based on a unit defined elsewhere, and specifically, let the StorageScheduler get its default SoC unit from the soc-unit field, which lets us allow time series values to be specified as string quantities while preserving backwards compatibility Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: maintain backwards compatibility for transforming Float fields into Quantity fields, by returning the magnitude upon deserialization Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: grammar Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: add todo Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: tests Signed-off-by: F.N. Claessen <felix@seita.nl> * style: black Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: update V2G flex-model Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: update trigger endpoint examples Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: update other tutorials Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: update scheduling feature section Signed-off-by: F.N. Claessen <felix@seita.nl> * feat: convert schedule results ffrom MW to sensor unit Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: minimum list length for soc-gain and soc-usage Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: apply default SoC unit to all fields starting with "soc_" Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: move warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "refactor: move warning" This reverts commit 8cfe9fd. * fix: update test Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: move deserialization logic to dedicated class methods Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: fix comment about converting time series using to_unit Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: rename new schema Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: sync code order: 1) Sensor, 2) time series, 3) Quantity Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: add test and add missing logic for handling a list of dictionaries representing a time series Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: check for real numeric values Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entries Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: fix typo Signed-off-by: F.N. Claessen <felix@seita.nl> * style: be more explicit about requiring the soc-unit field to be set Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: mention the new Marshmallow field Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: advise setting a unit per field explicitly Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: simplify if statement Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: move soc-unit guesswork into schema Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: Victor Garcia Reolid <victor@seita.nl>
FlexMeasures · Aug 16, 2024 · a128275 · a128275
1 parent 9f12e30
commit a128275
Show file tree

Hide file tree

Showing 20 changed files with 658 additions and 258 deletions.
diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -10,6 +10,33 @@ v3.0-19 | 2024-08-13
 
 -  Allow posting a single instantaneous belief as a list of one element to `/sensors/data` (POST).
 
+v3.0-19 | 2024-08-09
+""""""""""""""""""""
+
+- Allow setting a SoC unit directly in some fields (formerly ``Float`` fields, and now ``Quantity`` fields), while still falling back on the contents of the ``soc-unit`` field, for backwards compatibility:
+
+  - ``soc-at-start``
+  - ``soc-min``
+  - ``soc-max``
+
+- Allow setting a unit directly in fields that already supported passing a time series:
+
+  - ``soc-maxima``
+  - ``soc-minima``
+  - ``soc-targets``
+
+- Allow passing a time series in fields that formerly only accepted passing a fixed quantity or a sensor reference:
+
+  - ``power-capacity``
+  - ``consumption-capacity``
+  - ``production-capacity``
+  - ``charging-efficiency``
+  - ``discharging-efficiency``
+  - ``storage-efficiency``
+  - ``soc-gain``
+  - ``soc-usage``
+
+
 v3.0-18 | 2024-03-07
 """"""""""""""""""""
 

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -12,11 +12,11 @@ New features
 * Add basic sensor info to sensor page [see `PR #1115 <https://github.com/FlexMeasures/flexmeasures/pull/1115>`_]
 * Support zoom-in action on the asset and sensor charts [see `PR #1130 <https://github.com/FlexMeasures/flexmeasures/pull/1130>`_]
 * Added Primary and Secondary colors to account for white-labelled UI themes [see `PR #1137 <https://github.com/FlexMeasures/flexmeasures/pull/1137>`_]
- 
+* Introduce the ``VariableQuantityField`` to allow three ways of passing a variable quantity in most of the ``flex-model`` and ``flex-context`` fields [see `PR #1127 <https://github.com/FlexMeasures/flexmeasures/pull/1127>`_]
 
 Infrastructure / Support
 ----------------------
-* Support new single-belief fast track (looking uup only one belief) [see `PR #1067 <https://github.com/FlexMeasures/flexmeasures/pull/1067>`_]
+* Support new single-belief fast track (looking up only one belief) [see `PR #1067 <https://github.com/FlexMeasures/flexmeasures/pull/1067>`_]
 * Add new annotation types: ``"error"`` and ``"warning"`` [see `PR #1131 <https://github.com/FlexMeasures/flexmeasures/pull/1131>`_]
 * Removed deprecated ``app.schedulers`` and ``app.forecasters`` (use ``app.data_generators["scheduler"]`` and ``app.data_generators["forecaster"]`` instead) [see `PR #1098 <https://github.com/FlexMeasures/flexmeasures/pull/1098/>`_]
 

diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst
@@ -108,25 +108,26 @@ and what constraints or preferences should be taken into account.
      - Example value
      - Description 
    * - ``soc-at-start``
-     - ``"3.1"``
+     - ``"3.1 kWh"``
      - The (estimated) state of charge at the beginning of the schedule (defaults to 0).
    * - ``soc-unit``
      - ``"kWh"`` or ``"MWh"``
-     - The unit in which all SoC related flex-model values are to be interpreted.
+     - The unit used to interpret any SoC related flex-model value that does not mention a unit itself (only applies to numeric values, so not to string values).
+       However, we advise to mention the unit in each field explicitly (for instance, ``"3.1 kWh"`` rather than ``3.1``).
    * - ``soc-min``
-     - ``"2.5"``
+     - ``"2.5 kWh"``
      - A constant lower boundary for all values in the schedule (defaults to 0).
    * - ``soc-max``
-     - ``"7"``
+     - ``"7 kWh"``
      - A constant upper boundary for all values in the schedule (defaults to max soc target, if provided)
    * - ``soc-minima``
-     - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: 8.2}]``
+     - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: "8.2 kWh"}]``
      - Set point(s) that form lower boundaries, e.g. to target a full car battery in the morning. Can be single values or a range (defaults to NaN values).
    * - ``soc-maxima``
-     - ``{"value": 51, "start": "2024-02-05T12:00:00+01:00","end": "2024-02-05T13:30:00+01:00"}``
+     - ``{"value": "51 kWh", "start": "2024-02-05T12:00:00+01:00", "end": "2024-02-05T13:30:00+01:00"}``
      - Set point(s) that form upper boundaries at certain times. Can be single values or a range (defaults to NaN values).
    * - ``soc-targets``
-     - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: 3.2}]``
+     - ``[{"datetime": "2024-02-05T08:00:00+01:00", value: "3.2 kWh"}]``
      - Exact set point(s) that the scheduler needs to realize (defaults to NaN values).
    * - ``soc-gain``
      - ``.1kWh`` 

diff --git a/documentation/tut/flex-model-v2g.rst b/documentation/tut/flex-model-v2g.rst
@@ -33,9 +33,8 @@ Constraining the cycling to occur within a static 25-85% SoC range can be modell
 
     {
         "flex-model": {
-            "soc-min": 15,
-            "soc-max": 51,
-            "soc-unit": "kWh"
+            "soc-min": "15 kWh",
+            "soc-max": "51 kWh"
         }
     }
 
@@ -50,16 +49,15 @@ To enable a temporary target SoC of more than 85% (for car reservations, see the
 
     {
         "flex-model": {
-            "soc-min": 15,
-            "soc-max": 60,
+            "soc-min": "15 kWh",
+            "soc-max": "60 kWh",
             "soc-maxima": [
                 {
-                    "value": 51,
+                    "value": "51 kWh",
                     "start": "2024-02-04T10:35:00+01:00",
                     "end": "2024-02-05T04:25:00+01:00"
                 }
-            ],
-            "soc-unit": "kWh"
+            ]
         }
     }
 
@@ -80,30 +78,52 @@ Given a reservation for 8 AM on February 5th, constraint 2 can be modelled throu
         "flex-model": {
             "soc-minima": [
                 {
-                    "value": 57,
+                    "value": "57 kWh",
                     "datetime": "2024-02-05T08:00:00+01:00"
                 }
             ]
         }
     }
 
 This constraint also signals that if the car is not plugged out of the Charge Point at 8 AM, the scheduler is in principle allowed to start discharging immediately afterwards.
-To make sure the car remains at 95% SoC for some time, additional soc-minima constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM:
+To make sure the car remains at or above 95% SoC for some time, additional soc-minima constraints should be set accordingly, taking into account the scheduling resolution (here, 5 minutes). For example, to keep it charged (nearly) fully until 8.15 AM:
 
 .. code-block:: json
 
     {
         "flex-model": {
             "soc-minima": [
                 {
-                    "value": 57,
+                    "value": "57 kWh",
                     "start": "2024-02-05T08:00:00+01:00",
                     "end": "2024-02-05T08:15:00+01:00"
                 }
             ]
         }
     }
 
+The car may still charge and discharge within those 15 minutes, but it won't go below 95%.
+Alternatively, to keep the car from discharging altogether during that time, limit the ``production-capacity`` (likewise, use the ``consumption-capacity`` to prevent any charging):
+
+.. code-block:: json
+
+    {
+        "flex-model": {
+            "soc-minima": [
+                {
+                    "value": "57 kWh",
+                    "datetime": "2024-02-05T08:00:00+01:00"
+                }
+            ],
+            "production-capacity": [
+                {
+                    "value": "0 kW",
+                    "start": "2024-02-05T08:00:00+01:00",
+                    "end": "2024-02-05T08:15:00+01:00"
+                }
+            ]
+        }
+    }
 
 .. _earning_by_cycling:
 

diff --git a/documentation/tut/forecasting_scheduling.rst b/documentation/tut/forecasting_scheduling.rst
@@ -108,18 +108,17 @@ Here, we extend that (storage) example with an additional target value, represen
     {
         "start": "2015-06-02T10:00:00+00:00",
         "flex-model": {
-            "soc-at-start": 12.1,
-            "soc-unit": "kWh"
+            "soc-at-start": "12.1 kWh",
             "soc-targets": [
                 {
-                    "value": 25,
+                    "value": "25 kWh",
                     "datetime": "2015-06-02T16:00:00+00:00"
                 }
         }
     }
 
 
-We now have described the state of charge at 10am to be ``12.1``. In addition, we requested that it should be ``25`` at 4pm.
+We now have described the state of charge at 10am to be ``"12.1 kWh"``. In addition, we requested that it should be ``"25 kWh"`` at 4pm.
 For instance, this could mean that a car should be charged at 90% at that time.
 
 If FlexMeasures receives this message, a scheduling job will be made and put into the queue. In turn, the scheduling job creates a proposed schedule. We'll look a bit deeper into those further down in :ref:`getting_schedules`.

diff --git a/documentation/tut/posting_data.rst b/documentation/tut/posting_data.rst
@@ -287,8 +287,7 @@ The endpoint also allows to limit the flexibility range and also to set target v
         {
             "start": "2015-06-02T10:00:00+00:00",
             "flex-model": {
-                "soc-at-start": 12.1,
-                "soc-unit": "kWh"
+                "soc-at-start": "12.1 kWh"
             }
         }
 

diff --git a/flexmeasures/api/v3_0/sensors.py b/flexmeasures/api/v3_0/sensors.py
@@ -279,8 +279,7 @@ def trigger_schedule(
             {
                 "start": "2015-06-02T10:00:00+00:00",
                 "flex-model": {
-                    "soc-at-start": 12.1,
-                    "soc-unit": "kWh"
+                    "soc-at-start": "12.1 kWh"
                 }
             }
 
@@ -311,17 +310,16 @@ def trigger_schedule(
                 "start": "2015-06-02T10:00:00+00:00",
                 "duration": "PT24H",
                 "flex-model": {
-                    "soc-at-start": 12.1,
-                    "soc-unit": "kWh",
+                    "soc-at-start": "12.1 kWh",
                     "soc-targets": [
                         {
-                            "value": 25,
+                            "value": "25 kWh",
                             "datetime": "2015-06-02T16:00:00+00:00"
                         },
                     ],
                     "soc-minima": {"sensor" : 300},
-                    "soc-min": 10,
-                    "soc-max": 25,
+                    "soc-min": "10 kWh",
+                    "soc-max": "25 kWh",
                     "charging-efficiency": "120%",
                     "discharging-efficiency": {"sensor": 98},
                     "storage-efficiency": 0.9999,

diff --git a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py
@@ -46,7 +46,7 @@ def test_get_schedule_wrong_job_id(
             message_for_trigger_schedule(),
             "soc-min",
             "not-a-float",
-            "Not a valid number",
+            "Not a valid quantity",
         ),
         (message_for_trigger_schedule(), "soc-unit", "MWH", "Must be one of"),
         # todo: add back test in case we stop grandfathering ignoring too-far-into-the-future targets, or amend otherwise

diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py
@@ -62,7 +62,7 @@
     LongitudeField,
     SensorIdField,
     TimeIntervalField,
-    QuantityOrSensor,
+    VariableQuantityField,
 )
 from flexmeasures.data.schemas.sources import DataSourceIdField
 from flexmeasures.data.schemas.times import TimeIntervalSchema
@@ -1171,7 +1171,7 @@ def create_schedule(ctx):
 @click.option(
     "--site-power-capacity",
     "site_power_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Site consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1181,7 +1181,7 @@ def create_schedule(ctx):
 @click.option(
     "--site-consumption-capacity",
     "site_consumption_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Site consumption power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1191,7 +1191,7 @@ def create_schedule(ctx):
 @click.option(
     "--site-production-capacity",
     "site_production_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Site production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1256,7 +1256,7 @@ def create_schedule(ctx):
 @click.option(
     "--charging-efficiency",
     "charging_efficiency",
-    type=QuantityOrSensor("%"),
+    type=VariableQuantityField("%"),
     required=False,
     default=None,
     help="Storage charging efficiency to use for the schedule."
@@ -1266,7 +1266,7 @@ def create_schedule(ctx):
 @click.option(
     "--discharging-efficiency",
     "discharging_efficiency",
-    type=QuantityOrSensor("%"),
+    type=VariableQuantityField("%"),
     required=False,
     default=None,
     help="Storage discharging efficiency to use for the schedule."
@@ -1276,7 +1276,7 @@ def create_schedule(ctx):
 @click.option(
     "--soc-gain",
     "soc_gain",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Specify the State of Charge (SoC) gain as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1286,7 +1286,7 @@ def create_schedule(ctx):
 @click.option(
     "--soc-usage",
     "soc_usage",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Specify the State of Charge (SoC) usage as a quantity in power units (e.g. 1 MW or 1000 kW) "
@@ -1296,7 +1296,7 @@ def create_schedule(ctx):
 @click.option(
     "--storage-power-capacity",
     "storage_power_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Storage consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1306,7 +1306,7 @@ def create_schedule(ctx):
 @click.option(
     "--storage-consumption-capacity",
     "storage_consumption_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Storage consumption power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1316,7 +1316,7 @@ def create_schedule(ctx):
 @click.option(
     "--storage-production-capacity",
     "storage_production_capacity",
-    type=QuantityOrSensor("MW"),
+    type=VariableQuantityField("MW"),
     required=False,
     default=None,
     help="Storage production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
@@ -1326,7 +1326,7 @@ def create_schedule(ctx):
 @click.option(
     "--storage-efficiency",
     "storage_efficiency",
-    type=QuantityOrSensor("%", default_src_unit="dimensionless"),
+    type=VariableQuantityField("%", default_src_unit="dimensionless"),
     required=False,
     default="100%",
     help="Storage efficiency (e.g. 95% or 0.95) to use for the schedule,"
@@ -1469,9 +1469,9 @@ def add_schedule_for_storage(  # noqa C901
                 else:
                     unit = "MW"
 
-                scheduling_kwargs[key][field_name] = QuantityOrSensor(unit)._serialize(
-                    value, None, None
-                )
+                scheduling_kwargs[key][field_name] = VariableQuantityField(
+                    unit
+                )._serialize(value, None, None)
 
     if as_job:
         job = create_scheduling_job(asset_or_sensor=power_sensor, **scheduling_kwargs)

diff --git a/flexmeasures/conftest.py b/flexmeasures/conftest.py
@@ -819,6 +819,19 @@ def create_test_battery_assets(
     )
     db.session.add(test_battery_sensor)
 
+    test_battery_sensor_kw = Sensor(
+        name="power (kW)",
+        generic_asset=test_battery,
+        event_resolution=timedelta(minutes=15),
+        unit="kW",
+        attributes=dict(
+            daily_seasonality=True,
+            weekly_seasonality=True,
+            yearly_seasonality=True,
+        ),
+    )
+    db.session.add(test_battery_sensor_kw)
+
     test_battery_no_prices = GenericAsset(
         name="Test battery with no known prices",
         owner=setup_accounts["Prosumer"],