Feature ehighways resolution unts 318 #336

jonathan-peel · 2024-04-10T07:39:30Z

Fixes #318 .

Checklist

Any checks which are not relevant to the PR can be pre-checked by the PR creator. All others should be checked by the reviewer. You can add extra checklist items here if required by the PR.

…tion-unts-318

for more information, see https://pre-commit.ci

…n files.

…on files and schemas.

Now need to implement the actual function properly.

Debug configuration set up also.

…ability.

…nathan-peel/euro-calliope into feature-ehighways-resolution-unts-318

for more information, see https://pre-commit.ci

References to nuts-year "variable" still exist.

…for all Europe (except Kosovo)

for more information, see https://pre-commit.ci

jnnr · 2024-04-12T13:39:14Z

A a note: ehighways uses NUTS3 2006 (checked by comparing the region names in anderski_2014 with `statistal-to-ehighways-units.csv)

for more information, see https://pre-commit.ci

timtroendle

Awesome! It looks good, but it requires a few changes, like type hints, documentation, and solving some inconsistencies.

The one thing I am not liking stems from the source: There are now two different ways in which resolutions are defined: the previously-existing shape config in the config AND the new statistical-to-custom-units file. Ideally, there would be only one place.

I don't think this should be handled within this PR or ticket. But I would like to keep this in mind and consider solving in the future. @brynpickering, do you agree?

timtroendle · 2024-04-13T15:04:26Z

scripts/shapes/units.py

Many functions in this file lack type hints.

timtroendle · 2024-04-13T15:06:44Z

config/default.yaml

@@ -39,6 +39,9 @@ data-pre-processing:
            MNE: [HRV]
            SRB: [HUN]
            CYP: [ROU]
+    statistical-to-custom-units:
+        # FYI: E-highways were initially defined using NUTS3 units from 2006, but this mapping was manually created to map NUTS units from any year up to 2016 to E-highways. This is picked up by rule `units` to build custom spatial units.


This comments seams out of context here. I don't know where this should go, but this looks like the wrong place to me.

timtroendle · 2024-04-13T15:07:11Z

config/default.yaml

@@ -314,6 +317,7 @@ scope:
        first-year: 2016
        final-year: 2016
 shapes: # This config must be consistent with data from https://doi.org/10.5281/zenodo.3244985.
+    # TODO: update this link to Bryn's new one, which includes the potentials for ehighways


Please update the link

timtroendle · 2024-04-13T15:08:03Z

config/default.yaml

+        Norway: nuts3
+        Serbia: gadm0
+        Switzerland: nuts3
+        Iceland: nuts0


Iceland is not currently supported by Euro-Calliope, see #15.

timtroendle · 2024-04-13T15:08:29Z

config/schema.yaml

+                type: object
+                description: >-
+                    Path to a csv file which defines how lower-level statistical units
+                    (i.e., NUTS and GADM) are combined to form units of a custom


GADM are not statistical units.

timtroendle · 2024-04-13T15:28:25Z

scripts/shapes/units.py

+    function.
+
+    Inputs:
+    - path_to_statistical_to_custom_units: csv file that specifies which NUTS and GADM


Here again, there should be no statistical in the name if you want to be more flexible than eHighways.

timtroendle · 2024-04-13T15:28:52Z

scripts/shapes/units.py

+    - layer: the GeoDataFrame covering the whole scope, where each country is formed
+        from units corresponding to the resolution specified in resolution_config
+    Outputs:
+    -


Output is missing.

timtroendle · 2024-04-13T15:29:13Z

scripts/shapes/units.py

+    -
+
+    Assumes:
+    - the base-level statistical units in layer and are either NUTS3 or whole countries (NUTS0 or GADM0)


GADM is not... you know what. :D

timtroendle · 2024-04-13T15:30:28Z

config/schema.yaml

@@ -377,6 +394,10 @@ properties:
                type: string
                description: Nuclear technology capacity scenario. "current" will use today's capacities based on the JRC power plant database (PPDB). All other scenario names should point to a configuration CSV file specifying minimum and maximum capacities per country in MW with non-zero capacities (columns=[country, min, max]).
                pattern: ^(current)|(\w+.csv)$
+            nuts-year:


This parameter does not exist in the config. Please remove.

timtroendle · 2024-04-13T15:31:13Z

scripts/shapes/units.py

+    """
+
+    stat_to_custom_units_df = pd.read_csv(path_to_statistical_to_custom_units, header=0)
+    nuts_year = 2013  # This was a variable in Bryn's SC EC, but hardcoded to 2013 in EC. Consider that the year needs to match for both the (downloaded nuts units) and (nuts to custom region mapping)


Make this a constant at the top of the file.

I'd remove the first sentence. Alternatively, point to the line in the repo. "Bryn's SC EC" will not make sense to everyone and forever.

brynpickering · 2024-04-25T12:59:15Z

I don't think this should be handled within this PR or ticket. But I would like to keep this in mind and consider solving in the future. @brynpickering, do you agree?

I don't think it's necessarily a problem. You have standardised boundaries coming from various data sources (nuts, gadm, ...) and they are your building blocks in shapes. Then you have a mechanism to merge those shapes together with a custom mapping. It means we know that all shapes in the model are built from standard boundaries and there's no need for a mechanism to injest custom shapes.

brynpickering

Not sure what the issue is with setting a nuts year at the config level and then downloading that nuts year when generating any units (as I did in SC-EC). Since we only use NUTS for ehighways at the moment it effectively allows you to choose the NUTS year that will be used to create those shapes.

We'll need the ability at a later point to download arbitrary NUTS years (for subnationalising using Eurostat datasets) so I guess we can leave it until then to introduce it as a feature.

brynpickering · 2024-04-25T12:33:55Z

scripts/shapes/units.py

+        # sum all of the units together into one block
+        units = _continental_layer(units)
+    elif path_to_statistical_to_custom_units != []:
+        # this code is only executed if there is a [statistical-to-custom-units file corresponding to the chosen resolution] in the config


You should get the same result with elif path_to_statistical_to_custom_units:. An empty list resolves to False and a non-empty noe resolves to True in these kinds of conditional statements

jonathan-peel and others added 22 commits April 9, 2024 17:23

Initial step: copy across code from sector-coupled-euro-calliope

e293175

Added nuts-year into config and schema.

a1d6d53

Merge branch 'calliope-project:develop' into feature-ehighways-resolu…

6e9a803

…tion-unts-318

[pre-commit.ci] auto fixes from pre-commit.com hooks

e684d78

for more information, see https://pre-commit.ci

Added config options and updated custom_units rule

f91e811

rule custom units now runs successfully with all inputs defined

7d0a9af

rule units starts successfully. Pre-commit before restructuring Pytho…

4902ced

…n files.

Debugging snakemake rule to run properly. Mostly changing configurati…

bba893f

…on files and schemas.

Code enters function successfully.

ebd4bb4

Now need to implement the actual function properly.

Small code documentation changes.

1c8d34e

remove whitespace

97ad475

Refactoring for consistency; adding docstrings.

3a26c53

Entrance into units.py via shapes.smk working.

07db593

Debug configuration set up also.

Merged first function of custom units with input validation.

18ed036

1st build of custom units working but buggy due to nuts year incompat…

18582c5

…ability.

Adding files from Sector-Coupled Euro-Calliope for comparison.

da9c0cb

Merge branch 'feature-ehighways-resolution-unts-318' of github.com:jo…

742595f

…nathan-peel/euro-calliope into feature-ehighways-resolution-unts-318

[pre-commit.ci] auto fixes from pre-commit.com hooks

63dc596

for more information, see https://pre-commit.ci

Successfully build ehighways units using 2013 NUTS units.

e7fd29b

References to nuts-year "variable" still exist.

Removed references to nuts years. ehighways still build successfully …

8c2ac3a

…for all Europe (except Kosovo)

[pre-commit.ci] auto fixes from pre-commit.com hooks

85bba84

for more information, see https://pre-commit.ci

Removing reference files and cleaning debug configurations.

d6654b0

jonathan-peel marked this pull request as ready for review April 12, 2024 12:49

jonathan-peel and others added 3 commits April 12, 2024 14:55

Small comment

fac9b2c

[pre-commit.ci] auto fixes from pre-commit.com hooks

371913a

for more information, see https://pre-commit.ci

Refactoring of layer -> resolution

b296ec8

jnnr mentioned this pull request Apr 12, 2024

Build model in ehighways resolution #353

Closed

10 tasks

jonathan-peel and others added 2 commits April 12, 2024 16:32

Documenting statistical-to-custom-units

e227ce8

[pre-commit.ci] auto fixes from pre-commit.com hooks

f2fa62d

for more information, see https://pre-commit.ci

timtroendle requested changes Apr 13, 2024

View reviewed changes

brynpickering reviewed May 2, 2024

View reviewed changes

brynpickering mentioned this pull request May 3, 2024

Update ehighways resolution shape generation #370

Merged

5 tasks

timtroendle merged commit 52d34ac into calliope-project:develop May 14, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature ehighways resolution unts 318 #336

Feature ehighways resolution unts 318 #336

jonathan-peel commented Apr 10, 2024

jnnr commented Apr 12, 2024

timtroendle left a comment

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

timtroendle Apr 13, 2024

brynpickering commented Apr 25, 2024

brynpickering left a comment

brynpickering Apr 25, 2024

Feature ehighways resolution unts 318 #336

Feature ehighways resolution unts 318 #336

Conversation

jonathan-peel commented Apr 10, 2024

Checklist

jnnr commented Apr 12, 2024

timtroendle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brynpickering commented Apr 25, 2024

brynpickering left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment