handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible)

[p-glaum]

closes #626

With this PR, we want to unify the way how new and upgraded lines are handled in PyPSA-Eur in a more generalized manner. This PR incorporates the work done in #626 and makes them compatible with Gridkit data and OSM.

There are still some issues which need to be resolved first.

To-dos and issues:

• how to handle upgraded lines in overnight optimization
  • currently we decided to drop upgraded lines for overnight optimization
• make sure that clustering does not aggregate lines with different build years

Closes # (if applicable).

Changes proposed in this Pull Request

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in all of config.default.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv.
• A release note doc/release_notes.rst is added.

[fneum]

Before I review this PR in detail, can I request the following:

• With the new section in config.default.yaml called line_projects:, I think the setting links: include_tyndp is not used anymore and should be removed.

• The settings under_construction for lines and links refer to the TYNDP projects and should be moved to the new section (which I would rename to transmission_projects since it affects not only lines).

• If you think the configuration settings could differ for lines and links, you could duplicate the line_projects: under links: and lines: each.

• Since tyndp and nep are only used for the path of the files, one could generalise the config section (considering also previous suggestions)

transmission_projects:
  enable: true
  include:
  - nep
  - tyndp
  status:
  - confirmed
  - in_permitting
  - under_construction
  line_under_construction: keep
  link_under_construction: zero

• Update the documentation of the new configuration settings.

• Remove the scripts around the rule prepare_links_p_nom, since this seems to be superseded by the new code.

  • p-glaum: Not so sure because I this is mainly needed for the Gridkit links because they have all no capacity given.
  • fneum: done in remove unused rule prepare_links_p_nom #1203

• Since processing the planned projects requires quite a bit of code, consider moving it into a separate rule (e.g. build_transmission_projects). I don't think there would be an issue adding the planned projects only later in add_electricity.

• Upgraded lines in overnight optimisation should just increase the p_nom_min I think with already adjusted reactance x.

  • p-glaum: Not quite sure if I understand correctly. So you mean instead of adding new upgraded line and filtering the active components?

• Ensure and double-check that clustering does not aggregate lines with different build years. It should not be difficult to add this logic in simplify_network and cluster_network.

  • I think there should not be an issue with simplify network because only lines are removed but not clustered. However, in cluster_network currently all lines are grouped together. One could add a column to the grouper, e.g. "build_year", that grouping of lines with different build years does not happen.

• Are potentially duplicated projects in NEP and TYNDP or base network removed / taken account of?

  • currently, similar as before with include_tyndp, lines and links are automatically checked whether they already exist in the network. If yes, those line projects will not be added.

• Present a visual comparison of added projects in base.nc before and after (see e.g. Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079)

[p-glaum]

Hi @fneum , this PR is not up to date anymore. I had to revert some changes because I already wanted to make it compatible with @bobbyxng PR. I guess it is too early to review.

[fneum]

Hi @fneum , this PR is not up to date anymore. I had to revert some changes because I already wanted to make it compatible with @bobbyxng PR. I guess it is too early to review.

Alright, no problem. Let me know when I should give it another go.

[p-glaum]

Maps to investigate different transmission line projects:

• all: includes manual, tyndp and nep transmission project
• tyndp
• manual
• nep
  The projects are always compared to the default gridkit network.
  default_.zip

[fneum]

CI is failing because minimum PyPSA version needs to be bumped to v0.29 @p-glaum.

[fneum]

Visual comparison looks good. I think this is close to being ready to be merged. Some code style suggestions and questions:

• The setting links: under_construction: and lines: under_construction: are still accessed in rule base_network in _adjust_capacities_of_under_construction_branches(). Perhaps keep these options for now, stating that they refer to the status of the transmission assets contained in the GridKit extract.
• Rename script and rule to build_transmission_projects, since they are not "added" to the network.
• In simplify_network, need to remove project_status since it may cause inconsistency errors when trying to clusters (may not aggree). (

  pypsa-eur/scripts/simplify_network.py

  Line 593 in ea81fe1

  # and are lost when clustering (for example with the simpl wildcard), we remove them for consistency:

  )
• For NEP, it's not stated whether projects are underground or not, leading to NaN values later in elec.nc. Is this information available?
  • PG: Unfortunately not in the pdf, where I extract the lines from. But if I remember correctly, the HVDC lines are constructed underground. So I set underground to 1.
• Upgraded lines in overnight scenarios: I would disregard this first. It's just a few lines. A good intermediate option would be some control in config.yaml:

transmission_projects:
  skip:
  - upgraded_lines
  - ...

[fneum]

• potentially requires update according to Updated under_construction status of links that are commissioned by now #1205

(edit: this is actually not relevant)