Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079

bobbyxng · 2024-05-22T14:30:01Z

Changes proposed in this Pull Request

This pull request introduces a first functioning implementation of the high voltage-electricity grid (incl. 200 kV and above) extracted from OpenStreetMap (OSM, https://www.openstreetmap.org). The implementation is proposed in such a way that the user can freely determine whether to use the existing electricity network based on the GridKit extract of the ENTSO-E online map or based on OSM.

Components/scripts

retrieve_osm_data.py: Retrieves OSM data for the specified country using the overpass API and save it
to the specified output (json) files. Note that overpass requests are based on a fair use policy. retrieve_osm_data is meant to be used in a way that respects this policy by fetching the needed data once, only.
clean_osm_data.py: The script performs various cleaning operations on the OSM data, including:
- Cleaning voltage, circuits, cables, wires, and frequency columns
- Splitting semicolon-separated cells into new rows
- Determining missing parameters such as number of circuits heuristically, if such information is missing.
- Adding line endings to substations based on line data
build_osm_network.py: Modified implementation of PyPSA-Earth's build_osm_network.py script, which creates a connected network, by
- aggregating close buses and setting bus ids
- connecting cleaned lines to buses, respectively
- and finally creating PyPSA-Eur friendly input files (.csv / .geojson for quick debugging and visualisation) in order to create the base network base.nc

Checklist

I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in all of config.default.yaml.
Changes in configuration options are also documented in doc/configtables/*.csv.
A release note doc/release_notes.rst is added.

Known issues and simplifications

While DC links are also extracted from OSM, their power rating is unfortunately not consistently included in the raw data. As such, p_nom is currently calculated using the extracted voltage and the two DC line types available in PyPSA. In the future, we will determine the correct p_nom, e.g. by filling in the missing data to OSM, directly.
Currently, two implementations of base_network.py (and base_network_osm.py, respectively) exist. This is partly related to how links are currently being handled in the OSM implementation. In the near future, this will be consolidated, so only one base_network.py for both GridKit and OSM-sourced electricity networks is needed.
TYNDP and NEP projects are not implemented yet. This feature is under development.

Benchmarks and validation

Benchmarks (exemplary for Europe retion)

retrieve_osm_data.py: 03:01 min (100 Mbit/s connection)
clean_osm_data.py: 01:04 min
build_osm_network.py: 02:08 min
base_network.py: 00:18 min

Comparison of results/validation

Comparing route and circuit lengths against official ENTSO-E data (Inventory of transmission, https://www.entsoe.eu/data/power-stats/)

Base network overview

Line volume per NUTS1 region

RMSE (PyPSA-Earth): $1.156 * 10^6$
RMSE (PyPSA-Eur): $0.834 * 10^6$

Cross-regional capacity between NUTS1 regions

![scatterplot_crossregion_capacity](https://github.com/PyPSA/pypsa-eur/assets/36541459/cc8b5dc6-06c2-47a3-81

23-6d9daa0810ae)

RMSE (PyPSA-Earth): $5.659* 10^3$
RMSE (PyPSA-Eur): $3.448 * 10^3$

Delta in electricity generation per country

Transmission (25h resolution, full-year run, no TYNDP projects, no line expansion)

…r individual countries.

…m_data.py.

…ons.

into osm-network-fast Added build_osm_network

…ownloads.

…licy and potential request errors.

…d code. Speed improvements

for more information, see https://pre-commit.ci

into osm-network-fast

… connections.

…ation.

into osm-network-fast

…ed converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with PyPSA#1079 and PyPSA#1085

bobbyxng · 2024-08-22T09:30:19Z

@fneum I have finished implementing all proposals. I have also checked the compatibility with the latest geopandas version and @p-glaum 's PR #1085 on transmission-projects - see plots below, tested for a week, all optimisation problems converge. Note that the networks are clustered slightly differently and some links are missing in the ENTSO-E map/GridKit base network. The Zenodo repository for the prebuilt network was also updated to ensure compatibility with the latest PyPSA-Eur master.

From my side, this PR is ready to be merged.

scripts/base_network.py

scripts/build_osm_network.py

scripts/clean_osm_data.py

for more information, see https://pre-commit.ci

… osm-network-fast

* use module structure * change default DH return temperature to 50C * rename __main__ to run * update build_sector.smk * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove rural/urban temperature * remove cop_total input from prepare_sector_network * update base year COPs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update inputs for add_existing_baseyear * udpate add_brownfield COPs * add licensing info * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update release notes * add option to make central heating forward/return temperatures country-specific * ensure correct dimensions in forward/return temperature arayx and change naming from "generic" to "default" * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lifetime of Gas Pipelines (#1162) * inf lifetime of gas pipelines avoids adding new links each planning horizon * p_nom_extendable for existing gas pipelines is set false if retrofitting to H2 is not allowed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Rename ev battery master (#1116) * rename EV battery * add release note * adjust tech colors * fix for battery renaming in plot_summary --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * checkout sector configtable from master * update sector configtable * Update doc/configtables/sector.csv Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Update config/config.default.yaml Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * remove duplicate staticmethods * add link to paper in release notes * address groupby(axis=...) deprecation (#1182) * address fillna(method='{b|f}fill') deprecation (#1181) * address fillne(method='{b|f}fill') deprecation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * include Tonis Feedback * attempt to handle heat_pump_sources dynamically * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update industry to new JRC-idees * update ammonia data * retrieve ammonia demand data * change industry reference year to 2019 * update ammonia demand data to 2022 * adjust mapping energy demand per ct * pass heat source/system type to prepare_sector_network and add_existing_baseyear * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update industry sector ratios to new JRC data * small fixes in docstrings * add heat_pump_sources to config and docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * some small adjustments to run as single node model (#1183) * some small adjustments to run as single node model * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * use ffill and bfill * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix typo * update configtables and docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update class docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make test.sh executable * remove breakpoint * correct hyperlink in release notes * refactor prepare_sector_network.add_heat using Enums * some simplifications * add description for new config setting * add release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * naturalearth: automatically download and remove from data bundle * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change FRESNA references to PyPSA * enable electrobiofuels also without biomass spatially resolved * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add documentation for new config option * add release note * change units from EJ->TWh and clean up * add doc for new config * add release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix naming bugs * refactor naming * set e_max_pu to force municipal waste to be used * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug with e_max_pu * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * udpate docs * Refactor module structure * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update naming in add_existing_baseyear * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * base_network: bugfix - bring voronoi cells in correct order * add switch for msw waste * add release notes and doc for new config settings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Corrected Moyle Interconnector capacity in links_p_nom.csv to 500 MW (#1199) * update myopic mode to heat system declarations * update add_existing_baseyear * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update copyright notic in heatSector * fix boiler buses for existing heating * update solve_perfect.add_existing_baseyear input/params * remove obsolote cop inputs from solve_perfect.add_existing_baseyear * rename "enums" to "definitions" and write modules in snake case * rename "enums" to "definitions" and write modules in snake case * clean up, improve docs * save idees directly in data not in bundle Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * remove ammonia demand script * remove comment about uk flights * fix release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix intersphinx mapping * remove /bundle * rename eurostat_2015 * retrieve.smk: consistently use conda directive * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * include review fneum build_industrial_production_per_country * create previous idees split for aviation * directly read in nan values * add ch data as csv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update config and script * remove not needed function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * re-add default supply temperature * remove unused rule `prepare_links_p_nom` (#1203) * remove rule * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * EEZ: Update EEZ to v12, auto-download and remove from databundle (#1188) * eez: update to version 12, autodownload, remove pycountry * eez: do not simplify as it distorts topology * remove missed merge conflicts * Addition of unsustainable biomass potentials (#1139) * add columns to potential df defined by difference to eurostat * add network components * add unsustainable bioliquids * replaced stores by generators, still infeasible * remove municipal waste * remove separate treatment of waste from biomass potential calculation * phase out unsustainble biomass potentials * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * phase-out unsustainable bioliquids * remove waste_incineration from build_sector rule * multiple potential calculation for different planning horizons * raised costs of unsustainable solid biomass * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * stores instead of generators * change snakemake inputs * add phas-eout to config * add techcolor for unsustainable bioliquids * add config parameter to disable inclusion of unsustainable bioenergy potentials * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add biomass to params * remove call of snakemake object in define_spatial * Quick resolve of review part 1 (config parameters, if-clause-reduction, bioliquid spatial, fix bioliquid link capacity * Quick resolve of review part 2 (config table, helper function, fixed build_eurostat, removed dir-change, forced unsustainable usage, reverted overnight distinction in Snakefile) * Cast of planning_horizon parameter to int type after test run * added JRC fuel costs for solid and liquid biofuels, BtL VOM * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean-up after master merge * adressed review (increase threads for build_eurostat, fixed e_max_pu of Stores, changed version of technology-data); added release note --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lisazeyen <35347358+lisazeyen@users.noreply.github.com> * use relative imports of scripts.definitions where possible * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add option to make central heating forward/return temperatures country-specific * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set lower supply temperatures for Scandinavia * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update configtables * update release notes * correct sheet name chemical feedstocks * add coke oven transformation output * add docstring * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update doc/release_notes.rst Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Updated under_construction status of links that are commissioned by now (#1205) * Updated links 14802, 14801, 12931, 14803, 14848, 14804: under_construction=False * Updated links 14802, 14801, 14803, 14848, 14804: under_construction=False * Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link. * [pre-commit.ci] pre-commit autoupdate (#1208) updates: - [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](psf/black-pre-commit-mirror@24.4.2...24.8.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * style: move get_country_from_node_name function from _helpers tu run * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add option to use atlite to smooth wind turbine power curves (#1209) * use lower resolution EEZ shapes to reduce excessive RAM use (#1210) * Fix simplify_network.py to handle more complex topologies (#1211) * Fixed simplify_network.py to handle more complex topologies, i.e. if two links are connected to nearby AC buses separated by an AC line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix for Corsica in simplify_network: Include local substation (#1215) * Fixed simplify_network.py to handle more complex topologies, i.e. if two links are connected to nearby AC buses separated by an AC line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added fix for Corsica node: If region containing Corsica is modelled, include substation on Corsica as supernode (end point). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * bugfix: make sure to include Ukraine offshore shapes with new EEZ files * handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085) * add general implementation to add tyndp and nep. TODO: Remove duplicate? * pre_osm_merge * clean and update base_network.py update default config settings * clean and update base_network.py update default config settings * base_network.py:remove adding of transmission projects add_transmission_project.py: add new script for creating lines and link csv from transmission projects add_electricity.py: add new projects from created csv files * cluster_network.py: do not allow to group lines with different build years together-> requires pypsa update simplify_network.py: fix bug of simplify links * remove legacies of removing transmission projects from base_network * restructure folders:new folder for tranmission project csv add_transmission_projects: improve logging, cleanup * update lines csvs and add default line type for upgraded and new lines * remove duplicate lines which were already in gridkit * allow to connect ac lines only to ac buses * add manual links csv (adjusted links_tyndp.csv) and update default config * add realease note * remove links_tyndp.csv file, references in build_elec.smk and function in base_network.py to add them * add configuration options for transmission projects to documentation and add template folder for transmission projects * update pypsa version in environments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * integrate Fabian's review comments 2 * add electricity:add import pathlib and duplicate printing out of adding line projects * update NEP line csv * address Fabian's comments * build_transmission_projects: address Fabian's final comments simplify_network: use modus to get line type which appears most often * build_transmission_project: change from .geometry to ["geometry"] * build_transmission_projects: remove redundanty line * build_transmission_projects: remove buffer for europe shape because of higher resolution default config: fix wrong key for skip argument in transmission_projects * update configtables and default config * update manual links csv and delete undetected duplicate links in tyndp2020 * final adjustments --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * retrieve: remove cutout protection (#1220) * modified environments to include tabula-py update and fixed PDF econding (#1219) * Bug fix: Carrier type of added supernodes in simplify_network need to be set to "AC" (#1221) * Fixed simplify_network.py to handle more complex topologies, i.e. if two links are connected to nearby AC buses separated by an AC line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added fix for Corsica node: If region containing Corsica is modelled, include substation on Corsica as supernode (end point). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bug fix: Carrier type of all supernodes corrected to 'AC' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bug fix: Carrier type of all supernodes corrected to 'AC' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bug fix: Carrier type of all supernodes corrected to 'AC' --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix bug in simplyfy network (#1222) * Solid biomass to hydrogen process (#1213) * Added solid biomass to hydrogen, bioH2, process * Add doc for new config * add release notes * enable bioH2 also without biomass spatially resolved * Update doc/release_notes.rst Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Added overnight cost --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * increase min cluster to 38 because of corsica link * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * increase CH transport data until 2022 * bug fix for reference year<2018 for ammonia * build_transport_data: select swiss years according to data selection use index.unique instead of index.levels where needed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * avoid infinity in sector rations * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * co2 sequestration depending on period * update config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * right intend for co2 seq potential * update release notes and doc * Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) * Implemented which uses the overpass API to download power features for individual countries. * Extended rule by input. * Bug fixes and improvements to clean_osm_data.py. Added in retrieve_osm_data.py. * Updated clean_osm_data and retrieve_osm_data to create clean substations. * Finished clean_osm_data function. * Added check whether line is a circle. If so, drop it. * Extended build_electricity.smk by build_osm_network.py * Added build_osm_network * Working osm-network-fast * Bug fixes. * Finalised and cleaned including docstrings. * Added try catch to retrieve_osm_data. Allows for parallelisation of downloads. * Updated cleaning process. * Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors. * Intermediate update on clean_osm_data.py. Added docstrings. * Bug fix. * Bug fix. * Bug fixes in data types out of clean_osm_data * Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements * Cleaned config. * Fixes. * Bug fixes. * Updated default config * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed overpass from required packages. Not needed anymore. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Work-in-progress clean_osm_data * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * New code for integrating HVDC links. Using relations. Base network implementation functioning. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed manual line dropping. * Updated clean script * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reverted Snakefile to default: sync settings * added prebuilt functionality. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated build_electricity.smk to work with scenario management. * removed commented-out code. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed commented-out code. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed bug in pdf export by substituting pdf export with svg. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bug-fix Snakefile * dropped not needed columns from build_osm_network. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated build_shapes, config.default and clean_osm_data. * pre-commit changes. * test * Added initial prepare_osm_network_release.py script * Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files. * Added new rules/development.smk * Updated clean_osm_data to add substation_centroid to linestrings * Updated clean_osm_data to add substation_centroid to linestrings * Updated clean_osm_data to add substation_centroid to linestrings * Updated clean_osm_data to add substation_centroid to linestrings * Added osm-prebuilt functionality and zenodo sandbox repository. * Updated clean_osm_data to geopandas v.1.01 * Made base_network and build_osm_network function more robust for empty links. * Made base_network and build_osm_network function more robust for empty links. * Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica) * Merge with hcanges in upstream PR 1146. Fixing UA and MD. * Updated Zenodo and fixed prepare_osm_network_release * Updated osm network release. * Updated prepare osm network release. * Updated MD, UA scripts. * Cleaned determine_availability_matrix_MD_UA.py, removed redundant code * Bug fixes. * Bug fixes for UA MD scripts. * Rename of build script. * Bug fix: only distribute load to buses with substation. * Updated zenodo sandbox repository. * Updated config.default * Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. * Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes. * Updated Zenodo repositories for OSM-prebuilt to offcial publication. * Updated configtables * Updated links.csv: Under_construction lines to in commission. * Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link. * Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes. * remove config backup * Bug fix: Carrier type of all supernodes corrected to 'AC' * Bug fix: Carrier type of all supernodes corrected to 'AC' * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated rules and base_network for compatibility with TYNDP projects. * Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections. * Removed outdated config backup. * Implemented all comments from PR #1079. Cleaned up OSM implementation. * Bug fix: Added all voltages, 200 kV-750 kV, to default config. * Cleaning and bugfixes. * Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with #1079 and #1085 * Apply suggestions from code review * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * simplify_network: handle complicated transformer topologies * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * syntax fix --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * config: set correct minimum number of clusters * cluster_network: fix if no clustering required * Custom busmap (#1231) * Added functionality to set custom_busmaps for different base_networks. * Updated configtables and location for busmaps. * Added release nots. * removed run parameter in mock_snakemake (only used for debugging). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * add capital cost for liquid carbonaceous fuel stores (closes #489) (#1234) * add capital cost for liquid carbonaceous fuel stores (closes #489) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * simplify_network: do not use mode for linetype inferral * update GEM Europe gas tracker to May 2024 version (#1235) * update nuclear EAF based on 2021-2023 IAEA data (#1236) * build_electricity: raise memory for build_tranmission_projects (#1237) 2 GB RAM was not enough, to avoid out-of-memory problems raised to 4 GB. * temporarily disable n.shapes until memory issues resolved (#1238) * build_powerplants: can assign to all buses not just substations (#1239) * amend codespell blacklist * eurostat_co2: fix update to 2023 energy balances * change base_network default to 'osm-prebuilt' * prepare_sector_network: convert e_max_pu timeseries for msw stores to pd.DataFrame. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * additional merge changes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * retrieve urban population fraction from World Bank API (#1248) * retrieve urban population fraction from World Bank API * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * remove unused geth2015 hydro capacities (#1246) * build_ammonia_production: forward fill missing values * prepare_sector_network.py: add oil bus whenever industry is added (#1247) * prepare_sector_network.py: add oil bus whenever industry is added * prepare_sector_network: Move added oil and methnaol buses to top of "add_industry", and add oil buses where used in "add_biomass" * revert ffill of ammonia production * industry: steel GEM, ammonia, refinery, cement plants non-EU (#1241) * industry distribution: split EAF + integrated using GEM GSPT, add ammonia plants * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add release note [no ci] * add cement plant data for countries not in Hotmaps * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add refineries data for countries not in Hotmaps --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * prepare_sector_network: fix municipal waste transport links (#1250) * resolve Kosovo (XK) as separate country (#1249) * resolve Kosovo (XK) as separate country * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fillna * add missing key in data/existing_infrastructure/existing_heating_raw.csv --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * prepare_sector_network: correct if statement (#1252) * Dynamic central heating temperatures (#1206) * feat: Add rule to build central heating temperature profiles, adjust build_cop_profiles accordingly * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * style: make new config settings prettier * docs: add file headers * docs: update configtables and module docstring * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs: update release notes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * chore: use smooth ambient temperature through rolling window; remove default vals in class * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update doc/configtables/sector.csv --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Amos Schledorn <a.schledorn@tu-berlin.de> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> * Squashed commit of the following: commit a2b82195b872bae11a9e247c53756ca3ae512362 Author: Fabian Neumann <fabian.neumann@outlook.de> Date: Fri Aug 30 16:13:36 2024 +0200 minor adjustments commit 4f9eb04 Merge: 7b525a5 bf2d82a Author: Fabian Neumann <fabian.neumann@outlook.de> Date: Fri Aug 30 16:08:58 2024 +0200 Merge branch 'master' into master commit 7b525a5 Merge: d903337 a357ba1 Author: danielelerede-oet <daniele.lerede@openenergytransition.org> Date: Mon Aug 26 13:21:06 2024 +0200 Merge branch 'master' into master commit d903337 Author: danielelerede-oet <daniele.lerede@openenergytransition.org> Date: Mon Aug 26 10:34:27 2024 +0200 Update scripts/solve_network.py Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> commit 98e6c5b Author: danielelerede-oet <daniele.lerede@openenergytransition.org> Date: Mon Aug 26 10:34:21 2024 +0200 Update scripts/solve_network.py Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> commit be19a2b Author: danielelerede-oet <daniele.lerede@openenergytransition.org> Date: Mon Aug 26 10:34:15 2024 +0200 Update scripts/solve_operations_network.py Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> commit 8093eaa Author: danielelerede-oet <daniele.lerede@openenergytransition.org> Date: Mon Aug 26 10:34:08 2024 +0200 Update scripts/solve_network.py Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> commit 746d176 Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed Aug 21 15:29:12 2024 +0000 [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci commit f46fdee Author: Daniele Lerede <daniele.lerede@openenergytransition.org> Date: Wed Aug 21 17:11:32 2024 +0200 fix solve_operations_network * ci: code scanning with CodeQL (#1251) * ci: code scanning with CodeQL * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * prepare release v0.12.0 * Add emissions and fuel consumption from fuel refinieries (#1253) * Add emissions and fuel consumption from fuel refinieries * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * update version number, amend authors * fix version number in release notes * prepare_sector_network: convert e_max_pu timeseries for msw stores to pd.DataFrame. * add oil bus; extend global biomass limit; set unsustainable potentials to zero for overnight * keep sustainable potentials for switzerland * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * bugfix missing oil generator: remove else clause * darker color for oil refining plots * add missing copyright info --------- Co-authored-by: AmosSchledorn <amos.schledorn@icloud.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Toni Seibold <153275395+toniseibold@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de> Co-authored-by: Amos Schledorn <60692940+amosschle@users.noreply.github.com> Co-authored-by: lisazeyen <lisa.zeyen@web.de> Co-authored-by: lisazeyen <35347358+lisazeyen@users.noreply.github.com> Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com> Co-authored-by: cpschau <124347782+cpschau@users.noreply.github.com> Co-authored-by: Philipp Glaum <95913147+p-glaum@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Markus Millinger <50738187+millingermarkus@users.noreply.github.com> Co-authored-by: Tom Brown <t.brown@tu-berlin.de> Co-authored-by: Philipp Glaum <p.glaum@tu-berlin.de> Co-authored-by: Amos Schledorn <60692940+amos-schledorn@users.noreply.github.com> Co-authored-by: Amos Schledorn <a.schledorn@tu-berlin.de> Co-authored-by: Lukas Trippe <lkstrp@pm.me> Co-authored-by: cpschau <caspar.schauss@ieg.fraunhofer.de>

bobbyxng and others added 30 commits April 23, 2024 16:06

Implemented which uses the overpass API to download power features fo…

d92b69a

…r individual countries.

Merge branch 'PyPSA:master' into osm-network-fast

2f2d32e

Extended rule by input.

6352c03

Merge branch 'PyPSA:master' into osm-network-fast

6e8842f

Bug fixes and improvements to clean_osm_data.py. Added in retrieve_os…

87b4cce

…m_data.py.

Merge branch 'PyPSA:master' into osm-network-fast

b7167e7

Updated clean_osm_data and retrieve_osm_data to create clean substati…

e6c9acc

…ons.

Finished clean_osm_data function.

85aa3f1

Added check whether line is a circle. If so, drop it.

49c1baf

Merge branch 'PyPSA:master' into osm-network-fast

ff053e4

Extended build_electricity.smk by build_osm_network.py

75cffe4

Merge branch 'osm-network-fast' of https://github.com/bobbyxng/pypsa-eur

4c71c37

into osm-network-fast Added build_osm_network

Added build_osm_network

efb9611

Merge branch 'PyPSA:master' into osm-network-fast

193e605

Working osm-network-fast

266a8d0

Bug fixes.

6264bf8

Merge branch 'PyPSA:master' into osm-network-fast

bae4701

Finalised and cleaned including docstrings.

027fbde

Added try catch to retrieve_osm_data. Allows for parallelisation of d…

dc829d2

…ownloads.

Updated cleaning process.

7bb153b

Set maximum number of threads for retrieving to 4, wrt. fair usage po…

5ef4d71

…licy and potential request errors.

Intermediate update on clean_osm_data.py. Added docstrings.

f961ab8

Bug fix.

3b90765

Bug fix.

93f09a5

Bug fixes in data types out of clean_osm_data

98f50ac

Significant improvements to retrieve_osm_data, clean_osm_data. Cleane…

4f2308d

…d code. Speed improvements

Catch-up to master/osm-network-fast

50c7411

Merge branch 'PyPSA:master' into osm-network-fast

84bdb40

Merge branch 'PyPSA:master' into osm-network-fast

068d0c6

Cleaned config.

da94a96

bobbyxng and others added 8 commits August 15, 2024 17:52

Bug fix: Carrier type of all supernodes corrected to 'AC'

f87eec5

Bug fix: Carrier type of all supernodes corrected to 'AC'

e097fc4

Merge remote-tracking branch 'origin/master' into osm-network-fast

2864375

[pre-commit.ci] auto fixes from pre-commit.com hooks

864321e

for more information, see https://pre-commit.ci

Updated rules and base_network for compatibility with TYNDP projects.

9b66345

Merge branch 'osm-network-fast' of https://github.com/bobbyxng/pypsa-eur

dfcac7f

into osm-network-fast

Updated Zenodo repository and prebuilt network to include 150 kV HVDC…

412acd8

… connections.

Merge branch 'PyPSA:master' into osm-network-fast

1aa8b6c

bobbyxng self-assigned this Aug 20, 2024

bobbyxng and others added 7 commits August 20, 2024 11:20

Removed outdated config backup.

f1526fb

Merge branch 'PyPSA:master' into osm-network-fast

761d71c

Implemented all comments from PR PyPSA#1079. Cleaned up OSM implement…

7be5217

…ation.

Merge branch 'osm-network-fast' of https://github.com/bobbyxng/pypsa-eur

d3f439b

into osm-network-fast

Bug fix: Added all voltages, 200 kV-750 kV, to default config.

48c2cae

Cleaning and bugfixes.

89f0955

Updated Zenodo repository to https://zenodo.org/records/13358976. Add…

07a7a64

…ed converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with PyPSA#1079 and PyPSA#1085

fneum reviewed Aug 22, 2024

View reviewed changes

scripts/base_network.py Outdated Show resolved Hide resolved

fneum reviewed Aug 22, 2024

View reviewed changes

scripts/build_osm_network.py Outdated Show resolved Hide resolved

fneum reviewed Aug 22, 2024

View reviewed changes

scripts/clean_osm_data.py Outdated Show resolved Hide resolved

fneum and others added 6 commits August 22, 2024 11:53

Apply suggestions from code review

6be527f

[pre-commit.ci] auto fixes from pre-commit.com hooks

c7cfd45

for more information, see https://pre-commit.ci

simplify_network: handle complicated transformer topologies

259b4ec

[pre-commit.ci] auto fixes from pre-commit.com hooks

e1d4b49

for more information, see https://pre-commit.ci

syntax fix

cd9855f

Merge branch 'osm-network-fast' of github.com:bobbyxng/pypsa-eur into…

88c8faa

… osm-network-fast

fneum approved these changes Aug 22, 2024

View reviewed changes

fneum merged commit 0c36de9 into PyPSA:master Aug 22, 2024
5 of 6 checks passed

fneum mentioned this pull request Aug 23, 2024

Attaching loads at every bus (not only at 'substation_lv' buses) may improve the spatial load distribution #1214

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079

Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079

bobbyxng commented May 22, 2024 •

edited

Loading

bobbyxng commented Aug 22, 2024

Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079

Introducing OpenStreetMap high-voltage grid to PyPSA-Eur #1079

Conversation

bobbyxng commented May 22, 2024 • edited Loading

Changes proposed in this Pull Request

Components/scripts

Checklist

Known issues and simplifications

Benchmarks and validation

Benchmarks (exemplary for Europe retion)

Comparison of results/validation

Comparing route and circuit lengths against official ENTSO-E data (Inventory of transmission, https://www.entsoe.eu/data/power-stats/)

Base network overview

Line volume per NUTS1 region

Cross-regional capacity between NUTS1 regions

Delta in electricity generation per country

Transmission (25h resolution, full-year run, no TYNDP projects, no line expansion)

bobbyxng commented Aug 22, 2024

bobbyxng commented May 22, 2024 •

edited

Loading