From a7b4cee9ed428a8b333ad67108647792fb990c58 Mon Sep 17 00:00:00 2001 From: Daniel Busch Date: Mon, 15 Jul 2024 10:26:58 +0200 Subject: [PATCH 1/6] order --- README.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 8f8bebf..bf62a07 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,20 @@ The Open Methane prior can be installed from source into a virtual environment w make virtual-environment ``` +### Input Data + +To download all the required input files, run: + +```console +poetry run python scripts/omDownloadInputs.py +``` + +This will download input files that match the data in `.env.example`, +so you have a working set to get started with. + +The downloaded files will be stored in `data/inputs` by default. + + ### Domain Info The domain of interest for the prior is defined using an input domain netCDF file. @@ -109,19 +123,6 @@ we have provided a domain file for a 10km grid over Australia. This file will be downloaded with the other layer inputs (see below) using the default configuration values. -### Input Data - -To download all the required input files, run: - -```console -poetry run python scripts/omDownloadInputs.py -``` - -This will download input files that match the data in `.env.example`, -so you have a working set to get started with. - -The downloaded files will be stored in `data/inputs` by default. - ## Run ### All layers From ffcf8528fc253aae4ffd34f240e5077639d10ce7 Mon Sep 17 00:00:00 2001 From: Daniel Busch Date: Mon, 15 Jul 2024 15:05:49 +0200 Subject: [PATCH 2/6] docker --- README.md | 69 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index bf62a07..74dab1a 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ After creating an account and obtaining the credentials, you must also accept the general [license conditions for the data](https://ads.atmosphere.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products) in order to download the data. -You can check whether the license terms have been accepted by visiting the +You can check whether the license terms have been accepted by visiting the [GFAS Dataset](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-global-fire-emissions-gfas?tab=form) while logged in. The "Terms of Use" section at the bottom of the page shows the license @@ -36,6 +36,9 @@ The Open Methane prior can be installed from source into a virtual environment w make virtual-environment ``` +The `Makefile` contains the set of commands to create the virtual environment. +You can read the instructions out and run the commands by hand if you wish. + ### Input Data To download all the required input files, run: @@ -49,13 +52,13 @@ so you have a working set to get started with. The downloaded files will be stored in `data/inputs` by default. - ### Domain Info The domain of interest for the prior is defined using an input domain netCDF file. The format of the input domain is based on the CMAQ domain file format and uses a staggered grid. This input file should contain the following variables: + * LAT * LON * LANDMASK @@ -123,6 +126,16 @@ we have provided a domain file for a 10km grid over Australia. This file will be downloaded with the other layer inputs (see below) using the default configuration values. +### Clean outputs + +``` +make clean +``` + +``` +make clean-all +``` + ## Run ### All layers @@ -133,13 +146,16 @@ To calculate emissions for all layers, run `omPrior.py` with a start and end dat poetry run python scripts/omPrior.py 2022-07-01 2022-07-02 ``` -This takes a while to process (~10 minutes) with the vast majority of that time spent on the layers in `omAgLulucfWasteEmis.py`. +This takes a while to process (~10 minutes) with the vast majority of that time spent on the layers +in `omAgLulucfWasteEmis.py`. -To skip re-projecting raster layers (you only need to do this once for every time you change the raster input files), add the `--skip-reproject` option. +To skip re-projecting raster layers (you only need to do this once for every time you change the raster input files), +add the `--skip-reproject` option. ### Single layers -You can run and re-run individual layers one-by-one. Just run each file on it's own (GFAS and Wetlands require a start and end date as below): +You can run and re-run individual layers one-by-one. Just run each file on it's own (GFAS and Wetlands require a start +and end date as below): ```console poetry run python src/layers/omWetlandEmis.py 2022-07-01 2022-07-02 @@ -147,19 +163,24 @@ poetry run python src/layers/omWetlandEmis.py 2022-07-01 2022-07-02 ## Outputs -Outputs can be found in the `data/outputs` folder. The emissions layers will be written as variables to a copy of the input domain file, with an `OCH4_` prefix for the methane layer variable names. The sum of all layers will be stored in the `OCH4_TOTAL` layer. +Outputs can be found in the `data/outputs` folder. The emissions layers will be written as variables to a copy of the +input domain file, with an `OCH4_` prefix for the methane layer variable names. The sum of all layers will be stored in +the `OCH4_TOTAL` layer. The name of the layered output file will be `om-prior-output.nc`. -The `data/processed` folder will contain any re-projected raster data, and any files downloaded or generated in the process. +The `data/processed` folder will contain any re-projected raster data, and any files downloaded or generated in the +process. ## Layers Many sectors are taken from data sets used in by Saunois et al (2020) (doi:10.5194/essd-12-1561-2020) -- Livestock: Enteric fermentation emissions generated by CSIRO Ag. and Food using livestock census data and UNFCCC emissions factors +- Livestock: Enteric fermentation emissions generated by CSIRO Ag. and Food using livestock census data and UNFCCC + emissions factors - Electricity: Uses OpenNEM facility data to spatialise the Aust. Gov UNFCCC electricity emissions -- Agriculture: Agricultural emissions apart from livestock taken from the Agricultural emissions of the NGGI and spatialised according to the agriculture land-use mask +- Agriculture: Agricultural emissions apart from livestock taken from the Agricultural emissions of the NGGI and + spatialised according to the agriculture land-use mask - Fugitives: Facility-level data from ACF (more info?) - Industrial: Spatialises the industrial sector of the NGGI according to nighttime lights - Stationary: Spatialises the stationary energy sector of the NGGI according to nighttime lights @@ -167,5 +188,33 @@ Many sectors are taken from data sets used in by Saunois et al (2020) (doi:10.51 - Waste: Spatialises the NGGI waste emission according to the landuse map - LULUCF: Spatialises the LULUCF emission from the NGGI according to the landuse map - FIRE: daily emissions from the Global Fire Assimilation System (Kaiser et al., 2012, doi:10.5194/bg-9-527-2012) -- wetland: Monthly wetland emissions from the diagnostic ensemble used in Saunois et al. 2020 and described in Zhang et al. (2023 under review) +- wetland: Monthly wetland emissions from the diagnostic ensemble used in Saunois et al. 2020 and described in Zhang et + al. (2023 under review) - Termite: Termite emissions used in Saunois et al. 2020 supplied by Simona Castaldi and Sergio Noce + +## Run in a Docker container + +To carry out the steps described above in a Docker container, first build the Docker image with + +``` +make build +``` +Then run the commands to download the input data in the docker container + +``` +docker run --rm -v :/opt/project openmethane-prior python scripts/omDownloadInputs.py +``` + +Replace the python scripts for the other steps. + +## For developers + +``` +make ruff-fixes +``` + +``` +make test +``` + +Note: make sure to install the dev dependencies. From 82bcb7254752782b579bd743f596385eaeb82d55 Mon Sep 17 00:00:00 2001 From: Daniel Busch Date: Mon, 15 Jul 2024 16:55:58 +0200 Subject: [PATCH 3/6] make commands, clean up --- README.md | 112 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 59 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index 74dab1a..57bd6e7 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # OpenMethane prior emissions estimate -Method to calculate a gridded, prior emissions estimate for methane across Australia. +Method to calculate a gridded, prior emissions estimate for methane. This repository is matched with downloadable input data so that it will run out of the box. @@ -44,7 +44,7 @@ You can read the instructions out and run the commands by hand if you wish. To download all the required input files, run: ```console -poetry run python scripts/omDownloadInputs.py +make download ``` This will download input files that match the data in `.env.example`, @@ -55,15 +55,18 @@ The downloaded files will be stored in `data/inputs` by default. ### Domain Info The domain of interest for the prior is defined using an input domain netCDF file. -The format of the input domain is based on the CMAQ domain file format and uses a staggered grid. +The format of the input domain is based on the CMAQ domain file format. Note that CMAQ uses a +[staggered grid](https://www.cmascenter.org/ioapi/documentation/all_versions/html/GRIDS.jpg) +where some quantities are defined at the center of a grid cell, whereas other quantities are defined +at the edges of a grid cell. This circumstance is represented in `ROW_D = ROW + 1`. This input file should contain the following variables: -* LAT -* LON -* LANDMASK -* LATD -* LOND +* `LAT` +* `LON` +* `LANDMASK` +* `LATD` +* `LOND` The contents of the default domain is shown below: @@ -71,52 +74,52 @@ The contents of the default domain is shown below: >>> ncdump -h prior_domain_aust10km_v1.0.0.d01 netcdf prior_domain_aust10km_v1.0.0.d01 { dimensions: - TSTEP = 1 ; - ROW = 430 ; - COL = 454 ; - LAY = 1 ; - ROW_D = 431 ; - COL_D = 455 ; + TSTEP = 1; + ROW = 430; + COL = 454; + LAY = 1; + ROW_D = 431; + COL_D = 455; variables: - float LAT(TSTEP, ROW, COL) ; - LAT:_FillValue = NaNf ; - LAT:long_name = "LAT " ; - LAT:units = "DEGREES " ; - LAT:var_desc = "latitude (south negative) " ; - float LON(TSTEP, ROW, COL) ; - LON:_FillValue = NaNf ; - LON:long_name = "LON " ; - LON:units = "DEGREES " ; - LON:var_desc = "longitude (west negative) " ; - float LANDMASK(TSTEP, ROW, COL) ; - LANDMASK:_FillValue = NaNf ; - LANDMASK:long_name = "LWMASK " ; - LANDMASK:units = "CATEGORY " ; - LANDMASK:var_desc = "land-water mask (1=land, 0=water) " ; - float LATD(TSTEP, LAY, ROW_D, COL_D) ; - LATD:_FillValue = NaNf ; - LATD:long_name = "LATD " ; - LATD:units = "DEGREES " ; - LATD:var_desc = "latitude (south negative) -- dot point " ; - float LOND(TSTEP, LAY, ROW_D, COL_D) ; - LOND:_FillValue = NaNf ; - LOND:long_name = "LOND " ; - LOND:units = "DEGREES " ; - LOND:var_desc = "longitude (west negative) -- dot point " ; + float LAT(TSTEP, ROW, COL); + LAT:_FillValue = NaNf; + LAT:long_name = "LAT"; + LAT:units = "DEGREES"; + LAT:var_desc = "latitude (south negative)"; + float LON(TSTEP, ROW, COL); + LON:_FillValue = NaNf; + LON:long_name = "LON"; + LON:units = "DEGREES"; + LON:var_desc = "longitude (west negative)"; + float LANDMASK(TSTEP, ROW, COL); + LANDMASK:_FillValue = NaNf; + LANDMASK:long_name = "LWMASK"; + LANDMASK:units = "CATEGORY"; + LANDMASK:var_desc = "land-water mask (1=land, 0=water)"; + float LATD(TSTEP, LAY, ROW_D, COL_D); + LATD:_FillValue = NaNf; + LATD:long_name = "LATD"; + LATD:units = "DEGREES"; + LATD:var_desc = "latitude (south negative) -- dot point"; + float LOND(TSTEP, LAY, ROW_D, COL_D); + LOND:_FillValue = NaNf; + LOND:long_name = "LOND"; + LOND:units = "DEGREES"; + LOND:var_desc = "longitude (west negative) -- dot point"; // global attributes: - :DX = 10000.f ; - :DY = 10000.f ; - :TRUELAT1 = -15.f ; - :TRUELAT2 = -40.f ; - :MOAD_CEN_LAT = -27.644f ; - :STAND_LON = 133.302f ; - :XCELL = 10000. ; - :YCELL = 10000. ; - :XCENT = 133.302001953125 ; - :YCENT = -27.5 ; - :XORIG = -2270000. ; - :YORIG = -2165629.25 ; + :DX = 10000.f; + :DY = 10000.f; + :TRUELAT1 = -15.f; + :TRUELAT2 = -40.f; + :MOAD_CEN_LAT = -27.644f; + :STAND_LON = 133.302f; + :XCELL = 10000.; + :YCELL = 10000.; + :XCENT = 133.302001953125; + :YCENT = -27.5; + :XORIG = -2270000.; + :YORIG = -2165629.25; } ``` @@ -128,10 +131,13 @@ values. ### Clean outputs +Delete all files in `intermediates` and `outputs` directory. ``` make clean ``` +Delete all files in `intermediates`, `outputs`, and `inputs` directory. + ``` make clean-all ``` @@ -143,7 +149,7 @@ make clean-all To calculate emissions for all layers, run `omPrior.py` with a start and end date: ```console -poetry run python scripts/omPrior.py 2022-07-01 2022-07-02 +make run ``` This takes a while to process (~10 minutes) with the vast majority of that time spent on the layers @@ -205,7 +211,7 @@ Then run the commands to download the input data in the docker container docker run --rm -v :/opt/project openmethane-prior python scripts/omDownloadInputs.py ``` -Replace the python scripts for the other steps. +Replace the python files according to the commands in the Makefile for the other steps. ## For developers From 09321b5dae6277dfb90474ff70f74555ed733ef1 Mon Sep 17 00:00:00 2001 From: Daniel Busch Date: Mon, 15 Jul 2024 17:18:14 +0200 Subject: [PATCH 4/6] clean up --- README.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 57bd6e7..2dd36b7 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # OpenMethane prior emissions estimate -Method to calculate a gridded, prior emissions estimate for methane. +Method to calculate a gridded, prior emissions estimate for methane across Australia. This repository is matched with downloadable input data so that it will run out of the box. @@ -55,10 +55,10 @@ The downloaded files will be stored in `data/inputs` by default. ### Domain Info The domain of interest for the prior is defined using an input domain netCDF file. -The format of the input domain is based on the CMAQ domain file format. Note that CMAQ uses a -[staggered grid](https://www.cmascenter.org/ioapi/documentation/all_versions/html/GRIDS.jpg) -where some quantities are defined at the center of a grid cell, whereas other quantities are defined -at the edges of a grid cell. This circumstance is represented in `ROW_D = ROW + 1`. +The format of the input domain is based on the CMAQ domain file format. Note that CMAQ uses a +[staggered grid](https://www.cmascenter.org/ioapi/documentation/all_versions/html/GRIDS.jpg) +where some quantities are defined at the center of a grid cell, whereas other quantities are defined +at the edges of a grid cell. This circumstance is represented in `ROW_D = ROW + 1`. This input file should contain the following variables: @@ -126,17 +126,18 @@ variables: As part of the [OpenMethane](https://openmethane.org/) project, we have provided a domain file for a 10km grid over Australia. -This file will be downloaded with the other layer inputs (see below) using the default configuration +This file will be downloaded with the other layer inputs (see [Input Data](#input-data)) using the default configuration values. ### Clean outputs +These two commands are set up so that not all generated files have to be deleted manually +Delete all files in the `intermediates` and `outputs` directory with -Delete all files in `intermediates` and `outputs` directory. ``` make clean ``` -Delete all files in `intermediates`, `outputs`, and `inputs` directory. +Or delete all files in `intermediates`, `outputs`, and `inputs` directory with ``` make clean-all @@ -148,6 +149,12 @@ make clean-all To calculate emissions for all layers, run `omPrior.py` with a start and end date: +``` +poetry run python scripts/omPrior.py 2022-07-01 2022-07-01 +``` + +or use the make target + ```console make run ``` @@ -205,6 +212,7 @@ To carry out the steps described above in a Docker container, first build the Do ``` make build ``` + Then run the commands to download the input data in the docker container ``` @@ -215,12 +223,15 @@ Replace the python files according to the commands in the Makefile for the other ## For developers +The ruff-fixes target runs a series of ruff commands to format the code, check and fix linting +issues, and then format the code again to ensure that all formatting and fixes are applied. + ``` make ruff-fixes ``` +The test target will run all the tests + ``` make test ``` - -Note: make sure to install the dev dependencies. From a72dc394ba12c5b743725070cf57796a68ef07a9 Mon Sep 17 00:00:00 2001 From: Daniel Busch Date: Mon, 22 Jul 2024 11:05:40 +0200 Subject: [PATCH 5/6] data directories --- .env.example | 2 +- README.md | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/.env.example b/.env.example index 5f35b85..75431ac 100644 --- a/.env.example +++ b/.env.example @@ -16,7 +16,7 @@ PRIOR_REMOTE=https://prior.openmethane.org/ # Input files names # Use omDownloadInputs.py to fetch these files from the remote and place them in the inputs folder # If you're using different input files, place them in the inputs folder, and update the filenames below -# See README.md for a detailed explanation of the purpose of each file +# See README.md for a detailed explanation of the purpose of each layer CH4_ELECTRICITY=ch4-electricity.csv CH4_COAL=coal-mining_emissions-sources.csv CH4_OILGAS=oil-and-gas-production-and-transport_emissions-sources.csv diff --git a/README.md b/README.md index 2dd36b7..1322528 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,7 @@ This file will be downloaded with the other layer inputs (see [Input Data](#inpu values. ### Clean outputs + These two commands are set up so that not all generated files have to be deleted manually Delete all files in the `intermediates` and `outputs` directory with @@ -205,6 +206,17 @@ Many sectors are taken from data sets used in by Saunois et al (2020) (doi:10.51 al. (2023 under review) - Termite: Termite emissions used in Saunois et al. 2020 supplied by Simona Castaldi and Sergio Noce +## Data directories + +* `data/inputs` +This folder should contain all the required input files, which should be referenced in the `.env` file at the root. +A set of input files has been included in the repository so that it functions out of the box (see [Input Data](#input-data)), but you can add your own +data here. +* `data/inputs/domains` The domain of interest is stored in this folder (see [domain info](#domain-info)). +* `data/intermediates` This folder contains any intermediate files generated through the process. Everything within this folder should be ignored. +* `data/outputs` Outputs files will be saved here. + + ## Run in a Docker container To carry out the steps described above in a Docker container, first build the Docker image with From b95208ea5734d4fffb482f7784482e8343ca877f Mon Sep 17 00:00:00 2001 From: crdanielbusch <150670891+crdanielbusch@users.noreply.github.com> Date: Tue, 23 Jul 2024 09:19:56 +0200 Subject: [PATCH 6/6] Update README.md Co-authored-by: Jared Lewis --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1322528..fe4a9e7 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ To download all the required input files, run: make download ``` -This will download input files that match the data in `.env.example`, +This will download input files that match the data in `.env`, so you have a working set to get started with. The downloaded files will be stored in `data/inputs` by default.