Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) #3301

FranziskaWinterstein · 2023-07-19T08:44:08Z

Description

This adds download and formatting scripts to import NOAA marine boundary layer (MBL) CH4 data.

It is downloading the provided .csv table provided here. It formats the .csv file into a netcdf file, adds dummy latitude and longitude (since it is a global value) and fixes metadata.

Closes Add NOAA Marine Boundary Layer CH4 data #3299

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful

New or updated data reformatting script

🛠 Documentation is available
🛠 The dataset has been added to the CMOR check recipe
🧪 Numbers and units of the data look physically meaningful

FranziskaWinterstein · 2023-07-19T11:17:40Z

Note, that the tests fail, because the necessary cmor table for the variable ch4s is missing in the ESMValCore:
ESMValCore/esmvalcore/cmor/tables/custom/CMOR_ch4s.dat

SOURCE: CMIP5
!============
variable_entry:    ch4s
!============
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     mole_fraction_of_methane_in_air
units:             1e-06
cell_methods:      area: time: mean
cell_measures:     area: areacella
long_name:         Atmosphere CH4
comment:           As ch4, but only at the surface
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time
type:              real
!----------------------------------
!

I could make a pull request concerning this in the ESMValCore, however I also found the CMIP6 variable ch4global, which could fit at least to the NOAA dataset as well. I am testing that.

hb326 · 2023-08-08T08:05:01Z

Note, that the tests fail, because the necessary cmor table for the variable ch4s is missing in the ESMValCore: ESMValCore/esmvalcore/cmor/tables/custom/CMOR_ch4s.dat
SOURCE: CMIP5
!============
variable_entry:    ch4s
!============
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     mole_fraction_of_methane_in_air
units:             1e-06
cell_methods:      area: time: mean
cell_measures:     area: areacella
long_name:         Atmosphere CH4
comment:           As ch4, but only at the surface
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time
type:              real
!----------------------------------
!
I could make a pull request concerning this in the ESMValCore, however I also found the CMIP6 variable ch4global, which could fit at least to the NOAA dataset as well. I am testing that.

Hey @FranziskaWinterstein,
I think the new cmor table entry is good. I have opened a branch with this (for the methane surface stations) in ESMValCore. The only significant difference between the two definitions is the unit. I have specified 1e-09. Would that work for you as well?

FranziskaWinterstein · 2023-08-10T07:55:41Z

@hb326 Thank you! Yes, that is fine. 1e-09 is more common anyway I think.

hb326

@FranziskaWinterstein: Thanks for the formatter and cmorizer! I successfully downladed and formatted the data, tested the file with the recipe, and also looked at the data to check if it was in the correct units.
All works fine, the units look good, and everything runs smoothly with the recently merged "ch4s" entry to the custom cmor tables.

I have found a few minor things that I commented on, but nothing major. 👍

hb326 · 2023-08-24T13:22:46Z

esmvaltool/recipes/examples/recipe_check_obs.yml

+      ch4s:
+    additional_datasets:
+      - {dataset: NOAA-MBL-CH4, project: OBS, mip: Amon, type: atmos, version: 1.0, tier: 2}
+    scripts: null



@FranziskaWinterstein: did you leave the years out here because the dataset is delivered in one file anyway? With not asking for a specific time period, this is recipe is flexible enough to just read whatever is there.

Yes, I am unsure what would happen if the downloaded .csv file covers different years. Would you recommend to add years of the current available period?

I would probably add the years just because the other datasets in the recipe have a specified time period. Not sure if we have a clear rule here... @schlunma, @valeriupredoi, any recommendations?

@valeriupredoi: any recommendations here?

I would indeed - it's good to have it, if the user wants only a few years rather than download/cmorize the entire data span. In this case my review below becomes void since I am suggesting removing exactly these optional arguments (start/end times), if they are not used 👍

esmvaltool/references/noaa-mbl-ch4.bibtex

esmvaltool/cmorizers/data/formatters/datasets/gpcp_sg.py

esmvaltool/cmorizers/data/cmor_config/NOAA-MBL-CH4.yml

esmvaltool/recipes/examples/recipe_check_obs.yml

remi-kazeroni

Thanks for your contribution, @FranziskaWinterstein! The code looks good to me and all works fine: downloader, formatter, recipe_check_obs. I agree with @hb326's suggestions and added a couple of very minor points you may want to consider.

There are a few failing tests but nothing that can be addressed (for now):

the remaining Codacy issues can be ignored
the CircleCI run_tests fails because the recently added custom table (ch4s) is not yet part of a released version of ESMValCore. The test will pass once a new ESMValCore release will be made. This PR could only be merged after that. Thus, I added the "requires a new ESMValCore release" label.

I can pre-approve on the technical side 👍

esmvaltool/cmorizers/data/formatters/datasets/gpcp_sg.py

esmvaltool/cmorizers/data/formatters/datasets/noaa_mbl_ch4.py

…ool into cmorize_noaa_mbl_ch4

hb326

Looks good to me, @FranziskaWinterstein! Thanks for implementing the changes. I think the pull request is not ready to be merged (if we get the failing tests to pass...).

valeriupredoi

~~couple minor points from me~~. Many thanks @FranziskaWinterstein - the tests run with the latest development ESMValCore so all's OK on that front 🍺

EDIT: I decided not to stress on about unused arguments since these are endemic to a lot of cmorizers, and we should check for the need and make them optional in the main cmorizer module first; but clearly, that needs to be done in a dedicated PR, not here, and not in @schlunma 's PR (sorry for busting your balls on this, guys) 🍺

esmvaltool/cmorizers/data/downloaders/datasets/noaa_mbl_ch4.py

esmvaltool/cmorizers/data/formatters/datasets/noaa_mbl_ch4.py

valeriupredoi

squashing my stale review - all fine, and many thanks, Franziska! Ready for merge, unless you want to implement a start time/end time optionality 🍺

valeriupredoi

Blast! Re-blocking this due to the failed test - apologies 👍 Needing ESMValCore 2.10 that includes the cmor table for ch4s plopped by Birgit in ESMValGroup/ESMValCore#2168

FranziskaWinterstein · 2023-10-19T12:10:08Z

@valeriupredoi Thank you for taking care of this pull request. I am currently a little bit lost. Do you need me to do anything?

valeriupredoi · 2023-10-19T12:15:21Z

@FranziskaWinterstein no not at all, I just put a temporary blocker on it so one doesn't accidentally merge it until we have a new ESMValCore that includes ESMValGroup/ESMValCore#2168 (sorry it took me a few minutes to dig out the relevant PR that was merged in Core). All good by me otherwise 🍺

The review block was put in place to wait for the core release. That has happened and the tests do indeed work.

…OAA-MBL-CH4) (#3301) Co-authored-by: Birgit Hassler <33543691+hb326@users.noreply.github.com> Co-authored-by: Valeriu Predoi <valeriu.predoi@gmail.com> Co-authored-by: Klaus Zimmermann <klaus.zimmermann@smhi.se>

FranziskaWinterstein added 2 commits July 19, 2023 10:15

download and formatting of NOAA marine boundary layer data

70c7568

code formatting check

37ed4cc

FranziskaWinterstein added observations Data labels Jul 19, 2023

FranziskaWinterstein requested review from schlunma and hb326 July 19, 2023 08:44

FranziskaWinterstein self-assigned this Jul 19, 2023

FranziskaWinterstein requested a review from remi-kazeroni as a code owner July 19, 2023 08:44

added documentation

abddb55

Merge branch 'main' into cmorize_noaa_mbl_ch4

123fe2e

hb326 requested a review from a team as a code owner August 24, 2023 11:58

hb326 added looking for technical reviewer in scientific review labels Aug 24, 2023

hb326 added this to the v2.10.0 milestone Aug 24, 2023

hb326 requested changes Aug 24, 2023

View reviewed changes

remi-kazeroni changed the title ~~Download and formatting of NOAA marine boundary layer data for CH4~~ Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) Aug 25, 2023

remi-kazeroni added requires new ESMValCore release A new release of ESMValCore is needed to solve this issue/merge this pull request.. approved by technical reviewer and removed looking for technical reviewer labels Aug 25, 2023

remi-kazeroni reviewed Aug 25, 2023

View reviewed changes

esmvaltool/cmorizers/data/formatters/datasets/gpcp_sg.py Outdated Show resolved Hide resolved

esmvaltool/cmorizers/data/formatters/datasets/noaa_mbl_ch4.py Outdated Show resolved Hide resolved

esmvaltool/cmorizers/data/formatters/datasets/noaa_mbl_ch4.py Outdated Show resolved Hide resolved

FranziskaWinterstein and others added 4 commits September 5, 2023 13:52

changes due to comments

3818924

Merge branch 'cmorize_noaa_mbl_ch4' of github.com:ESMValGroup/ESMValT…

e60e13f

…ool into cmorize_noaa_mbl_ch4

added years to check and version to reference

f03f210

Merge branch 'main' into cmorize_noaa_mbl_ch4

8636c33

hb326 approved these changes Sep 18, 2023

View reviewed changes

hb326 added approved by scientific reviewer and removed in scientific review labels Sep 18, 2023

Merge branch 'main' into cmorize_noaa_mbl_ch4

9536397

valeriupredoi requested changes Oct 12, 2023

View reviewed changes

esmvaltool/cmorizers/data/downloaders/datasets/noaa_mbl_ch4.py Show resolved Hide resolved

esmvaltool/cmorizers/data/downloaders/datasets/noaa_mbl_ch4.py Show resolved Hide resolved

esmvaltool/cmorizers/data/formatters/datasets/noaa_mbl_ch4.py Show resolved Hide resolved

valeriupredoi approved these changes Oct 19, 2023

View reviewed changes

Merge branch 'main' into cmorize_noaa_mbl_ch4

052f97f

valeriupredoi previously requested changes Oct 19, 2023

View reviewed changes

Merge branch 'main' into cmorize_noaa_mbl_ch4

e05c3d7

zklaus requested a review from valeriupredoi November 3, 2023 09:01

zklaus merged commit 29b20e7 into main Nov 3, 2023
6 of 7 checks passed

zklaus deleted the cmorize_noaa_mbl_ch4 branch November 3, 2023 09:46

bouweandela mentioned this pull request Dec 14, 2023

recipe_checks_obs.yml cannot find required input data on Levante #3472

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) #3301

Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) #3301

FranziskaWinterstein commented Jul 19, 2023 •

edited

Loading

FranziskaWinterstein commented Jul 19, 2023

hb326 commented Aug 8, 2023

FranziskaWinterstein commented Aug 10, 2023

hb326 left a comment •

edited

Loading

hb326 Aug 24, 2023

FranziskaWinterstein Sep 5, 2023

hb326 Sep 7, 2023

hb326 Sep 18, 2023

valeriupredoi Oct 12, 2023

remi-kazeroni left a comment

hb326 left a comment

valeriupredoi left a comment •

edited

Loading

valeriupredoi left a comment

valeriupredoi left a comment •

edited

Loading

FranziskaWinterstein commented Oct 19, 2023

valeriupredoi commented Oct 19, 2023 •

edited

Loading

Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) #3301

Download and formatting of NOAA marine boundary layer data for CH4 (NOAA-MBL-CH4) #3301

Conversation

FranziskaWinterstein commented Jul 19, 2023 • edited Loading

Description

Before you get started

Checklist

New or updated data reformatting script

FranziskaWinterstein commented Jul 19, 2023

hb326 commented Aug 8, 2023

FranziskaWinterstein commented Aug 10, 2023

hb326 left a comment • edited Loading

Choose a reason for hiding this comment

hb326 Aug 24, 2023

Choose a reason for hiding this comment

FranziskaWinterstein Sep 5, 2023

Choose a reason for hiding this comment

hb326 Sep 7, 2023

Choose a reason for hiding this comment

hb326 Sep 18, 2023

Choose a reason for hiding this comment

valeriupredoi Oct 12, 2023

Choose a reason for hiding this comment

remi-kazeroni left a comment

Choose a reason for hiding this comment

hb326 left a comment

Choose a reason for hiding this comment

valeriupredoi left a comment • edited Loading

Choose a reason for hiding this comment

valeriupredoi left a comment

Choose a reason for hiding this comment

valeriupredoi left a comment • edited Loading

Choose a reason for hiding this comment

FranziskaWinterstein commented Oct 19, 2023

valeriupredoi commented Oct 19, 2023 • edited Loading

FranziskaWinterstein commented Jul 19, 2023 •

edited

Loading

hb326 left a comment •

edited

Loading

valeriupredoi left a comment •

edited

Loading

valeriupredoi left a comment •

edited

Loading

valeriupredoi commented Oct 19, 2023 •

edited

Loading