Add the "state of ground" to bring in the extra snow depth reports #1368

jiaruidong2017 · 2024-11-13T01:22:53Z

This PR uses the python BUFR2IODA API to read the GTS bufr snow observations and adds to read the state of ground (SOGR) for bringing in the extra snow depth reports. If the state of the ground says there is no snow, but the snow depth observation is missing, we should change the missing values to be 0 so that the reports can be assimilated.

This PR also incorporates the latest changes from submodule from NOAA-EMC/bufr-query.

This PR contributes to NOAA-EMC/global-workflow#3093

CoryMartin-NOAA · 2024-11-13T16:12:20Z

parm/snow/obs/config/bufr2ioda_mapping.yaml.j2

@@ -2,3 +2,4 @@ mkdir:
 - '{{ DATA }}/obs'
 copy:
 - ['{{PARMgfs}}/gdas/snow/obs/config/bufr_sfcsno_mapping.yaml', '{{ DATA }}/obs/']
+- ['{{PARMgfs}}/gdas/snow/obs/config/ioda_bufr_python_encoder.py', '{{ DATA }}']


does the python script need copied? and should it go here? we might need to talk to @emilyhcliu about the progress of SPOC

Thanks @CoryMartin-NOAA for your suggestions. Do you agree if I put this python code to sorc/gdas.cd/parm/snow/, so that this code can be used directly?
@emilyhcliu Any suggestions?

I moved this py code to the parm/gdas/snow/ directory for a temporary solution. @emilyhcliu Please let me know if you have any other suggestions. Thanks.

Scripts shouldn't be in parm/, until we move to SPOC, it should probably go here: https://github.com/NOAA-EMC/GDASApp/tree/develop/ush/ioda/bufr2ioda

Understood. Wait for @emilyhcliu suggestions.

ClaraDraper-NOAA

Thanks for doing this @jiaruidong2017 . I have requested one small change.

ClaraDraper-NOAA · 2024-11-13T17:53:46Z

parm/snow/ioda_bufr_python_encoder.py

+
+    sogr = container.get('variables/groundState')
+    snod = container.get('variables/totalSnowDepth')
+    snod[(sogr <= 11.0) | (sogr == 15.0)] = 0.0


Without having seen the data, I think there should be an if statement here - so that we only replace the snod when it was undefined / missing.

i.e., if snod > 0. (or some, slighty higher threshold), and sogr indictaes no snow cover, I assume we would want to use the snod value, not 0.

Above (with an if statement) is all consistent with how the PR description.

Good suggestions. I will investigate this.

@rmclaren Do you have any suggestions?

Thanks again for the good findings. I modified to only replace the filled/missing snod.

ClaraDraper-NOAA · 2024-11-13T17:55:15Z

parm/snow/obs/config/bufr_sfcsno_mapping.yaml

-    - bounding:
-        variable: totalSnowDepth
-        lowerBound: 0
-        upperBound: 10000000


For my understanding:

is this introducing a 10 m limit on snow depth obs?

if they are above 10 m, are they discarded, or set to 10.?

I previously put this large number (10000m) here for removing the missing values. Here, we don't need this any more because we need to set the missing snod values to 0 when sogr satisfies the defined conditions.

We might still have missing values though wouldn't we?

The missing values will be removed after applying sogr conditions.

ClaraDraper-NOAA · 2024-11-13T17:56:59Z

parm/snow/ioda_bufr_python_encoder.py

+
+    return new_container
+
+def create_obs_group(input_path):


Some comments here would be helpful to explain what we're doing - specifically what snogr is, and why we're selecting out the values that we are.

Excellent suggestions. I will add the comments here later. Thanks.

Add the comments to explain the steps.

CoryMartin-NOAA · 2024-11-14T20:26:51Z

parm/snow/ioda_bufr_python_encoder.py

@jiaruidong2017 move this to https://github.com/NOAA-EMC/GDASApp/tree/develop/ush/ioda/bufr2ioda but change its name to something more descriptive/specific until we can move things to SPOC

rmclaren · 2024-11-14T20:41:24Z

parm/snow/ioda_bufr_python_encoder.py

+    encoder = Encoder(YAML_PATH)
+    data = next(iter(encoder.encode(masked_container).values()))


Could be 1 liner data = next(iter(Encoder(YAML_PATH).encode(masked_container).values())). Up to you :)

Thanks @rmclaren for your suggestions. Will change it.

rmclaren · 2024-11-14T20:45:22Z

parm/snow/obs/config/sfcsno_snow.yaml.j2

+          input_path: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d'
        obsfile: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d'


Don't forget to add an issue so they do not assume the formatting of the YAML file (obsfile variable shoould not be necessary). Reading the backend engines YAML breaks encapsulation (external modules should not be dependent on the private configuration for a backend type. Some backends may not have any kind of file associated with them (like a db...).

yes thanks @rmclaren I'm aware of that issue, I/we/others need to figure out how best to handle it on the workflow side.

emilyhcliu · 2024-11-15T16:09:13Z

parm/snow/obs/config/sfcsno_snow.yaml.j2

+        type: script
+        script file: "{{ USHgfs }}/bufr2ioda_sfcsno_bufr_encoder.py"
+        args:
+          input_path: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d'


Suggestion:
Make the mapping file as input to the Python script for create_obs_group

args: input_path: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d' mapping_path: = "./obs/bufr_sfcsno_mapping.yaml"

This will make it clear that the script backend requires two inputs: bufr file and the mapping file

Thanks @emilyhcliu for the suggestions. I made the changes as suggested.

emilyhcliu · 2024-11-15T16:22:00Z

@CoryMartin-NOAA For v17 global-workflow, do we want to read the bufr as the backend (this PR), or do we want to convert the bufr info IODA format netCDF in the obsproc step?

CoryMartin-NOAA · 2024-11-15T18:03:01Z

@emilyhcliu I think my preference would be backend for this (and other) case where the input file is BUFR.

emilyhcliu · 2024-11-22T20:39:30Z

@jiaruidong2017 Have you run a cycle in global-workflow with the changes from this PR?

emilyhcliu · 2024-11-24T00:14:18Z

@jiaruidong2017 Could you point me where to get a sfcsno bufr data for testing? Thanks.

jiaruidong2017 · 2024-11-24T00:37:08Z

@jiaruidong2017 Could you point me where to get a sfcsno bufr data for testing? Thanks.

@emilyhcliu You can find the data on hera from the dump directory at: /scratch1/NCEPDEV/global/glopara/dump/gdas.$YYYY$MM$DD/$HH/atmos/gdas.t${HH}z.sfcsno.tm00.bufr_d for the period from 2021-07-01 to 2022-05-31.

jiaruidong2017 · 2024-11-24T00:38:40Z

@jiaruidong2017 Have you run a cycle in global-workflow with the changes from this PR?

Yes, it worked as expected.

emilyhcliu · 2024-11-24T21:00:30Z

@jiaruidong2017 Could you point me where to get a sfcsno bufr data for testing? Thanks.

@emilyhcliu You can find the data on hera from the dump directory at: /scratch1/NCEPDEV/global/glopara/dump/gdas.$YYYY$MM$DD/$HH/atmos/gdas.t${HH}z.sfcsno.tm00.bufr_d for the period from 2021-07-01 to 2022-05-31.

Oh, no wonder I could not find the near-real-time data. Thanks!!

emilyhcliu · 2024-11-25T00:51:52Z

@jiaruidong2017 I tested the Python script for converter for 2021080100 sfcsno bufr

The totalSnowDepth data count` for all data (missing + valid) is 138359
The totalSnowDepth data count` for the masked data (based on missing + state of the ground condition) is 3685

The final IODA output file has data account = 3685

This is the result of my test. Do those data count values look right to you?

jiaruidong2017 · 2024-11-25T01:20:24Z

@jiaruidong2017 Could you point me where to get a sfcsno bufr data for testing? Thanks.

@emilyhcliu You can find the data on hera from the dump directory at: /scratch1/NCEPDEV/global/glopara/dump/gdas.$YYYY$MM$DD/$HH/atmos/gdas.t${HH}z.sfcsno.tm00.bufr_d for the period from 2021-07-01 to 2022-05-31.

Oh, no wonder I could not find the near-real-time data. Thanks!!

The real-time data will be available soon (expected in next January) after the next release of obsproc v1.3.

jiaruidong2017 · 2024-11-25T01:21:27Z

@jiaruidong2017 I tested the Python script for converter for 2021080100 sfcsno bufr

The totalSnowDepth data count` for all data (missing + valid) is 138359

The totalSnowDepth data count` for the masked data (based on missing + state of the ground condition) is 3685

The final IODA output file has data account = 3685

This is the result of my test. Do those data count values look right to you?

Yeah, it looks good to me. Thanks.

jiaruidong2017 · 2024-11-25T01:39:53Z

@emilyhcliu You can find the near-real-time sfcsno dump file on wcoss2 at below:

/lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3/gdas.20241124/00/atmos/gdas.t00z.sfcsno.tm00.bufr_d

ush/ioda/bufr2ioda/bufr2ioda_sfcsno_bufr_encoder.py

jiaruidong2017 added 2 commits November 9, 2024 11:44

Add state of ground to increase the snow depth reports.

1c8096f

Update the defined index with zero snow depth.

365675d

jiaruidong2017 mentioned this pull request Nov 13, 2024

Add the "state of ground" to increase the snow depth reports NOAA-EMC/global-workflow#3093

Draft

13 tasks

jiaruidong2017 requested review from CoryMartin-NOAA, ClaraDraper-NOAA and TravisElless-NOAA November 13, 2024 01:28

Merge branch 'develop' into feature/snow-sogr

aee4dc2

CoryMartin-NOAA reviewed Nov 13, 2024

View reviewed changes

Move python code to parm/gdas/snow/ directory to aviod extra copy.

0c02522

ClaraDraper-NOAA requested changes Nov 13, 2024

View reviewed changes

jiaruidong2017 added 2 commits November 13, 2024 18:29

Modif to only replace the missing/filled snod values.

e57ac67

Merge branch 'develop' into feature/snow-sogr

4376049

jiaruidong2017 requested a review from ClaraDraper-NOAA November 13, 2024 23:37

CoryMartin-NOAA reviewed Nov 14, 2024

View reviewed changes

rmclaren previously approved these changes Nov 14, 2024

View reviewed changes

Address reviewer's comments.

244693c

jiaruidong2017 dismissed rmclaren’s stale review via 244693c November 14, 2024 22:46

Fix py norms error.

76a5e94

jiaruidong2017 requested review from CoryMartin-NOAA and rmclaren November 14, 2024 22:56

emilyhcliu requested changes Nov 15, 2024

View reviewed changes

jiaruidong2017 added 3 commits November 15, 2024 12:07

Add the mapping_path indicating the script backend requirements.

2a72e4d

Merge branch 'develop' into feature/snow-sogr

a2d87fd

Add comments

fbd1bb9

jiaruidong2017 added 2 commits November 22, 2024 15:31

Merge branch 'develop' into feature/snow-sogr

db46314

Update the missing value for elevation from bufr file.

91f160e

emilyhcliu approved these changes Nov 26, 2024

View reviewed changes

ush/ioda/bufr2ioda/bufr2ioda_sfcsno_bufr_encoder.py Outdated Show resolved Hide resolved

emilyhcliu mentioned this pull request Dec 9, 2024

add components for surface snow data NOAA-EMC/spoc#14

Open

		encoder = Encoder(YAML_PATH)
		data = next(iter(encoder.encode(masked_container).values()))

		input_path: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d'
		obsfile: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d'

Add the "state of ground" to bring in the extra snow depth reports #1368

Are you sure you want to change the base?

Add the "state of ground" to bring in the extra snow depth reports #1368

Conversation

jiaruidong2017 commented Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ClaraDraper-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyhcliu commented Nov 15, 2024

CoryMartin-NOAA commented Nov 15, 2024

emilyhcliu commented Nov 22, 2024

emilyhcliu commented Nov 24, 2024

jiaruidong2017 commented Nov 24, 2024

jiaruidong2017 commented Nov 24, 2024

emilyhcliu commented Nov 24, 2024 • edited Loading

emilyhcliu commented Nov 25, 2024

jiaruidong2017 commented Nov 25, 2024

jiaruidong2017 commented Nov 25, 2024

jiaruidong2017 commented Nov 25, 2024

jiaruidong2017 commented Nov 13, 2024 •

edited

Loading

emilyhcliu commented Nov 24, 2024 •

edited

Loading