-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the "state of ground" to bring in the extra snow depth reports #1368
base: develop
Are you sure you want to change the base?
Changes from 4 commits
1c8096f
365675d
aee4dc2
0c02522
e57ac67
4376049
244693c
76a5e94
2a72e4d
a2d87fd
fbd1bb9
db46314
91f160e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
import numpy as np | ||
import bufr | ||
from pyioda.ioda.Engines.Bufr import Encoder | ||
|
||
|
||
def mask_container(container, mask): | ||
new_container = bufr.DataContainer() | ||
for var_name in container.list(): | ||
print(f" ... variable name: {var_name} ...") | ||
var = container.get(var_name) | ||
paths = container.get_paths(var_name) | ||
new_container.add(var_name, var[mask], paths) | ||
|
||
return new_container | ||
|
||
def create_obs_group(input_path): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some comments here would be helpful to explain what we're doing - specifically what snogr is, and why we're selecting out the values that we are. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Excellent suggestions. I will add the comments here later. Thanks. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add the comments to explain the steps. |
||
YAML_PATH = "./obs/bufr_sfcsno_mapping.yaml" | ||
container = bufr.Parser(input_path, YAML_PATH).parse() | ||
|
||
sogr = container.get('variables/groundState') | ||
snod = container.get('variables/totalSnowDepth') | ||
snod[(sogr <= 11.0) | (sogr == 15.0)] = 0.0 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Without having seen the data, I think there should be an if statement here - so that we only replace the snod when it was undefined / missing. i.e., if snod > 0. (or some, slighty higher threshold), and sogr indictaes no snow cover, I assume we would want to use the snod value, not 0. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Above (with an if statement) is all consistent with how the PR description. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good suggestions. I will investigate this. @rmclaren Do you have any suggestions? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks again for the good findings. I modified to only replace the filled/missing snod. |
||
container.replace('variables/totalSnowDepth', snod) | ||
|
||
print(f" ... Remove filled/missing snow values ...") | ||
masked_container = mask_container(container, (~snod.mask)) | ||
|
||
encoder = Encoder(YAML_PATH) | ||
data = next(iter(encoder.encode(masked_container).values())) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could be 1 liner There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @rmclaren for your suggestions. Will change it. |
||
|
||
return data |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,7 +17,6 @@ bufr: | |
query: "[*/CLON, */CLONH]" | ||
stationIdentification: | ||
query: "*/RPID" | ||
|
||
stationElevation: | ||
query: "[*/SELV, */HSMSL]" | ||
|
||
|
@@ -26,11 +25,8 @@ bufr: | |
query: "[*/SNWSQ1/TOSD, */MTRMSC/TOSD, */STGDSNDM/TOSD]" | ||
transforms: | ||
- scale: 1000.0 | ||
filters: | ||
- bounding: | ||
variable: totalSnowDepth | ||
lowerBound: 0 | ||
upperBound: 10000000 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For my understanding:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I previously put this large number (10000m) here for removing the missing values. Here, we don't need this any more because we need to set the missing snod values to 0 when sogr satisfies the defined conditions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We might still have missing values though wouldn't we? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The missing values will be removed after applying sogr conditions. |
||
groundState: | ||
query: "[*/GRDSQ1/SOGR, */STGDSNDM/SOGR]" | ||
|
||
encoder: | ||
variables: | ||
|
@@ -65,11 +61,18 @@ encoder: | |
coordinates: "longitude latitude" | ||
source: variables/stationIdentification | ||
longName: "Identification of Observing Location" | ||
units: "m" | ||
units: "index" | ||
|
||
# ObsValue | ||
- name: "ObsValue/totalSnowDepth" | ||
coordinates: "longitude latitude" | ||
source: variables/totalSnowDepth | ||
longName: "Total Snow Depth" | ||
units: "mm" | ||
|
||
- name: "ObsValue/groundState" | ||
coordinates: "longitude latitude" | ||
source: variables/groundState | ||
longName: "STATE OF THE GROUND" | ||
units: "index" | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,9 +5,11 @@ | |
halo size: 250e3 | ||
obsdatain: | ||
engine: | ||
type: bufr | ||
type: script | ||
script file: "{{ PARMgfs }}/gdas/snow/ioda_bufr_python_encoder.py" | ||
args: | ||
input_path: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion:
This will make it clear that the script backend requires two inputs: bufr file and the mapping file There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @emilyhcliu for the suggestions. I made the changes as suggested. |
||
obsfile: '{{ DATA }}/obs/{{ OPREFIX }}sfcsno.tm00.bufr_d' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't forget to add an issue so they do not assume the formatting of the YAML file (obsfile variable shoould not be necessary). Reading the backend engines YAML breaks encapsulation (external modules should not be dependent on the private configuration for a backend type. Some backends may not have any kind of file associated with them (like a db...). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes thanks @rmclaren I'm aware of that issue, I/we/others need to figure out how best to handle it on the workflow side. |
||
mapping file: '{{ DATA }}/obs/bufr_sfcsno_mapping.yaml' | ||
obsdataout: | ||
engine: | ||
type: H5File | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jiaruidong2017 move this to https://github.com/NOAA-EMC/GDASApp/tree/develop/ush/ioda/bufr2ioda but change its name to something more descriptive/specific until we can move things to SPOC