Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for EPA AirNow ASCII data in ASCII2NC #2142

Closed
2 of 16 tasks
DanielAdriaansen opened this issue Apr 19, 2022 · 80 comments · Fixed by #2256 or #2373
Closed
2 of 16 tasks

Add support for EPA AirNow ASCII data in ASCII2NC #2142

DanielAdriaansen opened this issue Apr 19, 2022 · 80 comments · Fixed by #2256 or #2373
Assignees
@davidalbo @DanielAdriaansen @PerryShafran-NOAA
Labels
MET: PreProcessing Tools (Point) priority: high High Priority reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NOAA/other NOAA Laboratory, not otherwise specified required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project type: new feature Make it do something new
Milestone
MET 11.0.0

Comments

@DanielAdriaansen
Copy link
Contributor

DanielAdriaansen commented Apr 19, 2022 •
edited by JohnHalleyGotway
Loading

Describe the New Feature

Per dtcenter/METplus#1515 and @PerryShafran-NOAA, the EPA will switch from providing BUFR to providing ASCII data to NOAA. The new feature is to add support for this new dataset to ASCII2NC.

Acceptance Testing

Sample files exist on seneca here: /home/dadriaan/projects/airnow/shafran_data/

From dtcenter/METplus#1515:

There are several files here to look at, but they both provide the same data. The file with all the data is the HourlyAQObs data. The HourlyData is a neater looking file with the same data, but the lat/lon of each station is not provided there, so you'll have to use the HourlyAQObs file, which is a CSV file.

It would be great to be able to also read in the daily file, which I think we can use to validate daily model items. That is the daily_data.dat file, and there are two of them. Unfortunately, they are in the HourlyData format, but you should be able to use the lat/lon of the HourlyAQObs file to find these stations. Here's the fact sheet on the daily file:

As you can see, they don't make this easy. However, we could just stick with hourly data and just sum/average them the way we are doing now in PB2NC.

There are four file types:

  1. "HourlyAQObs" (Docs: HourlyAQObsFactSheet.pdf)
  2. "HourlyData" (Docs: HourlyDataFactSheet.pdf)
  3. "daily_data" (Docs: unknown)
  4. "daily_data_v2" (Docs: DailyDataFactSheet.pdf)

The documentation for file type 4) only describes "daily_data_v2", and no information was provided about the "daily_data" file from the user. @JohnHalleyGotway notes:

...looking at a diff of the two, I recommend using daily_data_v2.dat. The former does NOT contain lat/lon info for each site and would require a lookup table for that. The v2 version DOES contain the lat/lon info (it would seems) and would therefore be simpler to process.

Time Estimate

1 day of work

Sub-Issues

  • Document different file type metadata
  • Add support for reading these files in ASCII2NC
  • Test support

Relevant Deadlines

NONE.

Funding Source

2792541

Define Related Issue(s)

Consider the impact to the other METplus components.

  • METplus: need to update ASCII2NC wrapper

New Feature Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Linked issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@DanielAdriaansen DanielAdriaansen added type: new feature Make it do something new priority: low Low Priority requestor: NOAA/other NOAA Laboratory, not otherwise specified alert: NEED MORE DEFINITION Not yet actionable, additional definition required alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle MET: PreProcessing Tools (Point) labels Apr 19, 2022
@DanielAdriaansen DanielAdriaansen added this to the MET 11.0.0 milestone Apr 19, 2022
@DanielAdriaansen
Copy link
Contributor Author

@JohnHalleyGotway I wasn't sure what milestone, project, or requestor to select but I did my best. Same with priority.

@JohnHalleyGotway JohnHalleyGotway added required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project priority: high High Priority and removed alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle priority: low Low Priority labels Jun 13, 2022
@davidalbo
Copy link
Contributor

@DanielAdriaansen I'm looking into this now. A few initial questions.

Is the current way it is done (BUFR) in some other app that is part of MET? If so does it write netCDF?
Should I use that (if yes) to figure out how to convert these ascii files to netCDF?
(I'm wondering if the BUFR content is equivalent to the ascii content I'm looking at in your examples).

I'm sure I'll have more questions, still have a pretty big learning curve for work in MET.

@DanielAdriaansen
Copy link
Contributor Author

@JohnHalleyGotway are you aware of how Perry is processing the EPA BUFR data now, or do we need to reach out to him for a sample/example of their current workflow and datasets using BUFR?

@PerryShafran-NOAA
Copy link

I can help answer that. Currently, we use PB2NC to read in the prepbufr and write out a netCDF file to be used in point_stat. We actually do this several ways - directly convert the hourly data to hourly data, but we also need to write out a) the daily max, and b) the daily average. And these maxes and averages need to be summed over a period of 5Z to 5Z. I believe that both/either @JohnHalleyGotway and/or @georgemccabe assisted with that several years ago.

@davidalbo
Copy link
Contributor

Thanks for the quick response. As I'm pretty green on all of this, an example would be really useful so I could run it in the debugger and figure out how PB2NC does what you want, and how that might be added to ascii2nc. What that would mean is a data example, config file, and command line interface. Maybe start with the hourly data?

@PerryShafran-NOAA
Copy link

I could gather all that up and send that to you after the weekend. The hourly part is pretty straightforward.

The issue that we here at EMC have somewhat been struggling with is if we provided you enough ASCII data from the EPA for you to do this. I am assuming that we have, so I suggest continuing as if we did and then if there are any issues see if we need additional data from the EPA. From what I had seen from the ASCII examples and documentation that we had, I think we sent you everything you need to do this work.

It also has been suggested to us that the DTC is using code from the MONET interface to be able to read this and convert to ASCII. I'm curious if you were aware of that. Perhaps we can lift from that code to assist you in this development?

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Jul 7, 2022 •
edited by TaraJensen
Loading

@davidalbo please take a look at this MONET repo: https://github.com/noaa-oar-arl/monet and ask Barry Baker @bbakernoaa and/or David Fillmore @davidfillmore to see if support for this format already exists there.

@PerryShafran-NOAA - any progress on getting some sample data for us?

@bbakernoaa
Copy link

bbakernoaa commented Jul 7, 2022 •
edited
Loading

@davidalbo please look at https://github.com/noaa-oar-arl/monetio/blob/stable/monetio/obs/airnow.py

@zmoon

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Jul 11, 2022 •
edited
Loading

Hi,

(1) Both HourlyAQObs* and HourlyData* are hourly dataset, but HourlyAQObs* will not change after its initial posted time. HourlyData* will be updated again after its initial post. My assumption is that it will include the additional obs that arrived late. Thus, HourlyData* are prefer hourly dataset to be used for this new apps.

You can see the time stamp changed from this example
https://files.airnowtech.org/?prefix=airnow/2022/20220708/

(2) However, Perry found there is no (lat,lon) information in HourlyData* . As you can tell from the link above, both hourly files are not small. To pull in both and simply read the site location from HourlyAQObs* seems to waster the time for daily fetch of the data from the EPA to the NCO. We later found two location files and wonder whether they are sufficient to provide the sit location of those reported in HourlyData*, monitoring_site_locations.dat or Monitoring_Site_Locations_V2.dat or both. That is to use the site id reported in HourlyData* to find the (lat,lon) in the two location files described above.

EMC wants to work with the DTC and provide what we know to speed up this development.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 9, 2022 via email •
edited by JohnHalleyGotway
Loading

JohnHalleyGotway added a commit that referenced this issue Dec 9, 2022
@JohnHalleyGotway
Feature #2142 airnow_take2 (#2373)
8a910a6
@Ho-ChunHuang-NOAA
Copy link

Hi, John:

Can you give me your suggestion?

Users who used EPA ASCII AirNOW data reported that there is a negative values in the concentration field. People suggest it should be treated as concentration =0.

Is there a option in ASCII2NC to reset the values to 0 if it is negative and if yes can you show to do it?

Or we can reset the value at the POINT_STAT? i.e., using OBS_VAR1_OPTIONS =. I do not want to to OBS_VAR1_THRESH = >=0 because it is a valid data.

I try to go to discussion and search for the negative or reset, but the keyword search did not come back with useful information.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 10, 2022 •
edited
Loading

Are these the correct setting in POINT_STAT?

FCST_VAR1_NAME = OZCON
FCST_VAR1_LEVELS = A1
FCST_VAR1_OPTIONS = set_attr_name = "OZCON1";
OBS_VAR1_NAME= OZONE
OBS_VAR1_LEVELS= A1
OBS_VAR1_OPTIONS = message_type = "AIRNOW_HOURLY_AQOBS"; censor_thresh = [ <=0.0 ]; censor_val=[ 0.0 ];

FCST_VAR1_NAME = PMTF
FCST_VAR1_LEVELS = L1
OBS_VAR1_NAME= PM25
OBS_VAR1_LEVELS= A1
OBS_VAR1_OPTIONS = message_type = "AIRNOW_HOURLY_AQOBS"; censor_thresh = [ <=0.0 ]; censor_val=[ 0.0 ];

@Ho-ChunHuang-NOAA
Copy link

John:

I successfully create the hourly o3/pm25 statistic, but I encounter problem using daily files.

matching criteria is
FCST_VAR1_NAME = OZMAX1
FCST_VAR1_LEVELS = L1
FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04";
OBS_VAR1_NAME= OZONE-1HR
OBS_VAR1_LEVELS=A1
OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

FCST_VAR1_NAME = OZMAX8
FCST_VAR1_LEVELS = L1
FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_11";
OBS_VAR1_NAME= OZONE-8HR
OBS_VAR1_LEVELS=A8
OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

FCST_VAR1_NAME = PMTF
FCST_VAR1_LEVELS = A23
FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04";set_attr_name = "PMAVE";
OBS_VAR1_NAME= PM2.5-24hr
OBS_VAR1_LEVELS=A24
OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2"; valid_time= "{valid?fmt=%Y%m%d}_000000";

I still have problem in daily ascii file. in obs_var name and valid time
DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region SRockies, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4078
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3081
DEBUG 2: Rejected: valid time = 997

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4078
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3125
DEBUG 2: Rejected: valid time = 953

DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4078
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 2851
DEBUG 2: Rejected: valid time = 1227

@Ho-ChunHuang-NOAA
Copy link

Is "-" and "." allow in the point_stat variable name matching ? Should I used the quotation mark " in the OBS_VAR1_NAME?

@Ho-ChunHuang-NOAA Ho-ChunHuang-NOAA mentioned this issue Dec 11, 2022
Merged
15 tasks
@Ho-ChunHuang-NOAA
Copy link

set new obs window

12/12 17:49:32.519 metplus INFO: METPLUS_FCST_FIELD=field = [{ name="OZMAX1"; level="L1"; valid_time= "20221119_04"; }];
12/12 17:49:32.519 metplus INFO: METPLUS_INTERP_DICT=
12/12 17:49:32.519 metplus INFO: METPLUS_MASK_GRID=grid = [];
12/12 17:49:32.519 metplus INFO: METPLUS_MASK_POLY=poly = [];
12/12 17:49:32.519 metplus INFO: METPLUS_MASK_SID=sid = [];
12/12 17:49:32.519 metplus INFO: METPLUS_MESSAGE_TYPE=message_type = ["ONLYSF"];
12/12 17:49:32.519 metplus INFO: METPLUS_MET_CONFIG_OVERRIDES=
12/12 17:49:32.519 metplus INFO: METPLUS_MODEL=model = "PROD_ASCII";
12/12 17:49:32.519 metplus INFO: METPLUS_OBS_FIELD=field = [{ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }];
12/12 17:49:32.519 metplus INFO: METPLUS_OBS_QUALITY=
12/12 17:49:32.519 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = 864000000;end = -864000000;}
12/12 17:49:32.519 metplus INFO: METPLUS_OUTPUT_FLAG_DICT=
12/12 17:49:32.519 metplus INFO: METPLUS_OUTPUT_PREFIX=
12/12 17:49:32.519 metplus INFO: METPLUS_REGRID_DICT=regrid = {to_grid = NONE;}
12/12 17:49:32.520 metplus INFO: MET_TMP_DIR=/lfs/h2/emc/physics/noscrub/ho-chun.huang/metplus_aq/tmp
12/12 17:49:32.520 metplus INFO: MODEL=PROD_ASCII
12/12 17:49:32.520 metplus INFO: OBS_FIELD={ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }
12/12 17:49:32.520 metplus INFO: OBS_WINDOW_BEGIN=864000000
12/12 17:49:32.520 metplus INFO: OBS_WINDOW_END=-864000000

result are the same

DEBUG 1: Forecast File: /lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2
DEBUG 1: Observation File: /lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom_ascii2nc_airnow/20221118/airnow_daily_20221118.nc
DEBUG 3: MetGrib2DataFile::data_plane_array() -> Found exact match for "OZMAX1/L1" in GRIB2 record 1 field 1 of GRIB2 file "/lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2"

::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3111
DEBUG 2: Rejected: valid time = 977

@Ho-ChunHuang-NOAA
Copy link

For OZMAX8

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3143
DEBUG 2: Rejected: valid time = 945

For PMAVE
DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 2841
DEBUG 2: Rejected: valid time = 1247

@JohnHalleyGotway
Copy link
Collaborator

@Ho-ChunHuang-NOAA, please try multiplying your settings by -1. Here's what I see in the log messages you sent:

12/12 17:49:32.519 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = 864000000;end = -864000000;}

These are both relative to the forecast valid time. The end value should be >= beg value. But you have the signs switched. Please try setting beg = -864000000 and end = 864000000.

Perhaps we should add a sanity check, and have MET error out if end < beg?

@Ho-ChunHuang-NOAA
Copy link

Update the obs_window, the error message are the same

12/12 19:50:18.452 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -864000000;end = 864000000;}

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3111
DEBUG 2: Rejected: valid time = 977

DEBUG 2: Processing OZMAX8/L1 versus OZONE-8HR/A8, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 3143
DEBUG 2: Rejected: valid time = 945

DEBUG 2: Processing PMTF/A23 versus PM2.5-24hr/A24, for observation type AIRNOW_DAILY_V2, over region CONUS_Central, for interpolation method NEAREST(1), using 0 matched pairs.
DEBUG 2: Number of matched pairs = 0
DEBUG 2: Observations processed = 4088
DEBUG 2: Rejected: station id = 0
DEBUG 2: Rejected: obs var name = 2841
DEBUG 2: Rejected: valid time = 1247

@JohnHalleyGotway JohnHalleyGotway mentioned this issue Dec 12, 2022
Merged
@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented Dec 12, 2022 •
edited
Loading

@Ho-ChunHuang-NOAA, I asked Julie to pull down those two sample files from WCOSS2:

DEBUG 1: Forecast File: /lfs/h2/emc/ptmp/ho-chun.huang/com/aqm/prod_ascii/cs.20221118/aqm.t06z.max_1hr_o3.148.grib2
DEBUG 1: Observation File: /lfs/h2/emc/vpppg/noscrub/ho-chun.huang/dcom_ascii2nc_airnow/20221118/airnow_daily_20221118.nc

And I ran Point-Stat directly with this data:

/d1/projects/MET/MET_releases/met-10.1.0/bin/point_stat \
aqm.t06z.max_1hr_o3.148.grib2 airnow_daily_20221118.nc \
PointStatConfig.txt -outdir out -v 3 -log run_ps.log

I used MET version 10.1.0 because presumably that's what you're running on WCOSS2.
Point-Stat (both versions 10.1.0 and 11.0.0) produce 888 matched pairs:

DEBUG 2: Processing OZMAX1/L1 versus OZONE-1HR/A1, for observation type AIRNOW_DAILY_V2, over region FULL, for interpolation method NEAREST(1), using 888 matched pairs.
DEBUG 3: Number of matched pairs   = 888
DEBUG 3: Observations processed    = 4088
DEBUG 3: Rejected: obs var name    = 3111
DEBUG 3: Rejected: off the grid    = 89

Note that you should probably remove the valid_time setting from:

12/12 17:49:32.520 metplus INFO: OBS_FIELD={ name="OZONE-1HR"; level="A1"; message_type = "AIRNOW_DAILY_V2"; valid_time= "20221118_000000"; }

It serves no purpose here. Instead the obs_window settings are used to filter point observations by time.

Here are my config and log files:
PointStatConfig.txt
run_ps.log

Perhaps you could try running Point-Stat directly using this config file to take the METplus python wrappers out of the equation while debugging?

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 12, 2022 via email

@Ho-ChunHuang-NOAA
Copy link

I remove the OBS_VAR1_OPTIONS validtime description and re-run, I got the same result as with validtime description.

This is the only place I specify the OBS_VAR1_OPTIONS with valid_time.
FCST_VAR1_NAME = OZMAX1
FCST_VAR1_LEVELS = L1
FCST_VAR1_OPTIONS = valid_time= "{valid?fmt=%Y%m%d?shift=1d}_04";
OBS_VAR1_NAME= OZONE-1HR
OBS_VAR1_LEVELS=A1
OBS_VAR1_OPTIONS = message_type = "AIRNOW_DAILY_V2";

https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/run_ozmax1_2142.log

@JohnHalleyGotway
Copy link
Collaborator

Thanks for sending that log file. I'm pretty stumped. Its as if the METplus obs_window settings are actually being used by the Point-Stat tool. I clearly see that the environment variable is being set the way we want:

12/12 22:00:13.342 metplus INFO: METPLUS_OBS_WINDOW_DICT=obs_window = {beg = -864000000;end = 864000000;}

Next, I'd try running MET at verbosity level 10 (-v 10). Here's what I'm looking for in the output:

|    |    Name  = obs_window
|    |    Type  = DictionaryType
|    |    Dict/Array Value = 
|    |    |    Nentries = 2
|    |    |    Address  = 0x56158e382010
|    |    |    Parent   = 0x56158d542aa8
|    |    |    IsArray  = false
|    |    |    Dictionary Entry[0] ...
|    |    |    |    Name  = beg
|    |    |    |    Type  = IntegerType
|    |    |    |    Integer Value = -864000000
|    |    |    Dictionary Entry[1] ...
|    |    |    |    Name  = end
|    |    |    |    Type  = IntegerType
|    |    |    |    Integer Value = 864000000

BUT this is not written to the log output, its written to standard out. I'm not sure if METplus captures that output or not.

This would confirm that MET is parsing the obs_window as expected. If you see in your output that its some value other than 864000000, that'd indicate some issue with config options being overwritten.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 14, 2022 via email

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 14, 2022 via email •
edited by JohnHalleyGotway
Loading

@georgemccabe
Copy link
Collaborator

Hi @Ho-ChunHuang-NOAA, it looks like you are using your own MET config file that does not contain the environment variables that are set by the METplus wrappers. There are many warnings in your log file that mention that the environment variables are not utilized:

12/12 22:00:13.331 metplus (command_builder.py:153) WARNING: Environment variable ${METPLUS_OBS_WINDOW_DICT} is not utilized in MET config file: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/parm/met_config/PointStatConfig_AIRNOW_max_Bukovsky_G148

Removing or commenting out POINT_STAT_CONFIG_FILE in your METplus config file will use the wrapped MET config file that is provided with the METplus wrappers (METplus/parm/met_config/PointStatConfig_wrapped) and the values from the environment variables set by the wrapper will be used. If you have changed any values from the defaults in PointStatConfig_AIRNOW_max_Bukovsky_G148, then you will need to set the appropriate settings in your METplus config file. You can compare this file to the default config file found in the share/met/config directory of the MET install. The path to the default file will also be output in the point_stat log output. If you have issues reconciling the differences, you could send me the PointStatConfig_AIRNOW_max_Bukovsky_G148 file.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 14, 2022 via email •
edited by JohnHalleyGotway
Loading

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 14, 2022 via email

@georgemccabe
Copy link
Collaborator

Hi @Ho-ChunHuang-NOAA, I was suggesting that you use the PointStatConfig_wrapped file that is provided with the METplus install instead of modifying the defaults by hand. Using the PointStatConfig_AIRNOW_max_test file still produces the warnings that the environment variables are not in your file, so it does not include the variable that sets the obs_window values. It also referenced environment variables that are deprecated and will be soon removed from support (only environment variables that start with METPLUS_ will be set by METplus to use in MET config files).

You should set the following in your METplus config file:

POINT_STAT_CONFIG_FILE = {PARM_BASE}/met_config/PointStatConfig_wrapped
POINT_STAT_MASK_POLY = /lfs/h2/emc/vpppg/noscrub/emc.vpppg/verification/EVS_fix/masks/Bukovsky_G148_CONUS.nc
POINT_STAT_OUTPUT_FLAG_FHO = STAT
POINT_STAT_OUTPUT_FLAG_CTC = STAT
POINT_STAT_OUTPUT_FLAG_SL1L2 = STAT
POINT_STAT_MET_CONFIG_OVERRIDES = cat_thresh = [ >50, >60, >65, >70, >75, >85, >105, >125, >150 ]; message_type   = [];

This should set the desired overrides and also read the obs_window values that you have set. Let me know if this does not fix the issue.

I'm not sure which METplus config file to put these changes because it looks like you are passing in many config files to run_metplus.py. Previously I had helped @PerryShafran-NOAA update a CAM use case to make changes like this and reduce the number of config files used to run in this discussion. It would be good to update all of the use cases that you run in this way. I am happy help with these updates and/or meet with you to discuss what is needed to change.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 14, 2022 via email

@georgemccabe
Copy link
Collaborator

@Ho-ChunHuang-NOAA, it looks like you are running METplus 4.0.0 using the param file from 4.1.4. METPLUS_HSS_EC_VALUE is not set by METplus 4.0, but it is set in 4.1. The environment variable is referenced in the 4.1.4 config file, but it is not set by METplus 4.0.

@Ho-ChunHuang-NOAA
Copy link

Ho-ChunHuang-NOAA commented Dec 15, 2022 via email

@georgemccabe
Copy link
Collaborator

The last log file you sent shows that you are running METplus v4.0.0:

12/14 16:49:29.713 metplus INFO: Running METplus v4.0.0 called with command: /lfs/h2/emc/vpppg/save/ho-chun.huang/METplus-4.0.0/ush/run_metplus.py ...

After you run the module commands, could you run which run_metplus.py to see which version of that script is being used? It looks like it should be 4.1.4, but that isn't consistent with what is shown in the logs.

@georgemccabe
Copy link
Collaborator

@Ho-ChunHuang-NOAA, you could also check the call to run_metplus.py. The command may include the full path to the script which may point to an older version.

JohnHalleyGotway added a commit that referenced this issue Jan 19, 2023
@JohnHalleyGotway
Per #2142, ci-run-unit refine climo log messages to list time offsets…
de4aca7 
… in hours and days instead of seconds.
JohnHalleyGotway pushed a commit that referenced this issue Jan 19, 2023
Per #2142, testing interpolation for the date 20220214_12 revealed an…
b271c5d 
…other bug. With day_interval = 31, it find climo data for Jan 15, Feb 15, and Mar 15. The climo_hms_interp() function is smart enough to correctly interpolate between Jan 15 and Feb 15 and ignore Mar 15. However, there was a bug in the computation of the target valid time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidalbo davidalbo

@DanielAdriaansen DanielAdriaansen

@PerryShafran-NOAA PerryShafran-NOAA

Labels
MET: PreProcessing Tools (Point) priority: high High Priority reporting: DTC NOAA R2O NOAA Research to Operations DTC Project requestor: NOAA/other NOAA Laboratory, not otherwise specified required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project type: new feature Make it do something new
Projects
No open projects
MET-11.0.0 Pre-Release (12/9/22)
Status: Done
MET-11.0.0-beta3 (9/21/22)
Status: Done
Milestone
MET 11.0.0
7 participants
@davidalbo @DanielAdriaansen @JohnHalleyGotway @bbakernoaa @georgemccabe @Ho-ChunHuang-NOAA @PerryShafran-NOAA