Grid-Diag does not correctly ignore missing data values. #1562
Assignees
Labels
requestor: NCAR
National Center for Atmospheric Research
type: bug
Fix something that is not working
Milestone
Comments
JohnHalleyGotway
added
type: bug
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
…P. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing.
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
…P. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing.
This was
linked to
pull requests
Nov 16, 2020
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
* Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing.
JohnHalleyGotway
added a commit
that referenced
this issue
Nov 16, 2020
* Per #1562, add the same grid_diag fix for the develop branch. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing.
This was referenced Nov 17, 2020
jprestop
added a commit
that referenced
this issue
Nov 17, 2020
* Task 1455 doc (#1550) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Update Makefile.am PR #1550 broke the build. It removed the data/config/README file but left a reference to it in Makefile.am. I'm removing that reference directly in the develop branch to get the Docker build, nightly regression test, and nightly Fortify build working. * Bugfix 1554 develop ncdump (#1556) * Task 1455 doc (#1557) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Bugfix 1562 develop grid_diag (#1564) * Per #1562, add the same grid_diag fix for the develop branch. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu>
jprestop
pushed a commit
that referenced
this issue
Nov 17, 2020
* Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Bugfix 1554 main_v9.1 ncdump (#1555) * Bugfix 1562 main_v9.1 grid_diag (#1563) * Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure.
jprestop
added a commit
that referenced
this issue
Nov 18, 2020
* Task 1455 doc (#1550) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Update Makefile.am PR #1550 broke the build. It removed the data/config/README file but left a reference to it in Makefile.am. I'm removing that reference directly in the develop branch to get the Docker build, nightly regression test, and nightly Fortify build working. * Bugfix 1554 develop ncdump (#1556) * Task 1455 doc (#1557) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Bugfix 1562 develop grid_diag (#1564) * Per #1562, add the same grid_diag fix for the develop branch. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Removed references data/config/README * Feature 1528 plot_point_obs (#1560) * Per #1528, adding default config file for PlotPointObs tool. * Per #1528, added line_width to config file. * Per #1528, adding PlotPointObs config file and it actually compiles. Now I need to do all the work. * Per #1528, making some progress adding Observation objects and storing the unique locations in a set. Next, I need to parse the options from the config file. * Per #1528, add config constants for plot_point_obs options and update PlotInfo to include an on/off flag and colorbar flag. * Per #1528, update plot_point_obs config file options and plot_point_obs_conf_info.h/.cc to process them. * Per #1528, add a couple more config keys for Plot-Point-Obs. * Per #1528, add an obs_gc array to the default Plot-Point-Obs config file. * Per #1528, make a few local plotting functions global so that Plot-Point-Obs can call them. * Per #1528, lots of changes to Plot-Point-Obs. Getting closer. Still need to finish up coding, add tests, and update the documentation. * Per #1528, I found that PlotInfo.colorbar_spacing was never actually used in the code. So I removed it from the MODE and PlotPointObs config files and removed it from the PlotInfo object. * Per #1528, add timestring_to_time_t() utility function to vx_cal library. * Per #1528, cleanup... just removed commented out code. * Per #1528, update plot_point_obs.cc to parse/process the observation valid time correctly. * For #1528, change the library order to make the linker happy. * Per #1528, I had changed message_type to msg_typ but failed to updated the default config file. * Per #1527, making the usage statement slightly more concise. * Per #1528, changing the default line_color and fill_color entries to be empty arrays. * Per #1528, add in support for the convert and data censoring logic. * Per #1528, update the documentation for plot_point_obs to reflect the new usage. * 1528 Consider making the plotting options of plot_point_obs more configurable (#1559) * #1528 Added get_dim_size * #1528 Added qty_list * #1528 Get the quality flag string. Get the character dimensions from the variable * Per #1528, slight refomatting of source code for consistent line lengths. * Per #1528, setting the default dotsize to 1.0, as it was previously (not 10!). * Per #1528, fix logic so that if a fill color table is specified fill_point is set to true. Also adjust margins. * Per #1528, add a new call to plot_point_obs in unit_plot_point_obs.xml to exercise the new config file options. * Per #1528, only make the margins bigger if we're actually plotting a colorbar. Otherwise, retain the previous margin sizes. * Update met/data/config/PlotPointObsConfig_default Co-authored-by: jprestop <jpresto@ucar.edu> * Update PlotPointObsConfig_default * Update test/config/PlotPointObsConfig Co-authored-by: jprestop <jpresto@ucar.edu> * Consistent formatting * Per #1528, based on Michelle's feedback, add the -title string option to the plot_point_obs usage statement. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> * Feature 1568 ncl (#1570) * Per #1568, add python script to convert NCL *.rgb colormaps to MET *.ctable color tables. Also store 270 new or updated colortables that are the output from this script. Note that some of the existing colortables have changed ever so slightly. That's the result of some imprecise rounding earlier and more precise rounding now. * Per #1568, update and rerun the rgb2ctable.py conversion script to write a header at the top of each colortable file to list the source NCL colormap. Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu>
jprestop
pushed a commit
that referenced
this issue
Nov 20, 2020
* Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Bugfix 1554 main_v9.1 ncdump (#1555) * Bugfix 1562 main_v9.1 grid_diag (#1563) * Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Feature 1572 v9.1.1 (#1573) * Per #1572, delete the docs/version file as it is not needed here. * Per #1572, update the version number to 9.1.1. * Per #1572, list the met-9.1.1 release date as 20201119 for the docs. * Per #1572, add release notes for the met-9.1.1 verison. * Per #1572, add release notes for met-9.1.1 version. * Per #1572, let's try to release met-9.1.1 today 11/18. * Correct GitHub link. * Fix small typo in release notes.
jprestop
added a commit
that referenced
this issue
Nov 20, 2020
* Task 1455 doc (#1550) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Update Makefile.am PR #1550 broke the build. It removed the data/config/README file but left a reference to it in Makefile.am. I'm removing that reference directly in the develop branch to get the Docker build, nightly regression test, and nightly Fortify build working. * Bugfix 1554 develop ncdump (#1556) * Task 1455 doc (#1557) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Bugfix 1562 develop grid_diag (#1564) * Per #1562, add the same grid_diag fix for the develop branch. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Removed references data/config/README * Feature 1528 plot_point_obs (#1560) * Per #1528, adding default config file for PlotPointObs tool. * Per #1528, added line_width to config file. * Per #1528, adding PlotPointObs config file and it actually compiles. Now I need to do all the work. * Per #1528, making some progress adding Observation objects and storing the unique locations in a set. Next, I need to parse the options from the config file. * Per #1528, add config constants for plot_point_obs options and update PlotInfo to include an on/off flag and colorbar flag. * Per #1528, update plot_point_obs config file options and plot_point_obs_conf_info.h/.cc to process them. * Per #1528, add a couple more config keys for Plot-Point-Obs. * Per #1528, add an obs_gc array to the default Plot-Point-Obs config file. * Per #1528, make a few local plotting functions global so that Plot-Point-Obs can call them. * Per #1528, lots of changes to Plot-Point-Obs. Getting closer. Still need to finish up coding, add tests, and update the documentation. * Per #1528, I found that PlotInfo.colorbar_spacing was never actually used in the code. So I removed it from the MODE and PlotPointObs config files and removed it from the PlotInfo object. * Per #1528, add timestring_to_time_t() utility function to vx_cal library. * Per #1528, cleanup... just removed commented out code. * Per #1528, update plot_point_obs.cc to parse/process the observation valid time correctly. * For #1528, change the library order to make the linker happy. * Per #1528, I had changed message_type to msg_typ but failed to updated the default config file. * Per #1527, making the usage statement slightly more concise. * Per #1528, changing the default line_color and fill_color entries to be empty arrays. * Per #1528, add in support for the convert and data censoring logic. * Per #1528, update the documentation for plot_point_obs to reflect the new usage. * 1528 Consider making the plotting options of plot_point_obs more configurable (#1559) * #1528 Added get_dim_size * #1528 Added qty_list * #1528 Get the quality flag string. Get the character dimensions from the variable * Per #1528, slight refomatting of source code for consistent line lengths. * Per #1528, setting the default dotsize to 1.0, as it was previously (not 10!). * Per #1528, fix logic so that if a fill color table is specified fill_point is set to true. Also adjust margins. * Per #1528, add a new call to plot_point_obs in unit_plot_point_obs.xml to exercise the new config file options. * Per #1528, only make the margins bigger if we're actually plotting a colorbar. Otherwise, retain the previous margin sizes. * Update met/data/config/PlotPointObsConfig_default Co-authored-by: jprestop <jpresto@ucar.edu> * Update PlotPointObsConfig_default * Update test/config/PlotPointObsConfig Co-authored-by: jprestop <jpresto@ucar.edu> * Consistent formatting * Per #1528, based on Michelle's feedback, add the -title string option to the plot_point_obs usage statement. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> * Feature 1568 ncl (#1570) * Per #1568, add python script to convert NCL *.rgb colormaps to MET *.ctable color tables. Also store 270 new or updated colortables that are the output from this script. Note that some of the existing colortables have changed ever so slightly. That's the result of some imprecise rounding earlier and more precise rounding now. * Per #1568, update and rerun the rgb2ctable.py conversion script to write a header at the top of each colortable file to list the source NCL colormap. * Updated README and README_TC references in test/config and test/config/ref_config areas * Updated reference formatting and modified some versions to be consistent with other section of the User's Guide (#1577) * Feature 1574 rotlatlon (#1576) Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu>
JohnHalleyGotway
added a commit
that referenced
this issue
Dec 7, 2020
* Task 1455 doc (#1550) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Update Makefile.am PR #1550 broke the build. It removed the data/config/README file but left a reference to it in Makefile.am. I'm removing that reference directly in the develop branch to get the Docker build, nightly regression test, and nightly Fortify build working. * Bugfix 1554 develop ncdump (#1556) * Task 1455 doc (#1557) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Bugfix 1562 develop grid_diag (#1564) * Per #1562, add the same grid_diag fix for the develop branch. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Removed references data/config/README * Feature 1528 plot_point_obs (#1560) * Per #1528, adding default config file for PlotPointObs tool. * Per #1528, added line_width to config file. * Per #1528, adding PlotPointObs config file and it actually compiles. Now I need to do all the work. * Per #1528, making some progress adding Observation objects and storing the unique locations in a set. Next, I need to parse the options from the config file. * Per #1528, add config constants for plot_point_obs options and update PlotInfo to include an on/off flag and colorbar flag. * Per #1528, update plot_point_obs config file options and plot_point_obs_conf_info.h/.cc to process them. * Per #1528, add a couple more config keys for Plot-Point-Obs. * Per #1528, add an obs_gc array to the default Plot-Point-Obs config file. * Per #1528, make a few local plotting functions global so that Plot-Point-Obs can call them. * Per #1528, lots of changes to Plot-Point-Obs. Getting closer. Still need to finish up coding, add tests, and update the documentation. * Per #1528, I found that PlotInfo.colorbar_spacing was never actually used in the code. So I removed it from the MODE and PlotPointObs config files and removed it from the PlotInfo object. * Per #1528, add timestring_to_time_t() utility function to vx_cal library. * Per #1528, cleanup... just removed commented out code. * Per #1528, update plot_point_obs.cc to parse/process the observation valid time correctly. * For #1528, change the library order to make the linker happy. * Per #1528, I had changed message_type to msg_typ but failed to updated the default config file. * Per #1527, making the usage statement slightly more concise. * Per #1528, changing the default line_color and fill_color entries to be empty arrays. * Per #1528, add in support for the convert and data censoring logic. * Per #1528, update the documentation for plot_point_obs to reflect the new usage. * 1528 Consider making the plotting options of plot_point_obs more configurable (#1559) * #1528 Added get_dim_size * #1528 Added qty_list * #1528 Get the quality flag string. Get the character dimensions from the variable * Per #1528, slight refomatting of source code for consistent line lengths. * Per #1528, setting the default dotsize to 1.0, as it was previously (not 10!). * Per #1528, fix logic so that if a fill color table is specified fill_point is set to true. Also adjust margins. * Per #1528, add a new call to plot_point_obs in unit_plot_point_obs.xml to exercise the new config file options. * Per #1528, only make the margins bigger if we're actually plotting a colorbar. Otherwise, retain the previous margin sizes. * Update met/data/config/PlotPointObsConfig_default Co-authored-by: jprestop <jpresto@ucar.edu> * Update PlotPointObsConfig_default * Update test/config/PlotPointObsConfig Co-authored-by: jprestop <jpresto@ucar.edu> * Consistent formatting * Per #1528, based on Michelle's feedback, add the -title string option to the plot_point_obs usage statement. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> * Feature 1568 ncl (#1570) * Per #1568, add python script to convert NCL *.rgb colormaps to MET *.ctable color tables. Also store 270 new or updated colortables that are the output from this script. Note that some of the existing colortables have changed ever so slightly. That's the result of some imprecise rounding earlier and more precise rounding now. * Per #1568, update and rerun the rgb2ctable.py conversion script to write a header at the top of each colortable file to list the source NCL colormap. * Updated README and README_TC references in test/config and test/config/ref_config areas * Updated reference formatting and modified some versions to be consistent with other section of the User's Guide (#1577) * Feature 1574 rotlatlon (#1576) * Feature 1474 README (#1582) * Changed name of README and README_TC, modified references to those, and cleaned up some formatting. * Fixed formatting and language * Update data_io.rst * Update data_io.rst * Update config_options_tc.rst Co-authored-by: johnhg <johnhg@ucar.edu> * Added tilda files and temp files surrounded by # to .gitignore * Task 1455 doc (#1585) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC * small formatting changes * small formatting changes * fixing tabs * fixing spacing around number 11 * removing parenthesis around reference dates. * adding parenthesis back in. * fixing references * updating references * Update appendixC.rst Removed space from "HAUSDOR FF" * Update plotting.rst Changed a couple of references of Plot_Point_Obs to Plot-Point-Obs * Update point-stat.rst Added oxford commas Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Feature 1355 ioda (#1587) * #1355 Added makefile for ioda2.nc * #1355 Added ioda2nc * #1355 Added unit_ioda2nc.xml * #1355 Added yyyymmddThhmmss_to_unix and is_yyyymmddThhmmss * #1355 Added yyyymmddThhmmss_to_unix and is_yyyymmddThhmmss * #1355 Added parse_conf_metadata_map and parse_conf_obs_name_map * #1355 Added conf_key_obs_name_map, conf_key_metadata_map, and conf_key_missing_thresh * #1355 Exception handlijng at get_att_value_chars * #1355 Initial release * #1355 Initial release * #1355 Cleanup * #1355 Corretced NC_BYTE value * #1355 Initial release * #1355 Added IODA2NCConfig_efault * #1355 Added IODA2NCConfig_default * #1355 Turn off time_summary * #1355 Changed missing_thresh +-1e9 * #1355 Terminate string * #1355 Corrected echo statement * #1355 Removed unused variable * #1355 Make the string null terminated * #1355 Move the IODA2NCConfig_default to above to avoid merge conflict * #1355 To aoide merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * #1355 To avoid a merge conflict * Per #1355, add .gitignore file for ioda2nc. * Per #1355, had to add test stub for the new ioda2nc tool to enable 'make test' to run. * Per #1355, tweak met/scripts/Makefile to NOT ignore the ENABLE_PYTHON configuration option when constructing the list of tests. * Per #1355, ignore the scripts/ioda2nc file. Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> * Per #1590, change V10.0 to V10.0.0 to make use of X.Y.Z version numbering. (#1591) * Task_1455_doc (#1595) * first stab at converting to sphinx * removing all slashes * adding new link to README.rst file * working on lists * Made formatting changes * Finished fcst section * fixing spelling, bolding and italics issues * updating web links * working on formatting * updating formatting * formatting * first attempt to clean up formatting completed. * adding README to the index file * fixing warning errors * Bringing README_TC into sphinx. Updating section headers * Adding README_TC * Made formatting updates to README.rst * corrected section under wavelet * small changes * removing met/data/config/README since it is now in met/docs/Users_Guide * Added some formatting for headers * fixing chapters & sections * Fixed warnings from building * adding in code blocks * removing slashes * changes * Made changes to formatting * removing For example code blocks * major updates * fist pass at document conversion complete. * cleaning up questions about dashes * Made some formatting modifications * Removing README_TC because it is being replaced by README_TC.rst in met/docs/Users_Guide * Removing the reference to the README_TC file * Making title capitalization consistent with README * Added a space in timestring * changing to 'time string' with a space between the words. * adding a link to the new README_TC location in met/docs/Users_Guide * Modified references to README and README_TC * small formatting changes * small formatting changes * fixing tabs * fixing spacing around number 11 * removing parenthesis around reference dates. * adding parenthesis back in. * fixing references * updating references * Update appendixC.rst Removed space from "HAUSDOR FF" * Update plotting.rst Changed a couple of references of Plot_Point_Obs to Plot-Point-Obs * Update point-stat.rst Added oxford commas * bolding Config and italizing directory names. * Modified format * italicizing directories. * removed extra tool * italicizing directories * bolding * Update plotting.rst Changed Plot_Point_Obs to Plot-Point-Obs Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> * Per #1598, update comment in all the MET config files. (#1599) * Feature 1494 ens_stat (#1600) Merging into the develop branch to get these changes included for the met-10.0.0-beta2 development release. These are good and worthwhile changes to make on their own. However, I do still need @j-opatz to confirm that they actually fix the original issue with python embedding in ensemble-stat! * Per #1494, add DataPlane::is_all_bad_data() function to determine whether the field is all bad data. * Per #1494, update the process_data_plane() function in 2 ways. First, change its return from void to bool. And simply return false, if the raw input data is all bad data. Second, print debug(4) log messages to list out the range of valid data and timing information. It'll be really nice to make this consistent across all the input file types. * Per #1494, update the logic for all the calls to process_data_plane() to account for the fact that it now returns a bool instead of void. * Per #1494, refine the Data plane debug(4) log messages. * Per #1494, work on log message. * Per #1494, updated vx_data2d_python library to call the common process_data_plane() function to apply data shifting, censoring, conversion, and check for all missing data. * Per #1494, just fixing a small typo in an unrelated source code file that I ran across when testing for met-help. Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Julie.Prestopnik <jpresto@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@kiowa.rap.ucar.edu> Co-authored-by: hsoh-u <hsoh@ucar.edu>
JohnHalleyGotway
added a commit
that referenced
this issue
Jan 28, 2021
* Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Bugfix 1554 main_v9.1 ncdump (#1555) * Bugfix 1562 main_v9.1 grid_diag (#1563) * Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Feature 1572 v9.1.1 (#1573) * Per #1572, delete the docs/version file as it is not needed here. * Per #1572, update the version number to 9.1.1. * Per #1572, list the met-9.1.1 release date as 20201119 for the docs. * Per #1572, add release notes for the met-9.1.1 verison. * Per #1572, add release notes for met-9.1.1 version. * Per #1572, let's try to release met-9.1.1 today 11/18. * Correct GitHub link. * Fix small typo in release notes. * Bugfix 1618 main v91 pb2nc (#1622) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Update pull_request_template.md * Per #1638, apply the same 3 fixes to the main_v9.1 branch to be included in the next bugfix release for met-9.1. (#1640) Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu>
JohnHalleyGotway
added a commit
that referenced
this issue
Mar 12, 2021
* Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Bugfix 1554 main_v9.1 ncdump (#1555) * Bugfix 1562 main_v9.1 grid_diag (#1563) * Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Feature 1572 v9.1.1 (#1573) * Per #1572, delete the docs/version file as it is not needed here. * Per #1572, update the version number to 9.1.1. * Per #1572, list the met-9.1.1 release date as 20201119 for the docs. * Per #1572, add release notes for the met-9.1.1 verison. * Per #1572, add release notes for met-9.1.1 version. * Per #1572, let's try to release met-9.1.1 today 11/18. * Correct GitHub link. * Fix small typo in release notes. * Bugfix 1618 main v91 pb2nc (#1622) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Update pull_request_template.md * Per #1638, apply the same 3 fixes to the main_v9.1 branch to be included in the next bugfix release for met-9.1. (#1640) * Per #1646, one line fix for cut-and-paste error. (#1648) * Adding necessary files for ReadTheDocs (#1703) * Per #1706, update aggr_ecnt_lines() to also add an entry to the skip_ba array of booleans. This is used to keep track of whether or not the point should be excluded from the statistics and is used in the Ensemble-Stat. This same aggregation code in the vx_statistics library is used by Stat-Analysis. Since Stat-Analysis failed to populate that array, it led to an array parsing error in ECNTInfo::set() when trying to compute the sum of the weights. (#1707) * Per #1694, exactly the same set of bugfix changes but applied to main_v9.1 rather than the develop branch. (#1709) * Feature 1710 v9.1.2 (#1711) * Per #1710, update the release notes and version numbers for the 9.1.2 release. * Per #1710, add a release note about the move to read the docs. Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu>
JohnHalleyGotway
added a commit
that referenced
this issue
Mar 20, 2021
* Add debug level 4 message to list out the number of GRIB2 records inventoried. This helps debugging issues with MET potentially not reading all input GRIB2 records on WCOSS. * Bugfix 1554 main_v9.1 ncdump (#1555) * Bugfix 1562 main_v9.1 grid_diag (#1563) * Per #1562, check for bad data values before adding data to the PDF's for grid_diag. * Per #1562, removing the poly = CONUS.poly mask from GridDiagConfig_TMP. That settting masked a problem in the handling of missing data. Exercising the mask.poly option is tested in another unit test. This will change the output and break the nightly build, but that's good since we'll do more thorough testing. * Per #1508, change the verbosity in unit_tc_gen.xml from -v 2 to -v 5 to print out some additional log messages that may help in debugging the intermittent file list failure. * Feature 1572 v9.1.1 (#1573) * Per #1572, delete the docs/version file as it is not needed here. * Per #1572, update the version number to 9.1.1. * Per #1572, list the met-9.1.1 release date as 20201119 for the docs. * Per #1572, add release notes for the met-9.1.1 verison. * Per #1572, add release notes for met-9.1.1 version. * Per #1572, let's try to release met-9.1.1 today 11/18. * Correct GitHub link. * Fix small typo in release notes. * Bugfix 1618 main v91 pb2nc (#1622) Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> * Update pull_request_template.md * Per #1638, apply the same 3 fixes to the main_v9.1 branch to be included in the next bugfix release for met-9.1. (#1640) * Per #1646, one line fix for cut-and-paste error. (#1648) * Adding necessary files for ReadTheDocs (#1703) * Per #1706, update aggr_ecnt_lines() to also add an entry to the skip_ba array of booleans. This is used to keep track of whether or not the point should be excluded from the statistics and is used in the Ensemble-Stat. This same aggregation code in the vx_statistics library is used by Stat-Analysis. Since Stat-Analysis failed to populate that array, it led to an array parsing error in ECNTInfo::set() when trying to compute the sum of the weights. (#1707) * Per #1694, exactly the same set of bugfix changes but applied to main_v9.1 rather than the develop branch. (#1709) * Feature 1710 v9.1.2 (#1711) * Per #1710, update the release notes and version numbers for the 9.1.2 release. * Per #1710, add a release note about the move to read the docs. * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Do not combined if there are no overlapping beteewn TQZ and UV records * #1715 Added blank line for Warning * #1715 Added a blank line for Error * Bugfix 1716 main_v9.1 perc_thresh (#1721) * Per #1716, committing the same bugfix to the main_v9.1 branch for inclusion in the 9.1.3 bugfix release. * Per #1716, change SFP50 example to SFP33.3 to show an example of using floating point percentile values. * Per #1723, update the verison number to 9.1.3. (#1730) * Update pull_request_template.md Co-authored-by: hsoh-u <hsoh@ucar.edu> Co-authored-by: Howard Soh <hsoh@kiowa.rap.ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the Problem
This issue was found when applying the Grid-Diag tool to the Air Force cloud evaluation project.
When missing data values are present in the input gridded data, Grid-Diag includes their values in the resulting 1D and 2D PDF's.
Expected Behavior
The Grid-Diag tool should ignore missing data values and not include them in the output.
Environment
Describe your runtime environment:
1. Machine: behavior found on Cheyenne but reproduced anywhere
2. Software version number(s): bug in MET version 9.1
To Reproduce
Describe the steps to reproduce the behavior:
1. Edit test/config/GridDiagConfig_TMP
2. Reset mask.poly to an empty string to turn it off.
3. Run the unit test: perl/unit.pl xml/unit_grid_diag.xml
4. Use ncdump to inspect the output:
The ratio of 9 matches the number of input files which means that all point, including the bad data values are included.
Relevant Deadlines
Include fix in met-9.1.1.
Funding Source
Define the source of funding and account keys here or state NONE.
Define the Metadata
Assignee
Labels
Projects and Milestone
Define Related Issue(s)
Consider the impact to the other METplus components.
None.
Bugfix Checklist
See the METplus Workflow for details.
Branch name:
bugfix_<Issue Number>_main_<Version>_<Description>Pull request:
bugfix <Issue Number> main_<Version> <Description>Select: Reviewer(s), Project(s), Milestone, and Linked issues
Branch name:
bugfix_<Issue Number>_develop_<Description>Pull request:
bugfix <Issue Number> develop <Description>The text was updated successfully, but these errors were encountered: