diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json index 5684f4f2cd..6963d1127e 100644 --- a/.github/parm/use_case_groups.json +++ b/.github/parm/use_case_groups.json @@ -194,11 +194,6 @@ "index_list": "5", "run": false }, - { - "category": "s2s", - "index_list": "6", - "run": false - }, { "category": "s2s_mid_lat", "index_list": "0-2", @@ -224,6 +219,11 @@ "index_list": "4", "run": false }, + { + "category": "s2s_stratosphere", + "index_list": "0", + "run": false + }, { "category": "short_range", "index_list": "0", diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index 2deea9dba5..a8ef08d9a2 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -56,6 +56,7 @@ one of the following: * s2s (Subseasonal to Seasonal) * s2s_mid_lat (Subseasonal to Seasonal: Mid-Latitude) * s2s_mjo (Subseasonal to Seasonal: Madden-Julian Oscillation) +* s2s_stratosphere (Subseasonal to Seasonal: Stratosphere) * short_range (formerly convection_allowing_models) * space_weather * tc_and_extra_tc (Tropical Cyclone and Extratropical Cyclone) @@ -150,18 +151,24 @@ Use Case Rules - The use case should be run by someone other than the author to ensure that it runs smoothly outside of the development environment set up by the author. -.. _memory-intense-use-cases: +.. _actions-failure-use-cases: -Use Cases That Exceed Github Actions Memory Limit -------------------------------------------------- +Use Cases That Cannot be Run in GitHub Actions +---------------------------------------------- -Below is a list of use cases in the repository that cannot be run in Github -Actions due to their excessive memory usage. They have been tested and -cleared by reviewers of any other issues and can be used by METplus users in -the same manner as all other use cases. +Below is a list of use cases in the repository that cannot be run in GitHub +Actions due to either excessive memory or excessive disk space usage. They have +been tested and cleared by reviewers of any other issues and can be used by METplus +users in the same manner as all other use cases. +Use Cases that Exceed GitHub Actions Memory Limit +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - *model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst* +Use Cases that Exceed GitHub Actions Disk Space +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +- *model_applications/s2s/UserScript_fcstGFS_obsERA_StratospherePolar* + .. _use_case_documentation: Document New Use Case @@ -790,9 +797,11 @@ environment, potential reasons include: - Referencing variables set in the user's configuration file or local environment - Memory usage of the use case exceeds the available memory in the - Github Actions environment + GitHub Actions environment +- Disk space usage of the use casee exceeds the available space in the + GitHub Actions environment -Github Actions has +GitHub Actions has `limited memory `_ available and will cause the use case to fail when exceeded. A failure caused by exceeding the memory allocation in a Python Embedding script @@ -802,9 +811,15 @@ memory profiler to check the Python script's memory usage. If the use case exceeds the limit, try to pare down the data held in memory and use less memory intensive Python routines. +Additionally, GitHub Actions has limited disk space available. The use case will +fail if the data files exceed the available disk space. If this is the case, consider +removing any unneeded variables from the data files, reducing the time steps run, or +creating a new use case category to keep file sizes down for each group. + If memory mitigation cannot move the use case’s memory usage below the -Github Actions limit, -see :ref:`exceeded-Github-Actions` for next steps. +GitHub Actions limit or data cannot be reduced further to fit inside +the available disk space +see :ref:`exceeded-GitHub-Actions` for next steps. Verify that the use case ran in a reasonable amount of time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -859,19 +874,22 @@ processed in the automated tests. In **ci_overrides.conf**, set:: OBS_VAR1_THRESH = gt0, lt10.0 -.. _exceeded-Github-Actions: +.. _exceeded-GitHub-Actions: -Use Cases That Exceed Memory Allocations of Github Actions ----------------------------------------------------------- +Use Cases That Cannot be Run in GitHub Actions +---------------------------------------------- If a use case utilizing Python embedding does not run successfully in -Github Actions due to exceeding the memory limit and memory mitigation -steps were unsuccessful in lowering memory usage, please take the following steps. +GitHub Actions due to exceeding the memory limit and memory mitigation +steps were unsuccessful in lowering memory usage, or if a use case does +run successfully in GitHub Actions due to exceeding available disk space +and the data cannot be further pared down, please take the following steps. -- Document the Github Actions failure in the Github use case issue. +- Document the GitHub Actions failure in the GitHub use case issue. Utilize a Python memory profiler to identify as specifically as possible - where the script exceeds the memory limit. -- Add the use case to the :ref:`memory-intense-use-cases` list. + where the script exceeds the memory limit (if the failure is due to exceeding + the memory limit). +- Add the use case to the :ref:`actions-failure-use-cases` list. - In the *internal/tests/use_cases/all_use_cases.txt* file, ensure that the use case is listed as the lowest-listed use case in its respective category. Change the number in front of the new use case to an 'X', preceded @@ -882,13 +900,13 @@ steps were unsuccessful in lowering memory usage, please take the following step - In the *.github/parm/use_case_groups.json* file, remove the entry that was added during the :ref:`add_new_category_to_test_runs` for the new use case. This will stop the use case from running on a pull request. -- Push these two updated files to the working branch in Github and +- Push these two updated files to the working branch in GitHub and confirm that it now compiles successfully. - During the :ref:`create-a-pull-request` creation, inform the reviewer of - the Github Actions failure. The reviewer should confirm the use case is + the GitHub Actions failure. The reviewer should confirm the use case is successful when run manually, that the memory profiler output confirms that - the Python embedding script exceeds the Github Actions limit, and that - there are no other Github Actions compiling errors. + the Python embedding script exceeds the GitHub Actions limit, and that + there are no other GitHub Actions compiling errors. .. _create-a-pull-request: diff --git a/docs/_static/s2s-zonal_means.png b/docs/_static/s2s-zonal_means.png deleted file mode 100755 index 3e5ab96dc4..0000000000 Binary files a/docs/_static/s2s-zonal_means.png and /dev/null differ diff --git a/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratosphereBias.png b/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratosphereBias.png new file mode 100644 index 0000000000..9125074d99 Binary files /dev/null and b/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratosphereBias.png differ diff --git a/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratospherePolar.png b/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratospherePolar.png new file mode 100644 index 0000000000..253480cd5a Binary files /dev/null and b/docs/_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratospherePolar.png differ diff --git a/docs/use_cases/model_applications/s2s_stratosphere/README.rst b/docs/use_cases/model_applications/s2s_stratosphere/README.rst new file mode 100644 index 0000000000..85138329be --- /dev/null +++ b/docs/use_cases/model_applications/s2s_stratosphere/README.rst @@ -0,0 +1,3 @@ +Subseasonal to Seasonal: Stratosphere +------------------------------------- +Subseasonal-to-Seasonal model configurations for Stratosphere evaluation diff --git a/docs/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.py b/docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.py similarity index 56% rename from docs/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.py rename to docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.py index 9f3fda7684..29d9f26d7c 100644 --- a/docs/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.py +++ b/docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.py @@ -1,10 +1,10 @@ """ -UserScript: Make zonal and meridonial means -======================================================================== +Bias Plot on Zonal Mean Wind and Temperature: UserScript, Series-Analysis +========================================================================== model_applications/ -s2s/ -UserScript_obsERA_obsOnly_Stratosphere.py +s2s_stratosphere/ +UserScript_fcstGFS_obsERA_StratosphereBias.py """ @@ -13,21 +13,25 @@ # -------------------- # # This use case calls functions in METcalcpy to create zonal and meridonial -# means +# means on U and T. It then runs Series-Analysis on the output zonal means +# and creates a contour plot of bias in latitude and pressure level. # ############################################################################## # Datasets # -------- # -# SSWC_v1.0_varFull_ERAi_d20130106_s20121107_e20130307_c20160701.nc +# GFS 24 hour forecasts: GFS_2018_02_24h.nc +# ERA: ERA_2018_02.nc +# ############################################################################## # METplus Components # ------------------ # # This use case runs the UserScript wrapper tool to run a user provided script, -# in this case, meridonial.py. +# in this case, zonal_mean_driver.py, runs Series-Analysis to compute the bias, +# and then runs another UserScript, bias_plot_driver.py, to create the bias plots. # ############################################################################## @@ -36,9 +40,9 @@ # # This use case does not loop but plots the entire time period of data # -# UserScript -# This uses data from 20130106,20121107,20130307,20160701 -# +# UserScript: Computes zonal and meridional means +# Series-Analysis: Computes the bias on zonal mean wind and temperature +# UserScript: Creates bias plots # ############################################################################## @@ -47,24 +51,40 @@ # # METplus first loads all of the configuration files found in parm/metplus_config, # then it loads any configuration files passed to METplus via the command line -# with the -c option, i.e. -c parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf +# with the -c option, i.e. -c parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf # # .. highlight:: bash -# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf +# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf # ############################################################################# # MET Configuration # --------------------- # -# There are no MET tools used in this use case. +# METplus sets environment variables based on user settings in the METplus configuration file. +# See :ref:`How METplus controls MET config file settings` for more details. +# +# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!** +# +# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to: +# :ref:`Overriding Unsupported MET config file settings` +# +# **SeriesAnalysisConfig_wrapped** +# +# .. note:: See the :ref:`Series-Analysis MET Configuration` section of the User's Guide for more information on the environment variables used in the file below: +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/met_config/SeriesAnalysisConfig_wrapped # ############################################################################## # Python Embedding # ---------------- # -# There is no python embedding in this use case +# This use case uses a Python embedding script to read in the zonal mean data to Series-Analysis +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py # ############################################################################## @@ -73,14 +93,14 @@ # # This use case can be run two ways: # -# 1) Passing in meridonial_means.conf, +# 1) Passing in UserScript_fcstGFS_obsERA_StratosphereBias.conf, # then a user-specific system configuration file:: # -# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf -c /path/to/user_system.conf +# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf -c /path/to/user_system.conf # -# 2) Modifying the configurations in parm/metplus_config, then passing in meridonial.conf:: +# 2) Modifying the configurations in parm/metplus_config, then passing in UserScript_fcstGFS_obsERA_StratosphereBias.conf: # -# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf +# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf # # The former method is recommended. Whether you add them to a user-specific configuration file or modify the metplus_config files, the following variables must be set correctly: # @@ -89,9 +109,7 @@ # * **MET_INSTALL_DIR** - Path to location where MET is installed locally # # and for the [exe] section, you will need to define the location of NON-MET executables. -# If the executable is in the user's path, METplus will find it from the name. -# If the executable is not in the path, specify the full path to the executable here (i.e. RM = /bin/rm) -# The following executables are required for performing series analysis use cases: +# No executables are required for performing this use case. # # Example User Configuration File:: # @@ -100,13 +118,6 @@ # OUTPUT_BASE = /path/to/output/dir # MET_INSTALL_DIR = /path/to/met-X.Y # -# [exe] -# RM = /path/to/rm -# CUT = /path/to/cut -# TR = /path/to/tr -# NCAP2 = /path/to/ncap2 -# CONVERT = /path/to/convert -# NCDUMP = /path/to/ncdump # ############################################################################## @@ -126,10 +137,13 @@ # # * UserScriptUseCase # * S2SAppUseCase +# * S2SStratosphereAppUseCase +# * SeriesAnalysisUseCase # * METcalcpyUseCase +# * METplotpyUseCase # # Navigate to the :ref:`quick-search` page to discover other similar use cases. # # # -# sphinx_gallery_thumbnail_path = '_static/s2s-zonal_means.png' +# sphinx_gallery_thumbnail_path = '_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratosphereBias.png' diff --git a/docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.py b/docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.py new file mode 100644 index 0000000000..e4c8af1857 --- /dev/null +++ b/docs/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.py @@ -0,0 +1,151 @@ +""" +Bias Plot on Polar Cap Temperature and Polar Vortex U: UserScript, Stat-Analysis +================================================================================ + +model_applications/ +s2s_stratosphere/ +UserScript_fcstGFS_obsERA_StratospherePolar.py + +""" + +############################################################################## +# Scientific Objective +# -------------------- +# +# This use case calls functions in METcalcpy to create polar cap temperature +# and polar vortex wind. It then runs Stat-Analysis on the output zonal means +# and creates a contour plot of bias in lead time and pressure level. +# + +############################################################################## +# Datasets +# -------- +# +# * Forecast dataset: GFS Forecast U and T at multiple pressure levels +# * Observation dataset: ERA Reanlaysis U and T at multiple pressure levels +# +# Data for this use case is not contained in the sample data tar files due to +# its size. Rather, it is stored as additional data in a separate tar file. +# + +############################################################################## +# METplus Components +# ------------------ +# +# This use case runs the UserScript wrapper tool to run a user provided script, +# in this case, polar_t_u_driver.py which output data into MET's matched pair format. +# It then runs Stat-Analysis to compute the bias and RMSE, and another UserScript, +# bias_rmse_plot_driver.py, to create the plots. +# + +############################################################################## +# METplus Workflow +# ---------------- +# +# This use case loops over lead times for the first UserScript and Stat-Analysis, +# and the plotting proceeds over the entire time period +# +# UserScript: Computes polar cap temperature and polar vortex U +# Stat-Analysis: Computes ME and RMSE on polar cap temperature and polar vortex U +# UserScript: Creates ME and RMSE plots +# + +############################################################################## +# METplus Configuration +# --------------------- +# +# METplus first loads all of the configuration files found in parm/metplus_config, +# then it loads any configuration files passed to METplus via the command line +# with the -c option, i.e. -c parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf +# + +############################################################################# +# MET Configuration +# --------------------- +# +# METplus sets environment variables based on user settings in the METplus configuration file. +# See :ref:`How METplus controls MET config file settings` for more details. +# +# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!** +# +# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to: +# :ref:`Overriding Unsupported MET config file settings` +# +# **STATAnalysisConfig_wrapped** +# +# .. note:: See the :ref:`Series-Analysis MET Configuration` section of the User's Guide for more information on the environment variables used in the file below: +# +# .. highlight:: bash +# .. literalinclude:: ../../../../parm/met_config/STATAnalysisConfig_wrapped +# + +############################################################################## +# Python Embedding +# ---------------- +# +# This use case does not use python embedding +# + +############################################################################## +# Running METplus +# --------------- +# +# This use case can be run two ways: +# +# 1) Passing in UserScript_fcstGFS_obsERA_StratospherePolar.conf, +# then a user-specific system configuration file:: +# +# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf -c /path/to/user_system.conf +# +# 2) Modifying the configurations in parm/metplus_config, then passing in UserScript_fcstGFS_obsERA_StratospherePolar.conf: +# +# run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf +# +# The former method is recommended. Whether you add them to a user-specific configuration file or modify the metplus_config files, the following variables must be set correctly: +# +# * **INPUT_BASE** - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs). This is not required to run METplus, but it is required to run the examples in parm/use_cases +# * **OUTPUT_BASE** - Path where METplus output will be written. This must be in a location where you have write permissions +# * **MET_INSTALL_DIR** - Path to location where MET is installed locally +# +# and for the [exe] section, you will need to define the location of NON-MET executables. +# No executables are required for performing this use case. +# +# Example User Configuration File:: +# +# [dir] +# INPUT_BASE = /path/to/sample/input/data +# OUTPUT_BASE = /path/to/output/dir +# MET_INSTALL_DIR = /path/to/met-X.Y +# +# + +############################################################################## +# Expected Output +# --------------- +# +# A successful run will output the following both to the screen and to the logfile:: +# +# INFO: METplus has successfully finished running. +# + +############################################################################## +# Keywords +# -------- +# +# .. note:: +# +# * UserScriptUseCase +# * S2SAppUseCase +# * S2SStratosphereAppUseCase +# * StatAnalysisUseCase +# * METcalcpyUseCase +# * METplotpyUseCase +# +# Navigate to the :ref:`quick-search` page to discover other similar use cases. +# +# +# +# sphinx_gallery_thumbnail_path = '_static/s2s_stratosphere-UserScript_fcstGFS_obsERA_StratospherePolar.png' diff --git a/internal/tests/use_cases/all_use_cases.txt b/internal/tests/use_cases/all_use_cases.txt index 658f73d3dd..8aed023e48 100644 --- a/internal/tests/use_cases/all_use_cases.txt +++ b/internal/tests/use_cases/all_use_cases.txt @@ -105,6 +105,7 @@ Category: marine_and_cryosphere 9::PointStat_fcstGFS_obsASCAT_satelliteWinds::model_applications/marine_and_cryosphere/PointStat_fcstGFS_obsASCAT_satelliteWinds.conf:: icecover_env, py_embed 10::PointStat_fcstGFS_obsJASON3_satelliteAltimetry::model_applications/marine_and_cryosphere/PointStat_fcstGFS_obsJASON3_satelliteAltimetry.conf:: py_embed_base_env, py_embed + Category: medium_range 0::PointStat_fcstGFS_obsNAM_Sfc_MultiField_PrepBufr:: model_applications/medium_range/PointStat_fcstGFS_obsNAM_Sfc_MultiField_PrepBufr.conf 1::TCStat_SeriesAnalysis_fcstGFS_obsGFS_FeatureRelative_SeriesByInit:: model_applications/medium_range/TCStat_SeriesAnalysis_fcstGFS_obsGFS_FeatureRelative_SeriesByInit.conf:: netcdf4_env @@ -140,9 +141,8 @@ Category: s2s 1::TCGen_fcstGFSO_obsBDECKS_GDF_TDF:: model_applications/s2s/TCGen_fcstGFSO_obsBDECKS_GDF_TDF.conf:: metplotpy_env,cartopy,metplus 2::UserScript_obsPrecip_obsOnly_Hovmoeller:: model_applications/s2s/UserScript_obsPrecip_obsOnly_Hovmoeller.conf:: metplotpy_env,cartopy 3:: UserScript_fcstS2S_obsERAI_CrossSpectra:: model_applications/s2s/UserScript_fcstS2S_obsERAI_CrossSpectra.conf:: spacetime_env -4:: UserScript_obsERA_obsOnly_Stratosphere:: model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf:: metplotpy_env,metdataio -5::SeriesAnalysis_fcstCFSv2_obsGHCNCAMS_climoStandardized_MultiStatisticTool:: model_applications/s2s/SeriesAnalysis_fcstCFSv2_obsGHCNCAMS_climoStandardized_MultiStatisticTool.conf:: netcdf4_env -6::GridStat_fcstCFSv2_obsGHCNCAMS_MultiTercile:: model_applications/s2s/GridStat_fcstCFSv2_obsGHCNCAMS_MultiTercile.conf:: netcdf4_env +4::SeriesAnalysis_fcstCFSv2_obsGHCNCAMS_climoStandardized_MultiStatisticTool:: model_applications/s2s/SeriesAnalysis_fcstCFSv2_obsGHCNCAMS_climoStandardized_MultiStatisticTool.conf:: netcdf4_env +5::GridStat_fcstCFSv2_obsGHCNCAMS_MultiTercile:: model_applications/s2s/GridStat_fcstCFSv2_obsGHCNCAMS_MultiTercile.conf:: netcdf4_env Category: s2s_mid_lat @@ -160,6 +160,11 @@ Category: s2s_mjo 4:: UserScript_obsCFSR_obsOnly_MJO_ENSO:: model_applications/s2s_mjo/UserScript_obsCFSR_obsOnly_MJO_ENSO.conf:: spacetime_env, metdataio +Category: s2s_stratosphere +0:: UserScript_fcstGFS_obsERA_StratosphereBias:: model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf:: weatherregime_env,cartopy,metdataio +#X::UserScript_fcstGFS_obsERA_StratospherePolar:: model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf:: weatherregime_env,cartopy,metdataio + + Category: short_range 0::EnsembleStat_fcstHRRRE_obsHRRRE_Sfc_MultiField::model_applications/short_range/EnsembleStat_fcstHRRRE_obsHRRRE_Sfc_MultiField.conf 1::MODE_fcstHRRR_obsMRMS_Hail_GRIB2::model_applications/short_range/MODE_fcstHRRR_obsMRMS_Hail_GRIB2.conf diff --git a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf b/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf deleted file mode 100644 index 0d81594060..0000000000 --- a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.conf +++ /dev/null @@ -1,51 +0,0 @@ -[config] - -# Documentation for this use case can be found at -# https://metplus.readthedocs.io/en/latest/generated/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere.html - -# For additional information, please see the METplus Users Guide. -# https://metplus.readthedocs.io/en/latest/Users_Guide - -### -# Processes to run -# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list -### - -PROCESS_LIST = UserScript - - -### -# Time Info -# LOOP_BY options are INIT, VALID, RETRO, and REALTIME -# If set to INIT or RETRO: -# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set -# If set to VALID or REALTIME: -# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set -# LEAD_SEQ is the list of forecast leads to process -# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control -### - -LOOP_BY = REALTIME - -VALID_TIME_FMT = %Y -VALID_BEG = 2013 - -USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE - - -### -# UserScript Settings -# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#userscript -### - -USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.py - -[user_env_vars] -INPUT_FILE_NAME = {INPUT_BASE}/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/SSWC_v1.0_varFull_ERAi_d20130106_s20121107_e20130307_c20160701.nc -YAML_CONFIG_NAME = {METPLUS_BASE}/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.yaml - -LOG_FILE = "Meridonial_means.log" - -LOG_LEVEL = "INFO" - -OUTPUT_DIR = {OUTPUT_BASE} diff --git a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.py b/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.py deleted file mode 100755 index 777be36ea6..0000000000 --- a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.py +++ /dev/null @@ -1,82 +0,0 @@ -#!/usr/bin/env python3 - -""" -Create meridonial mean statistics - -""" -import os -import sys -import logging -import yaml -import xarray as xr # http://xarray.pydata.org/ -import metcalcpy.util.read_env_vars_in_config as readconfig -import metcalcpy.pre_processing.directional_means as directional_means -import METreadnc.util.read_netcdf as read_netcdf - - -def main(): - """ - Use existing default meridonial mean config file found in METcalcpy to - grab the test file - """ - - - """ - Read Meridial Mean YAML configuration file - user can use their own, if none specified at the command line, - use the "default" example YAML config file, spectra_plot_coh2.py - Using a custom YAML reader so we can use environment variables - """ - - try: - input_config_file = os.getenv("YAML_CONFIG_NAME","meridonial_mean.yaml") - config = readconfig.parse_config(input_config_file) - logging.info(config) - except yaml.YAMLError as exc: - logging.error(exc) - - """ - Read METplus config file paramaters - """ - #input_file_name = os.environ.get("INPUT_FILE_NAME","SSWC_v1.0_varFull_ERAi_d20130106_s20121107_e20130307_c20160701.nc") - input_file = config["input_filename"] - - """ - Setup logging - """ - logfile = "meridonial_mean.log" - logging_level = os.environ.get("LOG_LEVEL","logging.INFO") - logging.basicConfig(stream=logfile, level=logging_level) - - """ - Read dataset - """ - try: - logging.info('Opening ' + input_file[0]) - file_reader = read_netcdf.ReadNetCDF() - - #file_reader returns a list of xarrays even if there is only one file requested to be read - #so we change it from a list to a single - ds = file_reader.read_into_xarray(input_file)[0] - except IOError as exc: - logging.error('Unable to open ' + input_file) - logging.error(exc) - sys.exit(1) - logging.debug(ds) - ds = ds[['uwndFull_TS','vwndFull_TS','tempFull_TS','geopFull_TS']] - ds = ds.rename({'timeEv60':'time', - 'lat':'latitude', # pyzome currently expects dimensions named latitude and longitude - 'lon':'longitude', - 'uwndFull_TS':'u', - 'vwndFull_TS':'v', - 'tempFull_TS':'T', - 'geopFull_TS':'Z'}) - - uzm = directional_means.zonal_mean(ds.u) - Tzm = directional_means.zonal_mean(ds.T) - T_6090 = directional_means.meridional_mean(Tzm, 60, 90) - - print(T_6090) - -if __name__ == '__main__': - main() diff --git a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.yaml b/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.yaml deleted file mode 100644 index 6cadc5160b..0000000000 --- a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/meridonial_mean.yaml +++ /dev/null @@ -1,2 +0,0 @@ -input_filename: -- !ENV '${INPUT_FILE_NAME}' diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf new file mode 100644 index 0000000000..f2182e5c24 --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.conf @@ -0,0 +1,148 @@ +[config] + +# Documentation for this use case can be found at +# https://metplus.readthedocs.io/en/latest/generated/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias.html + +# For additional information, please see the METplus Users Guide. +# https://metplus.readthedocs.io/en/latest/Users_Guide + +### +# Processes to run +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list +### +PROCESS_LIST = UserScript(obs_means), UserScript(fcst_means), SeriesAnalysis(sa_ut), UserScript(bias_plot) + + +### +# Time Info +# LOOP_BY options are INIT, VALID, RETRO, and REALTIME +# If set to INIT or RETRO: +# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set +# If set to VALID or REALTIME: +# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set +# LEAD_SEQ is the list of forecast leads to process +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control +### + +LOOP_BY = VALID + +VALID_TIME_FMT = %Y%m%d +VALID_BEG = 20180201 +VALID_END = 20180228 +VALID_INCREMENT = 30d +LEAD_SEQ = 24 + +USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE + +LOOP_ORDER = processes + + +### +# User Environment Variables +### +[user_env_vars] +OBS_INPUT_FILE_NAME = {INPUT_BASE}/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/ERA_2018_02.nc +FCST_INPUT_FILE_NAME = {INPUT_BASE}/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/GFS_2018_02_024h.nc + +OUTPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias + +PLOT_U_INPUT_FILE = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/SeriesAnalysis/zonal_mean_U_stats_2018_02.nc +PLOT_U_BIAS_VAR = series_cnt_ME +PLOT_U_OBAR_VAR = series_cnt_OBAR +PLOT_U_LEVELS = 0,10,20,30,40,50,60,70,80,90,100 +PLOT_U_TITLE = GFS vs ERA 24h Forecast Zonal Mean Wind Bias (ME) 02/2018 +PLOT_U_OUTPUT_DIR = {OUTPUT_DIR}/plots +PLOT_U_OUTPUT_FILE = GFS_ERA_ME_2018_02_zonal_mean_U.png + +PLOT_T_INPUT_FILE = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/SeriesAnalysis/zonal_mean_T_stats_2018_02.nc +PLOT_T_BIAS_VAR = series_cnt_ME +PLOT_T_OBAR_VAR = series_cnt_OBAR +PLOT_T_LEVELS = 200,210,220,230,240,250,260,270,280,290,300 +PLOT_T_TITLE = GFS vs ERA 24h Forecast Zonal Mean Temperature Bias (ME) 02/2018 +PLOT_T_OUTPUT_DIR = {OUTPUT_DIR}/plots +PLOT_T_OUTPUT_FILE = GFS_ERA_ME_2018_02_zonal_mean_T.png + + +### +# Zonal Mean UserScript Settings +# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#userscript +### +[obs_means] +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/zonal_mean_driver.py obs time + + +[fcst_means] +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/zonal_mean_driver.py fcst time + + +### +# Series Analysis Settings +### +[sa_ut] +SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE + +FCST_SERIES_ANALYSIS_INPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/FCST +FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = FCST_zonal_mean_U_T_{valid?fmt=%Y%m}*_{valid?fmt=%H%M%S}.nc +FCST_SERIES_ANALYSIS_INPUT_DATATYPE = PYTHON_NUMPY + +OBS_SERIES_ANALYSIS_INPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/OBS +OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = OBS_zonal_mean_U_T_{valid?fmt=%Y%m}*_{valid?fmt=%H%M%S}.nc +OBS_SERIES_ANALYSIS_INPUT_DATATYPE = PYTHON_NUMPY + +SERIES_ANALYSIS_CLIMO_MEAN_INPUT_DIR = +SERIES_ANALYSIS_CLIMO_MEAN_INPUT_TEMPLATE = + +SERIES_ANALYSIS_CLIMO_STDEV_INPUT_DIR = +SERIES_ANALYSIS_CLIMO_STDEV_INPUT_TEMPLATE = + +SERIES_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/SeriesAnalysis + +SERIES_ANALYSIS_DESC = + +SERIES_ANALYSIS_CAT_THRESH = + +SERIES_ANALYSIS_VLD_THRESH = + +SERIES_ANALYSIS_BLOCK_SIZE = + +SERIES_ANALYSIS_CTS_LIST = + +SERIES_ANALYSIS_REGRID_TO_GRID = +SERIES_ANALYSIS_REGRID_METHOD = +SERIES_ANALYSIS_REGRID_WIDTH = +SERIES_ANALYSIS_REGRID_VLD_THRESH = +SERIES_ANALYSIS_REGRID_SHAPE = + +SERIES_ANALYSIS_RUN_ONCE_PER_STORM_ID = False + +SERIES_ANALYSIS_IS_PAIRED = False + +SERIES_ANALYSIS_CONFIG_FILE = {PARM_BASE}/met_config/SeriesAnalysisConfig_wrapped + +SERIES_ANALYSIS_OUTPUT_STATS_CNT = TOTAL, ME, RMSE, FBAR, OBAR + +MODEL = GFS + +OBTYPE = ERA + +SERIES_ANALYSIS_OUTPUT_TEMPLATE = zonal_mean_{fcst_level}_stats_2018_02.nc + +FCST_SERIES_ANALYSIS_VAR1_NAME = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py MET_PYTHON_INPUT_ARG u lat +# This level is used as a label only +FCST_SERIES_ANALYSIS_VAR1_LEVELS = U + +OBS_SERIES_ANALYSIS_VAR1_NAME = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py MET_PYTHON_INPUT_ARG u lat + + +FCST_SERIES_ANALYSIS_VAR2_NAME = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py MET_PYTHON_INPUT_ARG T lat +FCST_SERIES_ANALYSIS_VAR2_LEVELS = T + +OBS_SERIES_ANALYSIS_VAR2_NAME = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py MET_PYTHON_INPUT_ARG T lat + + +### +# UserScript Bias Plot Settings +# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#userscript +### +[bias_plot] +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/bias_plot_driver.py diff --git a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/README b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/README similarity index 70% rename from parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/README rename to parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/README index 8111901811..de1e1d0abe 100644 --- a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_Stratosphere/README +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/README @@ -1,8 +1,8 @@ These files are a use case to show how to compute meridial and zonal means -You need METcalcpy and METdataio in your python path or your conda environment +You need METcalcpy and METdatadb in your python path or your conda environment i.e. -export PYTHONPATH=/METdataio:/METcalcpy +export PYTHONPATH=/METdatadb:/METcalcpy The file SSWC_v1.0_varFull_ERAi_d20130106_s20121107_e20130307_c20160701.nc needs to be on disk somewhere on your computer and referenced correctly in the file diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/bias_plot_driver.py b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/bias_plot_driver.py new file mode 100755 index 0000000000..dd28c190f3 --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/bias_plot_driver.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 + +""" +Creates a bias plot +""" +import os +import METreadnc.util.read_netcdf as read_netcdf +from metplotpy.contributed.stratosphere_diagnostics.stratosphere_plots import plot_zonal_bias + + +def main(): + """ + Read METplus environment variables + """ + print('Reading Input Environment Variables') + infile_u = [os.environ['PLOT_U_INPUT_FILE']] + invar_u = os.environ['PLOT_U_BIAS_VAR'] + omvar_u = os.environ['PLOT_U_OBAR_VAR'] + plot_levels_u_str = os.environ['PLOT_U_LEVELS'].split(',') + plot_levels_u = [int(pp) for pp in plot_levels_u_str] + plot_title_u = os.environ['PLOT_U_TITLE'] + output_dir_u = os.environ['PLOT_U_OUTPUT_DIR'] + output_file_u = os.environ.get('PLOT_U_OUTPUT_FILE','bias_plot.png') + plot_output_file_u = os.path.join(output_dir_u,output_file_u) + + infile_t = [os.environ['PLOT_T_INPUT_FILE']] + invar_t = os.environ['PLOT_T_BIAS_VAR'] + omvar_t = os.environ['PLOT_T_OBAR_VAR'] + plot_levels_t_str = os.environ['PLOT_T_LEVELS'].split(',') + plot_levels_t = [int(pp) for pp in plot_levels_t_str] + plot_title_t = os.environ['PLOT_T_TITLE'] + output_dir_t = os.environ['PLOT_T_OUTPUT_DIR'] + output_file_t = os.environ.get('PLOT_T_OUTPUT_FILE','bias_plot.png') + plot_output_file_t = os.path.join(output_dir_t,output_file_t) + + """ + Make plot output directory if it doesn't exist + """ + if not os.path.exists(output_dir_u): + os.makedirs(output_dir_u) + + if not os.path.exists(output_dir_t): + os.makedirs(output_dir_t) + + """ + Read dataset + """ + print('Reading Zonal Mean U Bias File: '+infile_u[0]) + file_reader_u = read_netcdf.ReadNetCDF() + dsu = file_reader_u.read_into_xarray(infile_u)[0] + bias_u = dsu[invar_u].values + lats_u = dsu['lat'].values + obar_u = dsu[omvar_u].values + levels_u = dsu['level'].values + + print('Reading Zonal Mean T Bias File: '+infile_t[0]) + file_reader_t = read_netcdf.ReadNetCDF() + dst = file_reader_t.read_into_xarray(infile_t)[0] + bias_t = dst[invar_t].values + lats_t = dst['lat'].values + obar_t = dst[omvar_t].values + levels_t = dst['level'].values + + """ + Create Bias Plots + """ + print('Plotting Zonal Mean U Bias') + plot_zonal_bias(lats_u,levels_u,bias_u,obar_u,plot_output_file_u,plot_title_u,plot_levels_u) + print('Plotting Zonal Mean T Bias') + plot_zonal_bias(lats_t,levels_t,bias_t,obar_t,plot_output_file_t,plot_title_t,plot_levels_t) + + +if __name__ == '__main__': + main() diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py new file mode 100644 index 0000000000..b3b4bc1aeb --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/read_met_axis_mean.py @@ -0,0 +1,60 @@ +import os +import sys +import numpy as np +import datetime as dt +import netCDF4 as nc4 + + +if len(sys.argv) != 4: + print("Must specify exactly one input file, variable name, and summary axis (lat, lon, latlon).") + sys.exit(1) + +# Read the input file as the first argument +input_file = os.path.expandvars(sys.argv[1]) +var_name = sys.argv[2] +axis = sys.argv[3] + +# Read the data +f = nc4.Dataset(input_file, 'r') +data = np.float64(f.variables[var_name][:]) +met_data = np.transpose(data).copy() + +if axis == "lon": + lats = list() + lons = list(np.float64(f.variables["lon"][:])) +elif axis == "lat": + lats = list(np.float64(f.variables["lat"][:])) + lons = list() + +pres = list(list(np.float64(f.variables["pres"][:]))) +times = list() + +lead_ma = f.variables["lead_time"][:] +lead = lead_ma.__int__() +vtime = f.variables["time"] +cur_date = nc4.num2date(vtime[:], vtime.units, vtime.calendar) +init = cur_date - dt.timedelta(hours=lead) +accum = "00" + +attrs = { + 'valid': cur_date.strftime("%Y%m%d_%H%M%S"), + 'init': init.strftime("%Y%m%d_%H%M%S"), + 'lead': str(int(lead)).zfill(2), + 'accum': accum, + + 'name': var_name, + 'long_name': str(getattr(f.variables[var_name], "long_name")), + 'level': axis + "_mean", + 'units': str(getattr(f.variables[var_name], "units")), + + 'grid': { + 'type' : "SemiLatLon", + 'name' : axis + "_mean", + 'lats' : lats, + 'lons' : lons, + 'levels' : pres, + 'times' : times + } +} + +print("Attributes: " + repr(attrs)) diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/zonal_mean_driver.py b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/zonal_mean_driver.py new file mode 100755 index 0000000000..170bab1aba --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratosphereBias/zonal_mean_driver.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 + +""" +Create meridonial mean statistics + +""" +import os +import sys +import logging +import yaml +import xarray as xr # http://xarray.pydata.org/ +import metcalcpy.util.read_env_vars_in_config as readconfig +import metcalcpy.pre_processing.directional_means as directional_means +import METreadnc.util.read_netcdf as read_netcdf + + +def main(): + """ + Read arguments + """ + inlabel = sys.argv[1].upper() + timevar = sys.argv[2] + + """ + Read METplus environment variables + """ + print('Reading Input Environment Variables') + input_file = [os.environ[inlabel+'_INPUT_FILE_NAME']] + output_dir = os.environ['OUTPUT_DIR'] + full_output_dir = os.path.join(output_dir,inlabel) + + """ + Setup logging + """ + #logfile = "meridonial_mean.log" + #logging_level = os.environ.get("LOG_LEVEL","logging.INFO") + #logging.basicConfig(stream=logfile, level=logging_level) + + """ + Read dataset + """ + print('Reading input data') + file_reader = read_netcdf.ReadNetCDF() + ds = file_reader.read_into_xarray(input_file)[0] + ds = ds.rename({'lat':'latitude', + 'lon':'longitude'}) + + print('Computing Zonal means') + uzm = directional_means.zonal_mean(ds.u) + uzm = uzm.assign_coords(lat=("latitude",ds.latitude.values[:,0])) + Tzm = directional_means.zonal_mean(ds.T) + Tzm = Tzm.assign_coords(lat=("latitude",ds.latitude.values[:,0])) + + """ + Write output files if desired, by first creating a directory + """ + print('Writing output zonal mean files') + if not os.path.exists(full_output_dir): + os.makedirs(full_output_dir) + + datetimeindex = ds.indexes[timevar].to_datetimeindex() + out_ds = uzm.to_dataset() + out_ds.u.attrs = ds.u.attrs + out_ds['T'] = Tzm + out_ds.T.attrs = ds.T.attrs + for i in range(len(datetimeindex)): + cur_date = datetimeindex[i] + output_file = os.path.join(full_output_dir,inlabel+'_zonal_mean_U_T_'+cur_date.strftime('%Y%m%d_%H%M%S')+'.nc') + out_write = out_ds.isel(time=i) + # Add lead time as a variable + if inlabel == 'OBS': + out_write = out_write.assign(lead_time=[0.0]) + elif inlabel == 'FCST': + #Grab Forecast Lead + out_write = out_write.assign(lead_time=ds.lead_time[i]) + out_write.to_netcdf(output_file, 'w') + + + +if __name__ == '__main__': + main() diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf new file mode 100644 index 0000000000..c3cbbfc4ce --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.conf @@ -0,0 +1,130 @@ +[config] + +# Documentation for this use case can be found at +# https://metplus.readthedocs.io/en/latest/generated/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar.html + +# For additional information, please see the METplus Users Guide. +# https://metplus.readthedocs.io/en/latest/Users_Guide + +### +# Processes to run +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list +### + +PROCESS_LIST = UserScript(means), StatAnalysis(sanal_cnt), UserScript(plots_t), UserScript(plots_u) + + +SCRUB_STAGING_DIR = False + +### +# Time Info +# LOOP_BY options are INIT, VALID, RETRO, and REALTIME +# If set to INIT or RETRO: +# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set +# If set to VALID or REALTIME: +# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set +# LEAD_SEQ is the list of forecast leads to process +# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control +### + +LOOP_BY = VALID + +VALID_TIME_FMT = %Y%m%d +VALID_BEG = 20180201 +VALID_END = 20180228 +VALID_INCREMENT = 30d +LEAD_SEQ = begin_end_incr(0,240,3),begin_end_incr(252,384,12) + +LOOP_ORDER = processes + + +### +# UserScript Settings +# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#userscript +### +[means] +USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE_PER_LEAD + +# Template of filenames to input to the user-script +USER_SCRIPT_INPUT_TEMPLATE = {INPUT_BASE}/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/ERA/ERA_{valid?fmt=%Y}_{valid?fmt=%m}.nc, {INPUT_BASE}/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/GFS/GFS_{valid?fmt=%Y}_{valid?fmt=%m}_{lead?fmt=%HHH}h.nc + +# Name of the file containing the listing of input files +# The options are OBS_INPUT for observations or FCST_INPUT for forecast +# Or, set OBS_INPUT, FCST_INPUT if doing both and make sure the USER_SCRIPT_INPUT_TEMPLATE is ordered: +# observation_template, forecast_template +USER_SCRIPT_INPUT_TEMPLATE_LABELS = OBS_INPUT, FCST_INPUT + +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py time {lead?fmt=%HHH} + + + +[user_env_vars] +MODEL_NAME = GFS + +OUTPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar + +PLOT_OUTPUT_DIR = {OUTPUT_DIR}/plots + +PLOT_INPUT_FILE = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/SeriesAnalysis/zonal_mean_stats_2018_02.nc +PLOT_T_BIAS_LEVELS = -6,-5,-4,-3,-2,-1,0,1,2,3,4,5,6 +PLOT_T_BIAS_TITLE = GFS vs ERA Polar Cap Temperature Bias (ME) 02/2018 +PLOT_T_RMSE_LEVELS = 0,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5,5.5,6 +PLOT_T_RMSE_TITLE = GFS vs ERA Polar Cap Temperature RMSE 02/2018 +PLOT_T_BIAS_OUTPUT_FILE = ME_2018_02_polar_cap_T.png +PLOT_T_RMSE_OUTPUT_FILE = RMSE_2018_02_polar_cap_T.png + +PLOT_U_BIAS_LEVELS = -6,-5,-4,-3,-2,-1,0,1,2,3,4,5,6 +PLOT_U_BIAS_TITLE = GFS vs ERA Polar Vortex U Bias (ME) 02/2018 +PLOT_U_RMSE_LEVELS = 0,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5 +PLOT_U_RMSE_TITLE = GFS vs ERA Polar Vortex U RMSE 02/2018 +PLOT_U_BIAS_OUTPUT_FILE = ME_2018_02_polar_vortex_U.png +PLOT_U_RMSE_OUTPUT_FILE = RMSE_2018_02_polar_vortex_U.png + +[sanal_cnt] +MODEL1 = GFS +MODEL1_OBTYPE = ADPUPA + +STAT_ANALYSIS_CONFIG_FILE = {PARM_BASE}/met_config/STATAnalysisConfig_wrapped + +STAT_ANALYSIS_JOB1 = -job aggregate_stat -line_type MPR -out_line_type CNT -fcst_var PolarCapT -by FCST_LEV,FCST_LEAD -out_stat [out_stat_file]_PolarCapT_CNT.stat +STAT_ANALYSIS_JOB2 = -job aggregate_stat -line_type MPR -out_line_type CNT -fcst_var PolarVortexU -by FCST_LEV,FCST_LEAD -out_stat [out_stat_file]_PolarVortexU_CNT.stat + +MODEL_LIST = {MODEL1} +FCST_LEAD_LIST = 00, 03, 06, 09, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57, 60, 63, 66, 69, 72, 75, 78, 81, 84, 87, 90, 93, 96, 99, 102, 105, 108, 111, 114, 117, 120, 123, 126, 129, 132, 135, 138, 141, 144, 147, 150, 153, 156, 159, 162, 165, 168, 171, 174, 177, 180, 183, 186, 189, 192, 195, 198, 201, 204, 207, 210, 213, 216, 219, 222, 225, 228, 231, 234, 237, 240, 252, 264, 276, 288, 300, 312, 324, 336, 348, 360, 372, 384 + +GROUP_LIST_ITEMS = MODEL_LIST +LOOP_LIST_ITEMS = FCST_LEAD_LIST + +MODEL1_STAT_ANALYSIS_LOOKIN_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/mpr + +STAT_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/StatAnalysis + +MODEL1_STAT_ANALYSIS_OUT_STAT_TEMPLATE = {model?fmt=%s}_ERA_{obs_valid_beg?fmt=%Y%m%d}_{obs_valid_end?fmt=%Y%m%d}_{lead?fmt=%H%M%S}L + + +[plots_t] +USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE + +USER_SCRIPT_INPUT_TEMPLATE = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/StatAnalysis/GFS_ERA_{valid?fmt=%Y%m%d}_*_{lead?fmt=%H%M%S}L_PolarCapT_CNT.stat + +# Name of the file containing the listing of input files +# The options are OBS_INPUT for observations or FCST_INPUT for forecast +# Or, set OBS_INPUT, FCST_INPUT if doing both and make sure the USER_SCRIPT_INPUT_TEMPLATE is ordered: +# observation_template, forecast_template +USER_SCRIPT_INPUT_TEMPLATE_LABELS = STAT_INPUT + +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/bias_rmse_plot_driver.py T + + +[plots_u] +USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE + +USER_SCRIPT_INPUT_TEMPLATE = {OUTPUT_BASE}/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/StatAnalysis/GFS_ERA_{valid?fmt=%Y%m%d}_*_{lead?fmt=%H%M%S}L_PolarVortexU_CNT.stat + +# Name of the file containing the listing of input files +# The options are OBS_INPUT for observations or FCST_INPUT for forecast +# Or, set OBS_INPUT, FCST_INPUT if doing both and make sure the USER_SCRIPT_INPUT_TEMPLATE is ordered: +# observation_template, forecast_template +USER_SCRIPT_INPUT_TEMPLATE_LABELS = STAT_INPUT + +USER_SCRIPT_COMMAND = {PARM_BASE}/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/bias_rmse_plot_driver.py U diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/README b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/README new file mode 100644 index 0000000000..de1e1d0abe --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/README @@ -0,0 +1,9 @@ +These files are a use case to show how to compute meridial and zonal means + +You need METcalcpy and METdatadb in your python path or your conda environment +i.e. +export PYTHONPATH=/METdatadb:/METcalcpy + +The file SSWC_v1.0_varFull_ERAi_d20130106_s20121107_e20130307_c20160701.nc needs to be +on disk somewhere on your computer and referenced correctly in the file +meridial_mean.yaml diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/bias_rmse_plot_driver.py b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/bias_rmse_plot_driver.py new file mode 100755 index 0000000000..451a50aabc --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/bias_rmse_plot_driver.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 + +""" +Creates Polar Cap Bias and RMSE plots +""" +import os +import sys +import pandas as pd +import numpy as np +import METreadnc.util.read_netcdf as read_netcdf +from metplotpy.contributed.stratosphere_diagnostics.stratosphere_plots import plot_polar_bias,plot_polar_rmse + + +def main(): + plvar = sys.argv[1] + + """ + Read METplus environment variables + """ + plot_bias_levels_str = os.environ['PLOT_'+plvar+'_BIAS_LEVELS'].split(',') + plot_bias_levels = [float(pp) for pp in plot_bias_levels_str] + plot_bias_title = os.environ['PLOT_'+plvar+'_BIAS_TITLE'] + plot_rmse_levels_str = os.environ['PLOT_'+plvar+'_RMSE_LEVELS'].split(',') + plot_rmse_levels = [float(pp) for pp in plot_rmse_levels_str] + plot_rmse_title = os.environ['PLOT_'+plvar+'_RMSE_TITLE'] + output_dir = os.environ['PLOT_OUTPUT_DIR'] + bias_output_file = os.environ.get('PLOT_'+plvar+'_BIAS_OUTPUT_FILE','bias_plot.png') + rmse_output_file = os.environ.get('PLOT_'+plvar+'_RMSE_OUTPUT_FILE','rmse_plot.png') + plot_bias_output_file = os.path.join(output_dir,bias_output_file) + plot_rmse_output_file = os.path.join(output_dir,rmse_output_file) + + """ + Make plot output directory if it doesn't exist + """ + if not os.path.exists(output_dir): + os.makedirs(output_dir) + + """ + Read the list of files + """ + stat_filetxt = os.environ.get('METPLUS_FILELIST_STAT_INPUT','') + with open(stat_filetxt) as sf: + stat_infiles = sf.read().splitlines() + # Remove the first line if it's there + if (stat_infiles[0] == 'file_list'): + stat_infiles = stat_infiles[1:] + + """ + Read the first file and set up arrays + """ + fleads = len(stat_infiles) + dfin = pd.DataFrame(pd.read_csv(stat_infiles[0],delim_whitespace=True,header=0)) + dfin['FCST_LEV'] = dfin['FCST_LEV'].str.replace('P', '') + dfin['FCST_LEV'] = dfin['FCST_LEV'].astype('float64') + dfin = dfin.sort_values('FCST_LEV') + flvls = len(dfin) + plevels = np.empty([fleads,flvls],dtype=float) + pleads = np.empty([fleads,flvls],dtype=float) + prmse = np.empty([fleads,flvls],dtype=float) + pme = np.empty([fleads,flvls],dtype=float) + plevels[0,:] = dfin['FCST_LEV'].to_numpy() + pleads[0,:] = dfin['FCST_LEAD'].to_numpy()/10000 + prmse[0,:] = dfin['RMSE'].astype('float64') + pme[0,:] = dfin['ME'].astype('float64') + + """ + Read in the rest of the data + """ + for i in range(1,len(stat_infiles)): + df = pd.DataFrame(pd.read_csv(stat_infiles[i],delim_whitespace=True,header=0)) + df['FCST_LEV'] = df['FCST_LEV'].str.replace('P', '') + df['FCST_LEV'] = df['FCST_LEV'].astype('float64') + dfnew = df.sort_values('FCST_LEV') + plevels[i,:] = dfnew['FCST_LEV'].to_numpy() + #pleads[i] = dfnew['FCST_LEAD'].to_numpy()[0]/10000 + pleads[i,:] = dfnew['FCST_LEAD'].to_numpy()/10000 + prmse[i,:] = dfnew['RMSE'].astype('float64') + pme[i,:] = dfnew['ME'].astype('float64') + + """ + Create plots + """ + print(plot_bias_levels) + print(plot_rmse_levels) + plot_polar_bias(pleads,plevels,pme,plot_bias_output_file,plot_bias_title,plot_bias_levels) + plot_polar_rmse(pleads,plevels,prmse,plot_rmse_output_file,plot_rmse_title,plot_rmse_levels) + + +if __name__ == '__main__': + main() diff --git a/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py new file mode 100755 index 0000000000..87e2e433c5 --- /dev/null +++ b/parm/use_cases/model_applications/s2s_stratosphere/UserScript_fcstGFS_obsERA_StratospherePolar/polar_t_u_driver.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 + +""" +Create Polar Cap Temperatures and Polar Vortex Winds + +""" +import os +import sys +import datetime +import numpy as np +import metcalcpy.pre_processing.directional_means as directional_means +import METreadnc.util.read_netcdf as read_netcdf +from metcalcpy.util.write_mpr import write_mpr_file + + +def main(): + """ + Read arguments + """ + timevar = sys.argv[1] + leadvar = sys.argv[2] + + """ + Read METplus filename lists + """ + obs_filetxt = os.environ.get('METPLUS_FILELIST_OBS_INPUT','') + fcst_filetxt = os.environ.get('METPLUS_FILELIST_FCST_INPUT','') + + with open(obs_filetxt) as ol: + obs_infiles = ol.read().splitlines() + # Remove the first line if it's there + if (obs_infiles[0] == 'file_list'): + obs_infiles = obs_infiles[1:] + with open(fcst_filetxt) as fl: + fcst_infiles = fl.read().splitlines() + # Remove the first line if it's there + if (fcst_infiles[0] == 'file_list'): + fcst_infiles = fcst_infiles[1:] + + output_dir = os.environ['OUTPUT_DIR'] + full_output_dir = os.path.join(output_dir,'mpr') + + """ + Read dataset + """ + file_reader = read_netcdf.ReadNetCDF() + dsO = file_reader.read_into_xarray(obs_infiles)[0] + dsO = dsO.rename({'lat':'latitude', + 'lon':'longitude'}) + file_reader2 = read_netcdf.ReadNetCDF() + dsF = file_reader2.read_into_xarray(fcst_infiles)[0] + dsF = dsF.rename({'lat':'latitude', + 'lon':'longitude'}) + + """ + Limit Dataset to 100 - 1 hPa + """ + dsO = dsO.sel(pres=slice(1,100)) + dsF = dsF.sel(pres=slice(1,100)) + + + """ + Create Polar Cap Temparatures for Forecast and Obs + """ + TzmO = directional_means.zonal_mean(dsO.T) + TzmO.assign_coords(lat=("latitude",dsO.latitude.values[:,0])) + TO_6090 = directional_means.meridional_mean(TzmO, 60, 90) + TzmF = directional_means.zonal_mean(dsF.T) + TzmF.assign_coords(lat=("latitude",dsF.latitude.values[:,0])) + TF_6090 = directional_means.meridional_mean(TzmF, 60, 90) + + """ + Create Polar Vortex Winds + """ + UzmO = directional_means.zonal_mean(dsO.u) + UzmO.assign_coords(lat=("latitude",dsO.latitude.values[:,0])) + UO_6090 = directional_means.meridional_mean(UzmO, 50, 80) + UzmF = directional_means.zonal_mean(dsF.u) + UzmF.assign_coords(lat=("latitude",dsF.latitude.values[:,0])) + UF_6090 = directional_means.meridional_mean(UzmF, 50, 80) + + + """ + Add P to the levels since they are pressure levels + """ + obs_lvls = ['P'+str(int(op)) for op in dsO.pres.values] + obs_lvls2 = [str(int(op)) for op in dsO.pres.values] + fcst_lvls = ['P'+str(int(fp)) for fp in dsF.pres.values] + + """ + Write output MPR files + """ + dlength1 = len(TO_6090[0,:]) + dlength = dlength1*2 + modname = os.environ.get('MODEL_NAME','GFS') + maskname = os.environ.get('MASK_NAME','FULL') + datetimeindex = dsF.indexes[timevar].to_datetimeindex() + for i in range(len(datetimeindex)): + valid_str = datetimeindex[i].strftime('%Y%m%d_%H%M%S') + leadstr = str(int(leadvar)).zfill(2)+'0000' + outobs = np.concatenate((TO_6090[i,:].values,UO_6090[i,:].values)) + outfcst = np.concatenate((TF_6090[i,:].values,UF_6090[i,:].values)) + write_mpr_file(outfcst,outobs,[0.0]*dlength,[0.0]*dlength,[leadstr]*dlength,[valid_str]*dlength, + ['000000']*dlength,[valid_str]*dlength,modname,'NA',['PolarCapT']*dlength1+['PolarVortexU']*dlength1, + ['K']*dlength1+['m/s']*dlength1,fcst_lvls*2,['PolarCapT']*dlength1+['PolarVortexU']*dlength1, + ['K']*dlength1+['m/s']*dlength1,obs_lvls*2,maskname,obs_lvls2*2,full_output_dir,'polar_cap_T_stat_'+modname) + + + + +if __name__ == '__main__': + main()