From b7300a3b0a70ac64dc8e45c03e1c4ccc6a0e99cc Mon Sep 17 00:00:00 2001 From: j-opatz Date: Mon, 3 Jan 2022 18:09:10 -0700 Subject: [PATCH 1/6] added new sections, additional wording --- docs/Contributors_Guide/add_use_case.rst | 38 +++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index a074fa32f2..24bc35a96d 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -120,6 +120,11 @@ Use Case Rules processed for analysis. Users can take an example and modify the run times to produce more output as desired. - No errors should result from running the use case. +- The amount of required processing memory should not exceed the current limit + of Github Actions. To accomplish this, pare down input data to the minimum + for the use case to successfully run and, if the use case utilizes a Python + embedding script, make use of a Python memory profiler such as [memory-profiler] + to monitor Python embedding memory usage. - All data that is input to the use case (not generated by METplus) should be referenced relative to {INPUT_BASE} and the directory structure of the use case. For example, if adding a new model application use case found under @@ -144,8 +149,18 @@ Use Case Rules - The use case should be run by someone other than the author to ensure that it runs smoothly outside of the development environment set up by the author. -.. _use_case_documentation: +Use Cases That Exceed Github Actions Memory Limit +------------------------------------------------- + +Below is a list of use cases in the repository that cannot be run in Github Actions +due to their excessive memory usage. They have been tested and cleared by reviewers +of any other issues and can be used by METplus users in the same manner as all +other use cases. + +- model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst +.. _use_case_documentation: + Document New Use Case --------------------- @@ -1024,6 +1039,17 @@ with "Use Case Tests." Click on the job and search for the use case config filename in the log output by using the search box on the top right of the log output. +If the use case does not run successfully in Github Actions and standard error checking +is unsuccessful/your system runs the use case successfully, a recommended step for +use cases that utilize a Python embedding script is to employ a Python memory profiler +such as [memory-profiler] to check for the Python script’s memory usage. Github Actions +has a limited memory storage of [XX] and when exceeded by a use case, will cause the +use case to fail. If your use case exceeds this limit, try to pare down the data held +in memory and use less memory intensive Python routines. + +If memory mitigation cannot move the use case’s memory usage below the Github Actions limit, +see :ref:`exceeded-Github-Actions` for next steps. + Verify that the use case ran in a reasonable amount of time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1039,6 +1065,16 @@ run the set of use cases is now above 20 minutes or so, consider creating a new job for the new use case. See the :ref:`subset_category` section and the multiple medium_range jobs for an example. + +.. _exceeded-Github-Actions: + +Use Cases That Exceed Memory Allocations of Github Actions +---------------------------------------------------------- + +If a use case utilizing Python embedding does not run successfully in +Github Actions due to exceeding the [XX] memory limit and memory mitigation +steps were unsuccessful in lowering memory usage, please take the following steps. + Create a Pull Request ===================== From 2436c0b7ade874384d6b6cb7c8eb94cd109fb683 Mon Sep 17 00:00:00 2001 From: j-opatz Date: Tue, 4 Jan 2022 16:56:37 -0700 Subject: [PATCH 2/6] added more steps for memory-intense use cases --- docs/Contributors_Guide/add_use_case.rst | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index 24bc35a96d..b9790c114b 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -149,6 +149,8 @@ Use Case Rules - The use case should be run by someone other than the author to ensure that it runs smoothly outside of the development environment set up by the author. +.. _memory-intense-use-cases: + Use Cases That Exceed Github Actions Memory Limit ------------------------------------------------- @@ -1043,7 +1045,7 @@ If the use case does not run successfully in Github Actions and standard error c is unsuccessful/your system runs the use case successfully, a recommended step for use cases that utilize a Python embedding script is to employ a Python memory profiler such as [memory-profiler] to check for the Python script’s memory usage. Github Actions -has a limited memory storage of [XX] and when exceeded by a use case, will cause the +has a `limited memory storage `_ and when exceeded by a use case, will cause the use case to fail. If your use case exceeds this limit, try to pare down the data held in memory and use less memory intensive Python routines. @@ -1072,9 +1074,20 @@ Use Cases That Exceed Memory Allocations of Github Actions ---------------------------------------------------------- If a use case utilizing Python embedding does not run successfully in -Github Actions due to exceeding the [XX] memory limit and memory mitigation +Github Actions due to exceeding the memory limit and memory mitigation steps were unsuccessful in lowering memory usage, please take the following steps. +- Document the Github Actions failure in the Github use case issue. + Utilize a Python memory profiler to identify as specifically as possible + where the script exceeds the memory limit. +- Add the use case to the :ref:`memory-intense-use-cases`. +- In the internal_tests/use_cases/all_use_cases.txt file, ensure that the + use case is listed as the lowest-listed use case in its respective category. + Change the number in front of the new use case to a letter, preceeded + by the ‘#’ character:: + + #X::GridStat_fcstRTOFS_obsGHRSST_climWOA_sst::model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf, model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst/ci_overrides.conf:: icecover_env, py_embed + Create a Pull Request ===================== From 4fc334450a6534482869bf158c40d1b385fb28e4 Mon Sep 17 00:00:00 2001 From: j-opatz Date: Wed, 5 Jan 2022 17:01:33 -0700 Subject: [PATCH 3/6] provided final wording add-ons, still needs memory profiler determination --- docs/Contributors_Guide/add_use_case.rst | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index b9790c114b..04194bfa5b 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -1086,7 +1086,20 @@ steps were unsuccessful in lowering memory usage, please take the following step Change the number in front of the new use case to a letter, preceeded by the ‘#’ character:: - #X::GridStat_fcstRTOFS_obsGHRSST_climWOA_sst::model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf, model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst/ci_overrides.conf:: icecover_env, py_embed + #X::GridStat_fcstRTOFS_obsGHRSST_climWOA_sst::model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf, model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst/ci_overrides.conf:: icecover_env, py_embed + +- In the **.github/parm/use_case_groups.json** file, change the “run” variable + for the entry that was added during the :ref:`add_new_category_to_test_runs` + for the new use case to false. +- Push these two updated files to your branch in Github and confirm that it + now compiles successfully. +- During the :ref:`create-a-pull-request` creation, inform the reviewer of + the Github Actions failure. The reviewer should confirm the use case is + successful when run manually, that the memory profiler output confirms that + the Python embedding script exceeds the Github Actions limit, and that + there are no other Github Actions compiling errors. + +.. _create-a-pull-request: Create a Pull Request ===================== From 40d389ce2430a4b1123dd4dec6ed10198b29e7e1 Mon Sep 17 00:00:00 2001 From: j-opatz Date: Thu, 6 Jan 2022 16:34:09 -0700 Subject: [PATCH 4/6] change to wording --- docs/Contributors_Guide/add_use_case.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index 04194bfa5b..872dd59ff5 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -123,8 +123,8 @@ Use Case Rules - The amount of required processing memory should not exceed the current limit of Github Actions. To accomplish this, pare down input data to the minimum for the use case to successfully run and, if the use case utilizes a Python - embedding script, make use of a Python memory profiler such as [memory-profiler] - to monitor Python embedding memory usage. + embedding script, make use of a Python memory profiler of your choice to + monitor Python embedding memory usage. - All data that is input to the use case (not generated by METplus) should be referenced relative to {INPUT_BASE} and the directory structure of the use case. For example, if adding a new model application use case found under @@ -1044,7 +1044,7 @@ log output. If the use case does not run successfully in Github Actions and standard error checking is unsuccessful/your system runs the use case successfully, a recommended step for use cases that utilize a Python embedding script is to employ a Python memory profiler -such as [memory-profiler] to check for the Python script’s memory usage. Github Actions +to check for the Python script’s memory usage. Github Actions has a `limited memory storage `_ and when exceeded by a use case, will cause the use case to fail. If your use case exceeds this limit, try to pare down the data held in memory and use less memory intensive Python routines. @@ -1080,7 +1080,7 @@ steps were unsuccessful in lowering memory usage, please take the following step - Document the Github Actions failure in the Github use case issue. Utilize a Python memory profiler to identify as specifically as possible where the script exceeds the memory limit. -- Add the use case to the :ref:`memory-intense-use-cases`. +- Add the use case to the :ref:`memory-intense-use-cases` list. - In the internal_tests/use_cases/all_use_cases.txt file, ensure that the use case is listed as the lowest-listed use case in its respective category. Change the number in front of the new use case to a letter, preceeded From 9438301ec557dfe099810d0fa54f4a9a341b475e Mon Sep 17 00:00:00 2001 From: j-opatz Date: Mon, 10 Jan 2022 18:16:35 -0700 Subject: [PATCH 5/6] made edits per PR feedback --- docs/Contributors_Guide/add_use_case.rst | 32 +++++++++++++----------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index 872dd59ff5..7209334d18 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -120,11 +120,6 @@ Use Case Rules processed for analysis. Users can take an example and modify the run times to produce more output as desired. - No errors should result from running the use case. -- The amount of required processing memory should not exceed the current limit - of Github Actions. To accomplish this, pare down input data to the minimum - for the use case to successfully run and, if the use case utilizes a Python - embedding script, make use of a Python memory profiler of your choice to - monitor Python embedding memory usage. - All data that is input to the use case (not generated by METplus) should be referenced relative to {INPUT_BASE} and the directory structure of the use case. For example, if adding a new model application use case found under @@ -1041,13 +1036,20 @@ with "Use Case Tests." Click on the job and search for the use case config filename in the log output by using the search box on the top right of the log output. -If the use case does not run successfully in Github Actions and standard error checking -is unsuccessful/your system runs the use case successfully, a recommended step for -use cases that utilize a Python embedding script is to employ a Python memory profiler -to check for the Python script’s memory usage. Github Actions -has a `limited memory storage `_ and when exceeded by a use case, will cause the -use case to fail. If your use case exceeds this limit, try to pare down the data held -in memory and use less memory intensive Python routines. +If the use case fails in GitHub Actions but runs successfully in the user's environment, +potential reasons include: + +- Errors providing input data (see :ref:`use_case_input_data`) +- Using hard-coded paths from the user's machine +- Referencing variables set in the user's configuration file or local environment +- Memory usuage of the use case exceeds the available memory in hte Github Actions environment + +Github Actions has `limited memory `_ +available and will cause the use case to fail when exceeded. A failure caused by exceeding +the memory allocation in a Python Embedding script may result in an unclear error message. +If you suspect that this is the case, consider utilizing a Python memory profiler to check the +Python script's memory usage. If your use case exceeds the limit, try to pare +down the data held in memory and use less memory intensive Python routines. If memory mitigation cannot move the use case’s memory usage below the Github Actions limit, see :ref:`exceeded-Github-Actions` for next steps. @@ -1088,9 +1090,9 @@ steps were unsuccessful in lowering memory usage, please take the following step #X::GridStat_fcstRTOFS_obsGHRSST_climWOA_sst::model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf, model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst/ci_overrides.conf:: icecover_env, py_embed -- In the **.github/parm/use_case_groups.json** file, change the “run” variable - for the entry that was added during the :ref:`add_new_category_to_test_runs` - for the new use case to false. +- In the **.github/parm/use_case_groups.json** file, remove the entry that + was added during the :ref:`add_new_category_to_test_runs` + for the new use case. This will stop the use case from running on a pull request. - Push these two updated files to your branch in Github and confirm that it now compiles successfully. - During the :ref:`create-a-pull-request` creation, inform the reviewer of From 7a3042a8fa72a325ba0c29d8e6f3ad247643eadf Mon Sep 17 00:00:00 2001 From: j-opatz <59586397+j-opatz@users.noreply.github.com> Date: Mon, 10 Jan 2022 18:18:27 -0700 Subject: [PATCH 6/6] Update documentation per PR feedback Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --- docs/Contributors_Guide/add_use_case.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index 7209334d18..eb4e07ad2c 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -1085,7 +1085,7 @@ steps were unsuccessful in lowering memory usage, please take the following step - Add the use case to the :ref:`memory-intense-use-cases` list. - In the internal_tests/use_cases/all_use_cases.txt file, ensure that the use case is listed as the lowest-listed use case in its respective category. - Change the number in front of the new use case to a letter, preceeded + Change the number in front of the new use case to an 'X', preceeded by the ‘#’ character:: #X::GridStat_fcstRTOFS_obsGHRSST_climWOA_sst::model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.conf, model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst/ci_overrides.conf:: icecover_env, py_embed