Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Change the build log output file extension from log to txt #690

Merged

Conversation

BruceKropp-Raytheon
Copy link
Collaborator

DESCRIPTION OF CHANGES:

When pipeline files are archived to s3 bucket, retrieving the file via a browser attempts to render/display files of known extensions. A browser doesn't generally understand what to do with a .log extension (e.g. build.log). For ease of use in the CI Dashboard, which is a static HTML page, the s3 archived build log needs a .txt extension (e.g. build.txt).

Type of change

  • New feature (non-breaking change which adds functionality)

TESTS CONDUCTED:

This is CI/CD support, not model or app code. The resulting build log (.txt) was viewed within a browser following manual steps.

  • hera.intel
  • orion.intel
  • cheyenne.intel
  • cheyenne.gnu
  • gaea.intel
  • jet.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

would need a pipeline PR build stage (test stage not required) to see that the correct file extension is applied

DOCUMENTATION:

Any material that mentions the build log filename explicitly, in form of: srw_build-${PLATFORM}-${COMPILER}.log

ISSUE:

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • enhancement
  • run_ci

CONTRIBUTORS (optional):

@BruceKropp-Raytheon BruceKropp-Raytheon changed the title Change the build log output file extension from log to txt [develop] Change the build log output file extension from log to txt Mar 23, 2023
Copy link
Collaborator

@panll panll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Copy link
Collaborator

@MichaelLueken MichaelLueken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BruceKropp-Raytheon These changes look good to me!

@MichaelLueken MichaelLueken added enhancement New feature or request run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests and removed run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests labels Mar 24, 2023
@MichaelLueken
Copy link
Collaborator

@BruceKropp-Raytheon Due to the current communication issue between Jenkins and Jet, the Jet tests were aborted (why the continuous-integration/jenkins/pr-merge is showing that the commit was aborted). Visual inspection of the Jenkins tests show that the tests successfully completed - https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-690/1/pipeline.

However, I've noticed some weird behavior with the Publish artifacts to S3 Bucket entries for both Build and Test. It reports that access is denied and fails to push the artifacts to the S3 bucket:

ERROR: Failed to upload files

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied

Also, I'm not seeing the generation of a srw_build-${PLATFORM}-${COMPILER}.txt file or it's inclusion in the S3 bucket. I just want to make sure that everything looks correct on your end before merging this work in. Thanks!

@MichaelLueken
Copy link
Collaborator

@BruceKropp-Raytheon I have merged @kbooker79's PR to update the path to the S3 bucket to the OAR account. Once you have merged the latest develop into your branch and corrected the conflicts, I will be able to move forward with these changes. Thanks!

@MichaelLueken
Copy link
Collaborator

@BruceKropp-Raytheon Is there a specific reason that you closed this PR and opened PR #706? Since the approvals have been given and testing completed in this PR, it would be best to continue within this PR rather than open a new one. Please close PR #706 and reopen this PR, and I can move forward with merging this work. Please let me know if there are issues with doing so. Thank you very much.

@MichaelLueken
Copy link
Collaborator

Reopening previously reviewed PR. So long as the necessary changes are available, will merge this work into develop.

@MichaelLueken MichaelLueken reopened this Apr 3, 2023
@MichaelLueken
Copy link
Collaborator

@BruceKropp-Raytheon The changes in this PR are identical to those in PR #706. I will move forward with merging these changes to develop now. Thank you very much!

@MichaelLueken MichaelLueken merged commit da34aac into ufs-community:develop Apr 3, 2023
gsketefian pushed a commit to gsketefian/ufs-srweather-app that referenced this pull request May 4, 2023
* [develop] Adds a YAML interface for creating a Rocoto XML. (ufs-community#676)

Refactors the creation of a Rocoto XML to use a very generic Jinja2 template that is flexible enough to meet the needs of various workflow configurations supported by SRW. Specifically, it allows for a completely arbitrary workflow to be created under SRW, which includes the addition of completely arbitrary tasks on top of the predefined ones.

---------

Co-authored-by: Michael Kavulich <kavulich@ucar.edu>

* [develop] Change the build log output file extension from log to txt (ufs-community#690)

When pipeline files are archived to s3 bucket, retrieving the file via a browser attempts to render/display files of known extensions. A browser doesn't generally understand what to do with a .log extension (e.g. build.log). For ease of use in the CI Dashboard, which is a static HTML page, the s3 archived build log needs a .txt extension (e.g. build.txt).

* Add "MET_TOOL" definitions to new XML definition YAMLs

* Fix incorrect YAML if block in config_defaults, remove non-needed "USCORE_ENSMEM_NAME_OR_NULL" variable

* - Convert new test "MET_ensemble_verification_only_vx" to new YAML format
 - Fix f-string for utils.py error message

* Fixing more failures (still more to go)

* More fixes, got stand-alone verification test to pass!

 - Fix copy-paste errors in parm/workflow yamls
 - Update corrected variables for new names in exscripts

* Improvement for monitor jobs script: if in debug mode, print the number of tasks that succeeded and failed for failed experiments

* Forgot to include VX_FCST_INPUT_DIR definition for MET_ensemble_verification_only_vx test

* Correct script for task_run_MET_EnsembleStat_vx_APCP

* Pull out CATE and ENSMEM_INDEX from default VX_FCST_INPUT_DIR. My naive attempt to simplify things was the root of all my problems!

* Everything working! Just need to solve problem of non-existent metatask dependencies!

* Fix last failing ensemble test, fundamental tests and all verification tests now pass on Hera!

---------

Co-authored-by: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com>
Co-authored-by: Bruce Kropp - Raytheon <104453151+BruceKropp-Raytheon@users.noreply.github.com>
@BruceKropp-Raytheon BruceKropp-Raytheon deleted the feature/cicd_dashboard branch June 29, 2023 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request run_we2e_coverage_tests Run the coverage set of SRW end-to-end tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants