Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail run conformance test make target on test failure #1177

Merged
merged 1 commit into from
Oct 23, 2023

Conversation

ciarams87
Copy link
Member

Proposed changes

Problem: The conformance tests jobs are not being marked as failed in the pipeline even when there are failed tests. This is because the last command in the pod and the Make target relate to the conformance profile report and not the tests themselves, so the exit code is always 0 as long as the job completes.

Solution: Fetch the test result from the conformance profile yaml and exit with 2 if there's a failure result

Testing: Tested the conformance tests locally and verified that the make target fails when there's a failure in the conformance tests, and doesn't fail when there isn't

Closes #1176

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation
  • I have rebased my branch onto main
  • I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

@ciarams87 ciarams87 requested a review from a team as a code owner October 23, 2023 13:07
@github-actions github-actions bot added the bug Something isn't working label Oct 23, 2023
@ciarams87 ciarams87 merged commit 725ff40 into nginxinc:main Oct 23, 2023
23 checks passed
@ciarams87 ciarams87 deleted the fix/conformance-step-not-failing branch October 23, 2023 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Conformance job in pipeline is not failing when tests are failing
4 participants