Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

USHIFT-3136: Retry virt-install command in case of installation error #3448

Merged
merged 2 commits into from
Jun 13, 2024

Conversation

ggiguash
Copy link
Contributor

@ggiguash ggiguash commented Jun 6, 2024

  • The virt-install command is now retried once if any error occurs.
  • VM boot timeout is lowered to 20m to save some wait time in case of stuck VMs
  • VM greenboot timeout is left on 30m not to miss slow container image downloads

This change should provide a workaround for VM installation problems like ostree pull timeouts (USHIFT-3136), stuck ARM VM on reboot (USHIFT-3152), etc.

See this log as an example of a retry.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jun 6, 2024

@ggiguash: This pull request references USHIFT-3136 which is a valid jira issue.

In response to this:

  • The virt-install command is now retried once if any error occurs.
  • VM boot timeout is lowered to 20m to save some wait time in case of stuck VMs
  • VM greenboot timeout is left on 30m not to miss slow container image downloads

This change should provide a workaround for VM installation problems like ostree pull timeouts (USHIFT-3136), stuck ARM VM on reboot (USHIFT-3152), etc.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 6, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jun 6, 2024

@ggiguash: This pull request references USHIFT-3136 which is a valid jira issue.

In response to this:

  • The virt-install command is now retried once if any error occurs.
  • VM boot timeout is lowered to 20m to save some wait time in case of stuck VMs
  • VM greenboot timeout is left on 30m not to miss slow container image downloads

This change should provide a workaround for VM installation problems like ostree pull timeouts (USHIFT-3136), stuck ARM VM on reboot (USHIFT-3152), etc.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from jogeo and pmtk June 6, 2024 13:35
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2024
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 6, 2024

/test microshift-metal-tests

@ggiguash ggiguash force-pushed the virt_install_retries branch 4 times, most recently from 0b61bdb to 2d9dbeb Compare June 6, 2024 14:55
@ggiguash ggiguash force-pushed the virt_install_retries branch from 2d9dbeb to 701fb72 Compare June 6, 2024 14:56
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 6, 2024

/test metal-bootc-test metal-bootc-test-arm metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 6, 2024

/test metal-bootc-test metal-bootc-test-arm metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 7, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 7, 2024

Found a failed VM here. The code needs to be fixed.

ERROR    Error: --disk pool=vm-storage-cos9-src@multi-nic,size=20: Storage pool not found: no storage pool with matching name 'vm-storage-cos9-src@multi-nic'

@ggiguash ggiguash marked this pull request as draft June 7, 2024 10:00
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2024
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 7, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 7, 2024

Still needs a fix, see this log.

ERROR    Error: --disk pool=vm-storage-cos9-src@log-scan,size=20: cannot open directory '/home/ec2-user/microshift/_output/test-images/vm-storage/vm-storage-cos9-src@log-scan': No such file or directory

@ggiguash ggiguash force-pushed the virt_install_retries branch from 3fa18f2 to fda51e8 Compare June 7, 2024 16:02
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 7, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

3 similar comments
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 8, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 8, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 8, 2024

Need to find a case when VM creation fails to make sure the new logic works OK.
/test metal-bootc-test metal-bootc-test-arm metal-periodic-test metal-periodic-test-arm

@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 9, 2024

The retry seems to be working as intended. See this log as an example.

@ggiguash ggiguash marked this pull request as ready for review June 9, 2024 05:43
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 9, 2024
@openshift-ci openshift-ci bot requested a review from copejon June 9, 2024 05:44
@ggiguash
Copy link
Contributor Author

ggiguash commented Jun 9, 2024

/test metal-periodic-test

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jun 9, 2024

@ggiguash: This pull request references USHIFT-3136 which is a valid jira issue.

In response to this:

  • The virt-install command is now retried once if any error occurs.
  • VM boot timeout is lowered to 20m to save some wait time in case of stuck VMs
  • VM greenboot timeout is left on 30m not to miss slow container image downloads

This change should provide a workaround for VM installation problems like ostree pull timeouts (USHIFT-3136), stuck ARM VM on reboot (USHIFT-3152), etc.

See this log as an example of a retry.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ggiguash
Copy link
Contributor Author

/test ocp-full-conformance-rhel-eus

1 similar comment
@copejon
Copy link
Contributor

copejon commented Jun 11, 2024

/test ocp-full-conformance-rhel-eus

@ggiguash
Copy link
Contributor Author

The failures in this job are not related to the changes. Saving CI cycles.
/override ci/prow/ocp-full-conformance-rhel-eus

Copy link
Contributor

openshift-ci bot commented Jun 11, 2024

@ggiguash: Overrode contexts on behalf of ggiguash: ci/prow/ocp-full-conformance-rhel-eus

In response to this:

The failures in this job are not related to the changes. Saving CI cycles.
/override ci/prow/ocp-full-conformance-rhel-eus

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@copejon
Copy link
Contributor

copejon commented Jun 11, 2024

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 11, 2024
Copy link
Contributor

openshift-ci bot commented Jun 11, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon, ggiguash

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ggiguash
Copy link
Contributor Author

/cherry-pick release-4.16

@openshift-cherrypick-robot

@ggiguash: once the present PR merges, I will cherry-pick it on top of release-4.16 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 2ca55ac and 2 for PR HEAD fda51e8 in total

@ggiguash
Copy link
Contributor Author

/test metal-periodic-test

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 7b4a6fb and 1 for PR HEAD fda51e8 in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 7c991ee and 0 for PR HEAD fda51e8 in total

@openshift-ci-robot
Copy link

/hold

Revision fda51e8 was retested 3 times: holding

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 12, 2024
@ggiguash
Copy link
Contributor Author

The VM startup works as expected. The tests fail due to FIPS problems unrelated to this PR. Overriding to save CI cycles.
/override ci/prow/metal-periodic-test

Copy link
Contributor

openshift-ci bot commented Jun 13, 2024

@ggiguash: Overrode contexts on behalf of ggiguash: ci/prow/metal-periodic-test

In response to this:

The VM startup works as expected. The tests fail due to FIPS problems unrelated to this PR. Overriding to save CI cycles.
/override ci/prow/metal-periodic-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ggiguash
Copy link
Contributor Author

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2024
Copy link
Contributor

openshift-ci bot commented Jun 13, 2024

@ggiguash: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit eb597f2 into openshift:main Jun 13, 2024
11 checks passed
@openshift-cherrypick-robot

@ggiguash: new pull request created: #3482

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants