ci-operator/templates/cluster-launch-installer-e2e*: Destroy with --continue-on-error #1542
Conversation
…ontinue-on-error Taking advantage of openshift/installer@23915f2e (installer/*: Add --continue-on-error to destroy workflow, 2018-09-16, openshift/installer#252) so we can reap at least most of our resources even if the cluster doesn't come up enough for the machine API operator to be able to destroy workers. With various stages of cluster health: 1. Cluster never comes up at all. 2. Cluster healthy enough to create workers. 3. Cluster healthy enough to destroy workers. we're only worried about leakage in the space between 2 and 3. Hopefully there isn't any space there, but without this commit we're currently leaking resources from 1 as well. The two-part destroy attempts are originally from 51df634 (Support an aws installer CI job, 2018-06-07, openshift#928), although there's not much to motivate them there. With --continue-on-error destruction, we're already trying pretty hard to clean everything up. So excepting brief network hiccups and such, I think a single pass is sufficient. And we'll want a better backstop to catch any resources that leak through (e.g. orphaned workers), so I'm dropping the retry here.
openshift-ci-robot
added
the
size/XS
|
We actually were retrying because we were seeing things not deleted fully even when terraform successfully ran to completion. That looked like ec2 errors leaking out to us. Do we retry internally now even with error? |
Do you have examples? That sounds like a Terraform bug (at least in the pre-machine-API world we were living in when #928 landed).
Nope. I'd rather file fixes for any Terraform bugs we hit, and have a CI cleanup backstop (@crawford has been talking up... some tool whose name I forget ;). As I say in the initial message, subsequent tectonic destroy --dir=. --log-level=debug ||
tectonic destroy --dir=. --log-level=debug --continue-on-error |
Cloud Custodian |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: stevekuznetsov, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
|
@wking: Updated the following 2 configmaps:
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Taking advantage of openshift/installer#252, so we can reap at least most of our resources even if the cluster doesn't come up enough for the machine API operator to be able to destroy workers. With various stages of cluster health:
we're only worried about leakage in the space between 2 and 3. Hopefully there isn't any space there, but without this commit we're currently leaking resources from 1 as well.
The two-part destroy attempts are originally from #928, although there's not much to motivate them there. With
--continue-on-errordestruction, we're already trying pretty hard to clean everything up. So excepting brief network hiccups and such, I think a single pass is sufficient. And we'll want a better backstop to catch any resources that leak through (e.g. orphaned workers), so I'm dropping the retry here.CC @smarterclayton, @eparis.