-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci-operator/templates/openshift/installer/cluster-launch-installer-upi*: Switch on BRANCH for RHCOS AMI #6248
Conversation
Classifying the early errors: $ curl -s 'https://ci-search-ci-search-next.svc.ci.openshift.org/search?name=rehearse-6248&search=error:%20build%20error:%20no%20such%20image&search=could%20not%20resolve%20base%20image.*not%20found&search=syntax%20error%20near%20unexpected%20token' | jq -r '. | to_entries[] | .key + " " + (.value | keys | join(" "))'
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.2-e2e-aws-upi/1 error: build error: no such image
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.2-e2e-vsphere/1 error: build error: no such image
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.4-e2e-aws-upi/1 error: build error: no such image
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.4-e2e-gcp-upi/1 error: build error: no such image
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.4-e2e-vsphere/1 error: build error: no such image
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-aws-upi/1 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-gcp-upi/1 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-vsphere/1 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.4-e2e-vsphere/1 syntax error near unexpected token
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.5-e2e-vsphere/1 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-e2e-vsphere/1 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-launch-vsphere/1 could not resolve base image.*not found None of which seem related to this PR. |
…i*: Switch on BRANCH for RHCOS AMI As discussed in dc26075 (ci-operator/templates/openshift/installer: Bump AWS UPI to RHCOS 43.81.201911221453.0, 2019-12-02, openshift#6192), we want a way to switch on the z stream when selecting the RHCOS AMI. We amight also need this if other parts of the recommended user-provided-infrastructure flow diverge between z streams, although at the moment the AMI is the only divergence. In this commit, I'm sourcing my AMIs with: $ git cat-file -p origin/release-4.1:data/data/rhcos.json | jq -r '.amis["us-east-1"].hvm' ami-046fe691f52a953f9 $ git cat-file -p origin/release-4.2:data/data/rhcos.json | jq -r '.amis["us-east-1"].hvm' ami-01e7fdcb66157b224 $ git cat-file -p origin/release-4.3:data/data/rhcos.json | jq -r '.amis["us-east-1"].hvm' ami-014ce8846db8b463d $ git cat-file -p origin/master:data/data/rhcos.json | jq -r '.amis["us-east-1"].hvm' ami-014ce8846db8b463d as the installer stood at 2019-12-04T23:30Z. Comparing with the user-facing docs: $ git --no-pager grep -hA1 us-east-1 origin/enterprise-4.1:modules/installation-aws-user-infra-rhcos-ami.adoc |`us-east-1` |`ami-046fe691f52a953f9` $ git --no-pager grep -hA1 us-east-1 origin/enterprise-4.2:modules/installation-aws-user-infra-rhcos-ami.adoc |`us-east-1` |`ami-01e7fdcb66157b224` $ git --no-pager grep -hA1 us-east-1 origin/enterprise-4.3:modules/installation-aws-user-infra-rhcos-ami.adoc |`us-east-1` |`ami-01e7fdcb66157b224` $ git --no-pager grep -hA1 us-east-1 origin/enterprise-4.4:modules/installation-aws-user-infra-rhcos-ami.adoc fatal: Invalid object name 'origin/enterprise-4.4'. $ git --no-pager grep -hA1 us-east-1 origin/master:modules/installation-aws-user-infra-rhcos-ami.adoc |`us-east-1` |`ami-01e7fdcb66157b224` So we currently diverge for 4.3 and master, where I have [1,2] open to close the gaps. The BRANCH environment variable has been around since 22e30ea (The release promotion periodic can be conditional and no-op if no change, 2019-04-30, openshift#3649). I added it to all jobs based on the UPI template where it was missing (it was already on a number of those jobs). [1]: openshift/openshift-docs#18396 [2]: openshift/openshift-docs#18395
Early failures in the second round of CI: $ curl -s 'https://ci-search-ci-search-next.svc.ci.openshift.org/search?name=rehearse-6248-&search=error:+build+error:+no+such+image&search=could+not+resolve+base+image.*not+found&search=syntax+error+near+unexpected+token&search=Terraform+does+not+automatically+rollback+in+the+face+of+errors' | jq -r '. | to_entries[] | .key + " " + (.value | keys | join(" "))' | grep /2
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.2-e2e-vsphere/2 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-aws-upi/2 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-gcp-upi/2 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-vsphere/2 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.3-e2e-vsphere/2 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.5-e2e-vsphere/2 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-e2e-vsphere/2 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-launch-vsphere/2 could not resolve base image.*not found Now none of the failures look related to this PR. |
This is probably a place where our UPI flow has diverged, because openshift/installer#1794 was never backported to 4.1. |
…i*: WorkerSubnet for 4.1 456f258 (ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e: WorkerSubnet -> Subnet, 2019-06-09, openshift#4013) does not apply to 4.1. Switch on BRANCH to use the old WorkerSubnet for 4.1.
Oops. Fumbled the first attempt at $ curl -s 'https://ci-search-ci-search-next.svc.ci.openshift.org/search?name=rehearse-6248-&search=error:+build+error:+no+such+image&search=could+not+resolve+base+image.*not+found&search=syntax+error+near+unexpected+token&search=Terraform+does+not+automatically+rollback+in+the+face+of+errors&search=Subnet.*must+have+values&search=add-iam-policy-binding.*is+the+subject+of+a+conflict&search=Failing+tests' | jq -r '. | to_entries[] | .key + " " + (.value | keys | join(" "))' | grep /3
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.1-e2e-aws-upi/3 Failing tests
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.1-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.2-e2e-aws-upi/3 Subnet.*must have values
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.2-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.3-e2e-aws-upi/3 Subnet.*must have values
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.3-e2e-gcp-upi/3 add-iam-policy-binding.*is the subject of a conflict
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.3-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.4-e2e-aws-upi/3 Subnet.*must have values
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.4-e2e-vsphere/3 Failing tests
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-aws-upi/3 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-gcp-upi/3 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-installer-release-4.5-e2e-vsphere/3 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.3-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.4-e2e-vsphere/3 Failing tests
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-machine-config-operator-release-4.5-e2e-vsphere/3 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.1-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.1-launch-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.2-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.3-e2e-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.3-launch-vsphere/3 Terraform does not automatically rollback in the face of errors
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-e2e-vsphere/3 could not resolve base image.*not found
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/6248/rehearse-6248-pull-ci-openshift-origin-release-4.5-launch-vsphere/3 could not resolve base image.*not found |
Still one 4.4 vSphere job from round 4 in flight, but here are the rest of them (putting the regexp before the job name so we can collect by regexp): $ curl -s 'https://ci-search-ci-search-next.svc.ci.openshift.org/search?name=rehearse-6248-&search=error:+build+error:+no+such+image&search=could+not+resolve+base+image.*not+found&search=syntax+error+near+unexpected+token&search=Terraform+does+not+automatically+rollback+in+the+face+of+errors&search=Subnet.*must+have+values&search=add-iam-policy-binding.*is+the+subject+of+a+conflict&search=Failing+tests&search=Waiter+StackDeleteComplete+failed:+Waiter+encountered+a+terminal+failure+state&search=failed+to+initialize+the+cluster' | jq -r '. | to_entries[] | .key + " " + (.value | keys | join(" "))' | sed -n 's|.*ci-openshift-\([^ ]*/4\) \(.*\)|\2\t\1|p' | sort
could not resolve base image.*not found installer-release-4.5-e2e-aws-upi/4
could not resolve base image.*not found installer-release-4.5-e2e-gcp-upi/4
could not resolve base image.*not found installer-release-4.5-e2e-vsphere/4
could not resolve base image.*not found machine-config-operator-release-4.5-e2e-vsphere/4
could not resolve base image.*not found origin-release-4.5-e2e-vsphere/4
could not resolve base image.*not found origin-release-4.5-launch-vsphere/4
failed to initialize the cluster installer-release-4.4-e2e-gcp-upi/4
Failing tests installer-release-4.1-e2e-aws-upi/4
Failing tests installer-release-4.2-e2e-aws-upi/4
Failing tests installer-release-4.4-e2e-aws-upi/4
Failing tests installer-release-4.4-e2e-vsphere/4
Failing tests machine-config-operator-release-4.4-e2e-vsphere/4
Failing tests origin-release-4.4-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors installer-release-4.1-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors installer-release-4.2-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors installer-release-4.3-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors machine-config-operator-release-4.3-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.1-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.1-launch-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.2-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.2-launch-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.3-e2e-vsphere/4
Terraform does not automatically rollback in the face of errors origin-release-4.3-launch-vsphere/4
Waiter StackDeleteComplete failed: Waiter encountered a terminal failure state installer-release-4.3-e2e-aws-upi/4 None of those seem to be related to this PR, so I think we're good to go here if we want this direction. And taking another stab at motivating each of the
So there are a number of ways we could do this, and while I like the |
4.1) RHCOS_AMI=ami-046fe691f52a953f9;; | ||
4.2) RHCOS_AMI=ami-01e7fdcb66157b224;; | ||
4.3) RHCOS_AMI=ami-014ce8846db8b463d;; | ||
*) RHCOS_AMI=ami-014ce8846db8b463d;; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is OK but I would prefer automating openshift/installer#2092 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cgwalters, soltysh, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest Please review the full test history for this PR and help us cut down flakes. |
2 similar comments
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/hold Looks like generated-config is sad:
Not clear on why it doesn't like the environment variable... |
@wking: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Talked this over with @stevekuznetsov and @droslean out of band. Takeaway is that this sort of conditional logic is not supported. I'm going to wait for the multi-step stuff (e.g. openshift/ci-tools#319) and take another run at this then. /close |
@wking: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I've filed https://bugzilla.redhat.com/show_bug.cgi?id=1782062 and https://bugzilla.redhat.com/show_bug.cgi?id=1782063 to track the failures until we get a path to fixing them. |
As discussed in dc26075 (#6192), we want a way to switch on the z stream when selecting the RHCOS AMI. We amight also need this if other parts of the recommended user-provided-infrastructure flow diverge between z streams, although at the moment the AMI is the only divergence.
In this commit, I'm sourcing my AMIs with:
as the installer stood at 2019-12-04T23:30Z. Comparing with the user-facing docs:
So we currently diverge for 4.3 and master, where I have openshift/openshift-docs#18396 and openshift/openshift-docs#18395 open to close the gaps.
The BRANCH environment variable has been around since 22e30ea (#3649). I added it to all jobs based on the UPI template where it was missing (it was already on a number of those jobs).