Retrospective on go1.18 rollout

[justaugustus]

What happened:

From @BenTheElder in kubernetes/test-infra#25355 (comment):

I'm not sure what happened where this go round, but the go1.18 update was extremely not smooth, it doesn't feel like it got soaked in CI ahead of updating.

From @liggitt in kubernetes/test-infra#25355 (comment):

I agree we should stop bumping master (and now main) images to a new version of go before demonstrating green runs on the go-canary / experimental image, but that seems outside the scope of this PR.

@justaugustus, is there an issue tracking the fallout of the last CI go bump to 1.18 and how we'll change the rollout for go 1.19?

/assign
cc: @kubernetes/release-engineering
xref: #2307, #1139

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

[justaugustus]

From me in kubernetes/test-infra#25355 (comment):

I'm not sure what happened where this go round, but the go1.18 update was extremely not smooth, it doesn't feel like it got soaked in CI ahead of updating.

I agree we should stop bumping master (and now main) images to a new version of go before demonstrating green runs on the go-canary / experimental image, but that seems outside the scope of this PR.
@justaugustus, is there an issue tracking the fallout of the last CI go bump to 1.18 and how we'll change the rollout for go 1.19?

@BenTheElder @liggitt -- I've filed kubernetes/release#2487 for this.

The tl;dr is:

* too many cooks

* we got caught off-guard by the go1.18 official release

* canary presubmits were not run on the PR updating to go1.18 official: [golang: Update to 1.18 from 1.18rc1 kubernetes#108870](https://github.com/kubernetes/kubernetes/pull/108870)

* we bumped kubekins/krte variants (ref: [releng: update kubekins/krte to Go 1.18 #25723](https://github.com/kubernetes/test-infra/pull/25723)) with looking at the canary presubmits

The timing/order of operations for much of this is documented in our golang update issue template, so I think what it came down to here was a few too many folks working on this without following up on the previous steps.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.