Update to buildroot 2023.02.9 & Golang 1.21.6 #18020

travier · 2024-01-22T16:00:46Z

Heavily inspired by #17093

Building and launching an instance using the KVM2 driver & cri-o runtime works.

Did not do any further tests so far.

k8s-ci-robot · 2024-01-22T16:00:56Z

Hi @travier. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

minikube-bot · 2024-01-22T16:03:05Z

Can one of the admins verify this patch?

afbjorklund · 2024-01-22T16:08:29Z

You might want to separate the ISO updates from the CI updates

travier · 2024-01-22T16:13:29Z

They were all together in the parent PRs. A lot of the ISO updates are removing packages to use the one from the new buildroot version.

Could you clarify which changes do you think I should split?

afbjorklund · 2024-01-22T16:15:08Z

Could you clarify which changes do you think I should split?

I just meant the deploy/iso vs. the hack/update, but if they were together before it might be easier for @spowelljr

spowelljr · 2024-01-22T16:57:44Z

The hack/update change is because we were managing the CNI plugins package ourself, but the new Buildroot version has it included, so I was removing our CI from auto updating the CNI plugins in ISO since the files no longer exist, so I think it makes sense to include it

spowelljr · 2024-01-22T20:49:33Z

ok-to-build-iso

travier · 2024-01-28T16:09:32Z

Documenting missing bits / issues as I find them:

cni-plugins from buildroot does not include the host-local plugin (breaks cri-o)
crun version in buildroot is only 1.7.2

spowelljr · 2024-01-29T23:55:53Z

Just a heads up that we're resolving issues with our CI infrastructure so won't be able to build the ISO right now, but bear with us and we'll try building this as soon as we can

medyagh · 2024-01-30T18:16:37Z

@travier thank you very much for this contribution, glad to find another developer touching this code, this part of the code requires special experties, our CI machine is currently having issue, as soon as we fix that we can build the ISO again

travier · 2024-01-30T18:23:18Z

How would you prefer to go for crun & cni-plugins? Should we keep the updated/custom versions in minikube or should we patch the buildroot ones?

From experiments so far, we can not have external packages with the same names as the ones in buildroot so we would need to rename them to keep them in the minikube tree. Otherwise this will need "large" patches for the buildroot tree.

spowelljr · 2024-01-30T21:33:39Z

From a maintenance standpoint I think having a custom module with a different name would be ideal. Unless there is an easy to to update the patches. I'm just thinking if Buildroot updates crun they we have to remove the patches that are now included, and a new version comes out we have to add new patches, seems like more work than the custom package.

travier · 2024-01-31T18:57:29Z

So, with this update, this still runs crun 1.7.2 (not sure why it's not pulling the latest yet) but it uses the right cni-plugins and thus the pods can run.

I pulled in some other package updates. Let me know if you prefer to have them split from this PR.

spowelljr · 2024-01-31T23:11:48Z

Awesome thanks! I've made significant progress in resolving the CI infrastructure issues. The machines are running again, but there are 1,000 test jobs in the backlog, I'm working on resolving the login issue and then I can clear the backlog and we can try building this.

spowelljr · 2024-02-13T21:20:06Z

Sorry for the delay, just finished resolving the CI issues, let's try building this finally

spowelljr · 2024-02-13T21:20:12Z

/ok-to-build-iso

spowelljr · 2024-02-13T21:29:29Z

I pulled in some other package updates. Let me know if you prefer to have them split from this PR.

Due to the backup in the whole CI I'm happy to merge this with all the package updates as long as the build is successful

travier · 2024-02-14T11:29:24Z

For the GitHub workflow, this fails on:

---------------- 1 Failures :( ----------------------------
[
  "TestFunctional/parallel/DashboardCmd"
]

I don't have access to the ISO PR Build Push logs.

Edit: I've tried enabling the dashboard and "it worked" as far as I could see. I've not tried running the test yet.

spowelljr · 2024-02-14T17:41:53Z

The DashboardCmd failure you can ignore, the test is flaky and needs to be fixed.

The ISO build failed with VCS stamping cri-dockerd

16:44:34 mkdir -p build/linux/cri-dockerd
16:44:34 cd /home/jenkins/go/src/k8s.io/minikube/out/buildroot/output-aarch64/build/cri-dockerd-aarch64-b58acf8f78f9d7bce1241d1cddb0932e7101f278 && env CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w -buildid=b58acf8 -X github.com/Mirantis/cri-dockerd/cmd/version.Version=0.3.3 -X github.com/Mirantis/cri-dockerd/cmd/version.PreRelease=`grep -q dev <<< "0.3.3" && echo "pre" || echo ""` -X github.com/Mirantis/cri-dockerd/cmd/version.GitCommit=b58acf8" -o cri-dockerd
16:44:35 error obtaining VCS status: exit status 128
16:44:35 	Use -buildvcs=false to disable VCS stamping.

Also, looking at the log we're in an aarch64 (output-aarch64/build/cri-dockerd-aarch64) directory, but the build command is using amd64 (GOARCH=amd64 go build).

Edit: Looks like you added the disable VCS for x86 but forgot to add it to aarch64, so that should be a simple fix.

Edit 2: It's using amd64 because we're using the make command in cri-dockerd's Makefile, which is:

GOARCH=$(ARCH) go build -trimpath $(CRI_DOCKERD_LDFLAGS) -o $@

And since the machine building the ISO is amd64 it's setting GOARCH to amd64.

spowelljr · 2024-02-15T14:14:25Z

/ok-to-build-iso

travier · 2024-02-15T14:55:46Z

Updated to fix the issues above. It builds on x86_64 for me. Not tested.

Only update to 1.28.3 due to Go 1.21 requirement for 1.29+.

This file is no longer needed. See: buildroot/buildroot@3efb5e3 See: https://github.com/buildroot/buildroot/blob/2021.02.12/board/pc/genimage-efi.cfg?rgh-link-date=2024-02-21T17%3A42%3A39Z See: https://github.com/buildroot/buildroot/blob/2023.02.9/board/pc/genimage-efi.cfg?rgh-link-date=2024-02-21T17%3A42%3A39Z See: kubernetes#18020 (comment)

travier · 2024-02-22T19:01:18Z

Thanks! Updated.

spowelljr · 2024-02-22T21:40:41Z

ok-to-build-iso

minikube-bot · 2024-02-23T03:44:36Z

Hi @travier, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

spowelljr · 2024-02-23T16:16:44Z

/ok-to-test

minikube-pr-bot · 2024-02-23T18:03:14Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18020) |
+----------------+----------+---------------------+
| minikube start | 51.9s    | 50.5s               |
| enable ingress | 27.8s    | 24.9s               |
+----------------+----------+---------------------+

Times for minikube ingress: 27.6s 27.0s 28.6s 28.1s 27.6s
Times for minikube (PR 18020) ingress: 23.6s 26.7s 24.1s 23.1s 27.2s

Times for minikube start: 52.5s 49.7s 52.4s 49.7s 55.1s
Times for minikube (PR 18020) start: 49.4s 50.4s 51.4s 49.9s 51.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18020) |
+----------------+----------+---------------------+
| minikube start | 24.4s    | 23.2s               |
| enable ingress | 20.8s    | 21.3s               |
+----------------+----------+---------------------+

Times for minikube ingress: 20.8s 20.8s 20.8s 20.8s 20.8s
Times for minikube (PR 18020) ingress: 20.9s 21.8s 20.9s 21.8s 20.9s

Times for minikube start: 25.6s 25.9s 24.5s 21.7s 24.5s
Times for minikube (PR 18020) start: 21.7s 24.8s 21.8s 25.2s 22.7s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18020) |
+----------------+----------+---------------------+
| minikube start | 21.7s    | 22.8s               |
| enable ingress | 31.1s    | 34.5s               |
+----------------+----------+---------------------+

Times for minikube start: 20.8s 20.4s 20.2s 23.6s 23.6s
Times for minikube (PR 18020) start: 23.9s 20.9s 20.6s 23.6s 25.0s

Times for minikube ingress: 31.3s 31.4s 31.3s 31.3s 30.3s
Times for minikube (PR 18020) ingress: 31.3s 31.3s 31.4s 31.4s 47.4s

minikube-pr-bot · 2024-02-23T20:14:06Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux_containerd	TestDownloadOnly/v1.29.0-rc.2/json-events (gopogh)	0.00 (chart)
Docker_Linux_crio	TestDownloadOnly/v1.29.0-rc.2/json-events (gopogh)	0.00 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	0.00 (chart)
Hyper-V_Windows	TestDockerFlags (gopogh)	0.00 (chart)
Hyper-V_Windows	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	0.00 (chart)
Hyper-V_Windows	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/flannel/NetCatPod (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh)	0.00 (chart)
KVM_Linux_crio	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	0.00 (chart)
KVM_Linux_crio	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	0.00 (chart)
KVM_Linux_crio	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
KVM_Linux_crio	TestKubernetesUpgrade (gopogh)	0.00 (chart)
KVM_Linux_crio	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	0.00 (chart)
KVM_Linux_crio	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	0.00 (chart)
KVM_Linux	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	0.00 (chart)
KVM_Linux	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	0.00 (chart)
KVM_Linux	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	0.00 (chart)
KVM_Linux	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	0.00 (chart)
KVM_Linux	TestKubernetesUpgrade (gopogh)	0.00 (chart)
KVM_Linux	TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh)	0.00 (chart)
KVM_Linux	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	0.00 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

travier · 2024-02-24T14:35:49Z

I've started looking at TestStartStop/group/old-k8s-version/serial/DeployApp and it's trying to run Kubernetes 1.16 with the latest cri-o, which is likely not going to work as cri-o is "version"-locked with kubernetes.

I have not looked at the other failures yet.

spowelljr · 2024-02-26T20:47:01Z

We talked about it in office hours today, basically as of 1.28(?) the cri-o is built for the matching version of Kubernetes and there is no backwards compatibility guaranteed. So the updated cri-o is not working with the old version of Kubernetes we're testing with (1.16). As long as it works with a newer version say 1.24 then I'm happy to bump the "old" version we use for testing.

spowelljr · 2024-02-28T23:41:11Z

This worked down to K8s v1.20 for me, so looks good to me

k8s-ci-robot · 2024-02-28T23:42:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spowelljr, travier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spowelljr · 2024-02-28T23:43:22Z

Thank you very much for your continued pursuit of this PR @travier, happy to be back on the current version of Buildroot, and unblocks using Go 1.21 and will allow us to install new packages that were blocked like Docker buildx

travier · 2024-02-29T14:27:26Z

@spowelljr Thanks for the help and reviews! I'll likely be looking at crun and other components updates now.

k8s-ci-robot requested review from afbjorklund and medyagh January 22, 2024 16:00

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 22, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 22, 2024

travier mentioned this pull request Jan 22, 2024

ISO: Update Buildroot & Go version #17093

Closed

travier force-pushed the buildroot-update branch from 139ed6d to 3a735df Compare January 31, 2024 18:38

travier force-pushed the buildroot-update branch 2 times, most recently from fe5dab7 to 9b496d8 Compare February 15, 2024 09:52

travier and others added 7 commits February 22, 2024 20:00

iso/package: Move cni-plugins to cni-plugins-latest

52e126b

Kicbase/ISO: Update cri-o from v1.24.1 to v1.28.3

90492da

Only update to 1.28.3 due to Go 1.21 requirement for 1.29+.

Kicbase/ISO: Update cri-o from v1.28.31 to v1.29.1

b3365f0

Kicbase/ISO: Update runc from v1.1.10 to v1.1.11

b69ff3a

Makefile & hack: Add update-crun-version target & update script

c46714b

Kicbase/ISO: Update crun from v1.2 to v1.14

84691f8

travier force-pushed the buildroot-update branch from 312637e to 28687d6 Compare February 22, 2024 19:01

Updating ISO to v1.32.1-1708638130-18020

dc93288

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 23, 2024

spowelljr approved these changes Feb 28, 2024

View reviewed changes

spowelljr merged commit d4e0a20 into kubernetes:master Feb 28, 2024
22 of 41 checks passed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 28, 2024

travier deleted the buildroot-update branch February 29, 2024 14:26

travier mentioned this pull request Feb 29, 2024

Add crun update script and update crun to 1.13 #17952

Closed

This was referenced Mar 12, 2024

CI: Auto build ISO when bumping Go version #18368

Merged

CI: Fix CNI Plugins version variable name #18374

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to buildroot 2023.02.9 & Golang 1.21.6 #18020

Update to buildroot 2023.02.9 & Golang 1.21.6 #18020

travier commented Jan 22, 2024 •

edited

Loading

k8s-ci-robot commented Jan 22, 2024

minikube-bot commented Jan 22, 2024

afbjorklund commented Jan 22, 2024

travier commented Jan 22, 2024

afbjorklund commented Jan 22, 2024

spowelljr commented Jan 22, 2024 •

edited

Loading

spowelljr commented Jan 22, 2024

travier commented Jan 28, 2024 •

edited

Loading

spowelljr commented Jan 29, 2024

medyagh commented Jan 30, 2024

travier commented Jan 30, 2024 •

edited

Loading

spowelljr commented Jan 30, 2024

travier commented Jan 31, 2024

spowelljr commented Jan 31, 2024

spowelljr commented Feb 13, 2024 •

edited

Loading

spowelljr commented Feb 13, 2024

spowelljr commented Feb 13, 2024

travier commented Feb 14, 2024 •

edited

Loading

spowelljr commented Feb 14, 2024 •

edited

Loading

spowelljr commented Feb 15, 2024

travier commented Feb 15, 2024 •

edited

Loading

travier commented Feb 22, 2024

spowelljr commented Feb 22, 2024

minikube-bot commented Feb 23, 2024

spowelljr commented Feb 23, 2024

minikube-pr-bot commented Feb 23, 2024

minikube-pr-bot commented Feb 23, 2024

travier commented Feb 24, 2024

spowelljr commented Feb 26, 2024

spowelljr commented Feb 28, 2024

k8s-ci-robot commented Feb 28, 2024

spowelljr commented Feb 28, 2024

travier commented Feb 29, 2024

Update to buildroot 2023.02.9 & Golang 1.21.6 #18020

Update to buildroot 2023.02.9 & Golang 1.21.6 #18020

Conversation

travier commented Jan 22, 2024 • edited Loading

k8s-ci-robot commented Jan 22, 2024

minikube-bot commented Jan 22, 2024

afbjorklund commented Jan 22, 2024

travier commented Jan 22, 2024

afbjorklund commented Jan 22, 2024

spowelljr commented Jan 22, 2024 • edited Loading

spowelljr commented Jan 22, 2024

travier commented Jan 28, 2024 • edited Loading

spowelljr commented Jan 29, 2024

medyagh commented Jan 30, 2024

travier commented Jan 30, 2024 • edited Loading

spowelljr commented Jan 30, 2024

travier commented Jan 31, 2024

spowelljr commented Jan 31, 2024

spowelljr commented Feb 13, 2024 • edited Loading

spowelljr commented Feb 13, 2024

spowelljr commented Feb 13, 2024

travier commented Feb 14, 2024 • edited Loading

spowelljr commented Feb 14, 2024 • edited Loading

spowelljr commented Feb 15, 2024

travier commented Feb 15, 2024 • edited Loading

travier commented Feb 22, 2024

spowelljr commented Feb 22, 2024

minikube-bot commented Feb 23, 2024

spowelljr commented Feb 23, 2024

minikube-pr-bot commented Feb 23, 2024

minikube-pr-bot commented Feb 23, 2024

travier commented Feb 24, 2024

spowelljr commented Feb 26, 2024

spowelljr commented Feb 28, 2024

k8s-ci-robot commented Feb 28, 2024

spowelljr commented Feb 28, 2024

travier commented Feb 29, 2024

travier commented Jan 22, 2024 •

edited

Loading

spowelljr commented Jan 22, 2024 •

edited

Loading

travier commented Jan 28, 2024 •

edited

Loading

travier commented Jan 30, 2024 •

edited

Loading

spowelljr commented Feb 13, 2024 •

edited

Loading

travier commented Feb 14, 2024 •

edited

Loading

spowelljr commented Feb 14, 2024 •

edited

Loading

travier commented Feb 15, 2024 •

edited

Loading