Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

antrea/antrea-ubi:v1.14.0 image is broken #5722

Closed
antoninbas opened this issue Nov 17, 2023 · 3 comments · Fixed by #5723
Closed

antrea/antrea-ubi:v1.14.0 image is broken #5722

antoninbas opened this issue Nov 17, 2023 · 3 comments · Fixed by #5723
Assignees
@antoninbas
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. reported-by/end-user Issues reported by end users.

Comments

@antoninbas
Copy link
Contributor

Describe the bug
The UBI-based image for the v1.14.0 release is unusable. Antrea Pods go into CrashLoopBackOff state immediately, with the following error:

antrea-agent: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by antrea-agent)
antrea-agent: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by antrea-agent)

Thanks Marcus for reporting this issue on our Slack channel.

To Reproduce
Deploy Antrea with the antrea/antrea-ubi:v1.14.0 image instead of the antrea/antrea-ubuntu:v1.14.0 image.

Versions:
Antrea v1.14.0

Additional context
According to @salv-orlando, simply rebuilding the image should resolve the issue.
@antoninbas antoninbas added kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Nov 17, 2023
@antoninbas antoninbas self-assigned this Nov 17, 2023
@antoninbas
Copy link
Contributor Author

A rebuild is not a solution. This is not a transient error.

In Antrea v1.14, we upgraded from Go 1.19 to Go 1.21. Go 1.21 includes the following change:

On Linux, the linker now selects the dynamic interpreter for glibc or musl at link time.

The effect can be a bit unclear just based on this statement, but more details can be found there:

What this means is that when compiling with cgo (which we do for the antrea-agent and antrea-controller binaries), the runtime system needs to have a version of glibc greater or equal than the version of glibc present on the build system. Antrea uses golang:1.21 as the build container, and registry.access.redhat.com/ubi8 as the base image for the final container. golang:1.21 includes a recent version of glibc (2.36), while registry.access.redhat.com/ubi8 comes with a very old version from 2018 (2.28).

We do not have this issue for antrea/antrea-ubuntu:v1.14.0, even though it seems that the image includes glibc 2.35. There seems to be some acceptable margin. But we may have the same issue there in the future.

There are 2 ways to resolve this:

  1. disable cgo (CGO_ENABLED=0) for all binaries

  2. use the same container image for building as for running

  3. may be the right thing to do longterm, but I feel like we need to spend more time investigating it

  • any impact on binary size
  • libc is used for name resolution (net package) under certain conditions, but does that really apply to us

I will use 2) for now to solve the UBI image issue. I will also open a separate issue to investigate 1).

antoninbas added a commit to antoninbas/antrea that referenced this issue Nov 17, 2023
@antoninbas
Fix antrea-ubi image build
ab8d0ec 
Starting with Go 1.21, the runtime system needs to have a version of
glibc "compatible" with the version available at build time. The ubi8
image comes with a very old version of glibc compared to the golang
builder, and the Antrea binaries no longer run.

To address this, we use the ubi8 image to build Antrea, which means we
need to install Go manually.

Fixes antrea-io#5722

Signed-off-by: Antonin Bas <abas@vmware.com>
This was referenced Nov 17, 2023
Merged
Closed
Closed
antoninbas added a commit that referenced this issue Nov 17, 2023
@antoninbas
Fix antrea-ubi image build (#5723)
2afab06 
Starting with Go 1.21, the runtime system needs to have a version of
glibc "compatible" with the version available at build time. The ubi8
image comes with a very old version of glibc compared to the golang
builder, and the Antrea binaries no longer run.

To address this, we use the ubi8 image to build Antrea, which means we
need to install Go manually.

Fixes #5722

Signed-off-by: Antonin Bas <abas@vmware.com>
antoninbas added a commit to antoninbas/antrea that referenced this issue Nov 17, 2023
@antoninbas
Fix antrea-ubi image build
3aac460 
Starting with Go 1.21, the runtime system needs to have a version of
glibc "compatible" with the version available at build time. The ubi8
image comes with a very old version of glibc compared to the golang
builder, and the Antrea binaries no longer run.

To address this, we use the ubi8 image to build Antrea, which means we
need to install Go manually.

Fixes antrea-io#5722

Signed-off-by: Antonin Bas <abas@vmware.com>
@antoninbas
Copy link
Contributor Author

Re-opening until we have a published working tag

@antoninbas antoninbas reopened this Nov 17, 2023
antoninbas added a commit to antoninbas/antrea that referenced this issue Nov 17, 2023
@antoninbas
Fix antrea-ubi image build
a8b33ba 
Starting with Go 1.21, the runtime system needs to have a version of
glibc "compatible" with the version available at build time. The ubi8
image comes with a very old version of glibc compared to the golang
builder, and the Antrea binaries no longer run.

To address this, we use the ubi8 image to build Antrea, which means we
need to install Go manually.

Fixes antrea-io#5722

Signed-off-by: Antonin Bas <abas@vmware.com>
@antoninbas antoninbas mentioned this issue Nov 17, 2023
Closed
tnqn pushed a commit that referenced this issue Nov 20, 2023
@antoninbas @tnqn
Fix antrea-ubi image build
04d4041 
Starting with Go 1.21, the runtime system needs to have a version of
glibc "compatible" with the version available at build time. The ubi8
image comes with a very old version of glibc compared to the golang
builder, and the Antrea binaries no longer run.

To address this, we use the ubi8 image to build Antrea, which means we
need to install Go manually.

Fixes #5722

Signed-off-by: Antonin Bas <abas@vmware.com>
@antoninbas
Copy link
Contributor Author

Can close this now

  • antrea/antrea-ubi:v1.14.1 and projects.registry.vmware.com/antrea/antrea-ubi:v1.14.1 have been published as part of the v1.14.1 release
  • antrea/antrea-ubi:v1.14.0 and projects.registry.vmware.com/antrea/antrea-ubi:v1.14.0 have been deleted from their respective registries
  • the Github release notes for v1.14.0 release have been updated with the necessary information: https://github.com/antrea-io/antrea/releases/tag/v1.14.0

@tnqn tnqn added the reported-by/end-user Issues reported by end users. label Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@antoninbas antoninbas

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. reported-by/end-user Issues reported by end users.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix antrea-ubi image build antoninbas/antrea
2 participants
@antoninbas @tnqn