Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minikube addons enable ingress fails on Minikube v1.33.0 #18738

Closed
mhrheaume opened this issue Apr 24, 2024 · 9 comments · Fixed by #18779
Closed

minikube addons enable ingress fails on Minikube v1.33.0 #18738

mhrheaume opened this issue Apr 24, 2024 · 9 comments · Fixed by #18779

Comments

@mhrheaume
Copy link

What Happened?

This is on a brand new Minikube:

$ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]

Looking at the pod it appears there is a problem pulling the image:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-84df5799c-ld4hn
Name:             ingress-nginx-controller-84df5799c-ld4hn
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.6
Start Time:       Tue, 23 Apr 2024 20:13:36 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=84df5799c
Annotations:      <none>
Status:           Pending
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-84df5799c
Containers:
  controller:
    Container ID:
    Image:         registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
    Image ID:
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-84df5799c-ld4hn (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lqq2s (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-lqq2s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    12m                  default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-84df5799c-ld4hn to minikube
  Warning  FailedMount  12m (x3 over 12m)    kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      10m (x4 over 12m)    kubelet            Pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"
  Warning  Failed       10m (x4 over 11m)    kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c": failed to register layer: lsetxattr security.capability /nginx-ingress-controller: operation not supported
  Warning  Failed       10m (x4 over 11m)    kubelet            Error: ErrImagePull
  Warning  Failed       10m (x5 over 11m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff      119s (x39 over 11m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c"

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

QEMU

@mhrheaume
Copy link
Author

FYI, I just tried this on v1.32.0 and it works:

$ kubectl -n ingress-nginx describe pod ingress-nginx-controller-7c6974c4d8-wcm5v
Name:             ingress-nginx-controller-7c6974c4d8-wcm5v
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.105.7
Start Time:       Tue, 23 Apr 2024 20:32:21 -0700
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=7c6974c4d8
Annotations:      <none>
Status:           Running
IP:               10.244.0.5
IPs:
  IP:           10.244.0.5
Controlled By:  ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  docker://a77144fb810e8b5f6abb892471c476f3ba61942b8efcbd352ce04a2fa99798c7
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 23 Apr 2024 20:32:31 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-7c6974c4d8-wcm5v (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fx9dm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-fx9dm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From                      Message
  ----     ------       ----               ----                      -------
  Normal   Scheduled    44s                default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-wcm5v to minikube
  Warning  FailedMount  43s (x3 over 44s)  kubelet                   MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      40s                kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3"
  Normal   Pulled       34s                kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3" in 6.373s (6.373s including waiting)
  Normal   Created      34s                kubelet                   Created container controller
  Normal   Started      34s                kubelet                   Started container controller
  Normal   RELOAD       32s                nginx-ingress-controller  NGINX reload triggered due to a change in configuration

@humcqc
Copy link

humcqc commented Apr 24, 2024

seems telepresence helm install have the same issue

@medyagh
Copy link
Member

medyagh commented Apr 30, 2024

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.

 minikube start --driver=qemu2 --container-runtime=containerd

@spowelljr
Copy link
Member

spowelljr commented Apr 30, 2024

Based on AdguardTeam/AdGuardHome#6816, it seems like the issue should be specific to users using a VM driver on arm64. We don't have reliable VM arm64 tests so this would explain how this went undetected.

We have CONFIG_EXT4_FS_SECURITY enabled on amd64:

But it's missing from arm64, created #18779 to add it

@mhrheaume
Copy link
Author

@mhrheaume thank you I confirm I have this issue as well, on Qemu Driver (arm64)

Which driver are you using ?

interestingly it works with qemu driver and containerd runtime for me.

 minikube start --driver=qemu2 --container-runtime=containerd

My driver is also QEMU arm64 (M2 MacBook Pro).

@spowelljr
Copy link
Member

spowelljr commented May 1, 2024

Hi @mhrheaume, I've confirmed that my fix resolves the issue. We're going to make a patch release in the future but we don't have an exact timeline for it currently. In the meantime, you can pass this iso-url flag on start that will use the ISO from my fix and resolve your issue until the release is out.

minikube start --driver qmeu --iso-url="https://storage.googleapis.com/minikube-builds/iso/18779/minikube-v1.33.0-1714498396-18779-arm64.iso"

@spowelljr
Copy link
Member

Fix is included in latest release of minikube: https://github.com/kubernetes/minikube/releases/tag/v1.33.1

@AdamGoodApp
Copy link

I'm still having the same issue even with minikube version 1.33.1.

MacBook Pro M1

minikube start --cpus=4 --memory=8G --disk-size=20G --driver qemu --network socket_vmnet

minkube.log

@Underknowledge
Copy link

Fedora 40, version v1.33.1, similar problem.

tldr, the ingress-nginx pod cannot be scheduled due to the node affinity/selector not matching any available nodes.

Install

nix-shell -p "kubectl" "minikube"

minikube config set rootless true
minikube start --driver=podman --cpus='2'--memory=4096 --delete-on-failure=true 

minikube addons enable dashboard
minikube addons enable auto-pause
minikube addons enable ingress

the problem:

❯ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
🔎  Verifying ingress addon...




^C
(stuck)

Checking the deployment
minikube kubectl -- get deployment ingress-nginx-controller -n ingress-nginx -o yaml

nodeSelector:
  kubernetes.io/os: linux
  minikube.k8s.io/primary: "true"

while minikube kubectl -- get nodes --show-labels

NAME       STATUS   ROLES    AGE   VERSION   LABELS
minikube   Ready    <none>   15m   v1.30.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=minikube,kubernetes.io/os=linux

Fix:

minikube kubectl -- label node minikube minikube.k8s.io/primary=true

then another try:

❯ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

yaayy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants