CVE-2021-25742: Ingress-nginx custom snippets allows retrieval of ingress-nginx serviceaccount token and secrets across all namespaces

[cjcullen]

Issue Details

A security issue was discovered in ingress-nginx where a user that can create or update ingress objects can use the custom snippets feature to obtain all secrets in the cluster.

This issue has been rated High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:L), and assigned CVE-2021-25742.

Affected Components and Configurations

This bug affects ingress-nginx.

Multitenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.

Affected Versions with no mitigation

• v1.0.0
• <= v0.49.0

Versions allowing mitigation

This issue cannot be fixed solely by upgrading ingress-nginx. It can be mitigated in the following versions:

• v1.0.1
• v0.49.1

Mitigation

To mitigate this vulnerability:

1. Upgrade to a version that allows mitigation, (>= v0.49.1 or >= v1.0.1)

2. Set allow-snippet-annotations to false in your ingress-nginx ConfigMap based on how you deploy ingress-nginx:

   Static Deploy Files
   Edit the ConfigMap for ingress-nginx after deployment:

   kubectl edit configmap -n ingress-nginx ingress-nginx-controller
   

   Add directive:

   data:
     allow-snippet-annotations: “false”
   

   More information on the ConfigMap here

   Deploying Via Helm
   Set controller.allowSnippetAnnotations to false in the Values.yaml or add the directive to the helm deploy:

   helm install [RELEASE_NAME] --set controller.allowSnippetAnnotations=false ingress-nginx/ingress-nginx
   

   https://github.com/kubernetes/ingress-nginx/blob/controller-v1.0.1/charts/ingress-nginx/values.yaml#L76

Detection

If you find evidence that this vulnerability has been exploited, please contact security@kubernetes.io
Additional Details
See ingress-nginx Issue #126811 for more details.

Acknowledgements

This vulnerability was reported by Mitch Hulscher.

Thank You,
CJ Cullen on behalf of the Kubernetes Security Response Committee

[k8s-ci-robot]

@cjcullen: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rikatz]

/assign

[rikatz]

Fixed in 0.X branch -> kubernetes/ingress-nginx#7666
Fixed in v1.X branch -> kubernetes/ingress-nginx#7670

[paolomainardi]

I don't know if it related but our ingresses stopped to work 2 hours ago (more or less) with:

Error: UPGRADE FAILED: cannot patch "develop-website-ingress" with kind Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": service "ingress-nginx-controller-admission" not found

And the service was not in place at all, just the ingress-nginx-admission was present, i had to manually delete it, the fun fact is that the ingress-nginx installation had the admission webhook disabled, still trying to face what happened here.

GKE - 1.19.13-gke.1200

[iamNoah1]

@paolomainardi I don't think so. The CVE says, that non admin users who create or update ingress nginx instances, could possibly all secrets of the cluster. Sound more like a bug TBH. Feel free to raise an issue :)

[paolomainardi]

@iamNoah1 yes i guess so, but you know the timing was so close that i thought it was worth sharing, thanks :)

[Typositoire]

Might be obvious to others but there's an actually FIX coming right ? The final fix isn't removing a feature ?

[dezmodue]

Hi, I would appreciate some clarification.

Is the ability to retrieve "secrets across all namespaces" a consequence of "Ingress-nginx custom snippets allows retrieval of ingress-nginx serviceaccount token"?

In other words, the vulnerability allows a user to access the serviceaccount token and as a result of that the service token can then be used to access any resource that the token is allowed?

[raesene]

@dezmodue I had a look at this issue today, and there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved). There may be other attack paths as well, but that is one of them.

[longwuyuan]

Can you kindly check if disabling snippets helps. Thanks, ; Long

…

On Fri, 22 Oct, 2021, 8:03 PM Rory McCune, ***@***.***> wrote: @dezmodue <https://github.com/dezmodue> I had a look at this issue today, and there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved). There may be other attack paths as well, but that is one of them. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#126811>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWXORRCOBXHFRRBPJJDUIFY4DANCNFSM5GOP5DSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dezmodue]

Hi @raesene, thanks for your reply.

In some setups the ingress controller is "namespaced" and the service account is only allowed to get secrets that belong to the same namespace. The users with access to the namespace are the ones managing the ingress controller and have access to the token already so this seems to be a non issue in this scenario.
Is my understanding correct?

[dwertent]

Well Kubescape already came out with a control that will detect whether your cluster has this vulnerability

[raesene]

@dezmodue So for the case I found, I think so. Basically the snippets facility seems to provide effectively full control over the nginx config and access to the ingress-controller container, so if those are effectively not issues (i.e. all the resources owned by the ingress controller are already owned by the individual teams) then it doesn't seem like there'd be a big impact (AFAIK and I'm not one of the devs, just a curious security person :) )

[roy-work]

What is the actual vuln.?

See ingress-nginx Issue #126811 for more details.

That link is to this issue, i.e., it links to itself "for more details".

[rikatz]

Hi folks, I'm gonna try to answer all your questions here, and we are also scheduling an office hours to explain about this vulnerability and why you should take care of it.:

@Typositoire: Might be obvious to others but there's an actually FIX coming right ? The final fix isn't removing a feature ?
We didn't discussed actually how to fix this, and it may be hard. The main problem is that the custom snippet annotations allows users to add code that might be executed by Nginx (via Lua) so we should think in a way to map and drop those actions, or create maybe a "safe/non safe" directives that may be dropped when used in annotations.

@dezmodue Is the ability to retrieve "secrets across all namespaces" a consequence of "Ingress-nginx custom snippets allows retrieval of ingress-nginx serviceaccount token"?
Yes. Ingress nginx demands a full secret access on your cluster, as the TLS certificates that you may point in your ingress object are secrets. So using ingress-nginx service account allows you to query any secret in your cluster (and other stuff, take a look into the RBAC file)

@raesene So for the case I found, I think so. Basically the snippets facility seems to provide effectively full control over the nginx config and access to the ingress-controller container, so if those are effectively not issues (i.e. all the resources owned by the ingress controller are already owned by the individual teams) then it doesn't seem like there'd be a big impact (AFAIK and I'm not one of the devs, just a curious security person :) )

Correct. If you trust users with access to the ingress object there is no big deal. The problem here is, if you share a cluster, and some user uses the custom annotation to add random code that can be used to exfiltrate secrets (or do other stuff).

@roy-work What is the actual vuln.?

This is the actual vuln. You can use custom snippets to run arbitrary code and exfiltrate secrets from the container running ingress nginx.

Got a question in Slack if disabling snippets will disable all of the directives (modsecurity, etc).
The answer for this is in here but explaining:

• If a USER tries to use some snippet annotation (configuration-snippet, server-snippet, modsecurity-snippet, externalauth-snippet) that annotation gets dropped
• If the cluster admin uses the configmap snippet (like https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#main-snippet) then it works. This is because we TRUST on the ingress admin (and the definition to enable annotations is already inside this configmap anyway).

[roy-work]

@roy-work What is the actual vuln.?

This is the actual vuln. You can use custom snippets to run arbitrary code and exfiltrate secrets from the container running ingress nginx.

@rikatz I should have been more precise, I guess. I surmised that it was probably the case that some sort of code execution was occurring (others are guessing that that is used to access the service account credentials, and from there, "it's obvious" suffices). My understanding of snippets is that they're nginx configurations; how is it that I can run code from an nginx configuration? (Does nginx have a directive that amounts to an exec(2)?)

[Typositoire]

@roy-work

The main problem is that the custom snippet annotations allows users to add code that might be executed by Nginx (via Lua) so we should think in a way to map and drop those actions, or create maybe a "safe/non safe" directives that may be dropped when used in annotations.

With

Yes. Ingress nginx demands a full secret access on your cluster, as the TLS certificates that you may point in your ingress object are secrets. So using ingress-nginx service account allows you to query any secret in your cluster (and other stuff, take a look into the RBAC file)

It's not split from the original questions, I guess this was his answer :p

[rikatz]

@roy-work I prefer in this case not enter in details, just bare in mind that NGINX does not have this directive, but we use some other modules that may allow the vulnerability written here :)

[julienlau]

@rikatz could you please detail which other vulnerability is present when using the latest version with --enable-snippets=true ?

It is not very clear whether the character filtering solves this vuln or if the fix is char-filtering & enable-snippets=false.

@foxylion unfortunately not. You can still block the usage of *snippets, but there;s another vulnerability in main ingress object that should be dealt with. What version are you?

I'm closing this issue as it's already fixed in the latest version, and we are planning to deprecate the legacy version so I highly recommend migrating to ingress v1.X

Thanks! /close

[cji]

/transfer kubernetes

[cji]

/area security
/kind bug
/committee security-response
/triage accepted
/lifecycle frozen
/label official-cve-feed