feat: cannonical label for skipping pod identity webhook

[DerekTBrown]

Adds a condition to the mutating webhook, preventing pods from being processed with a particular label.

See: #215

[prateekgogia]

I am fine with this change, would love to hear @micahhausler opinion.

[DerekTBrown]

@micahhausler could you take a look at this?

[DerekTBrown]

@prateekgogia Can I have you re-review this?

[DanielCKennedy]

Can we change this to “eks.amazonaws.com/skip-pod-identity-webhook”. Identity webhook may be too generic if EKS adds more identity related webhooks in the future.

[DerekTBrown]

@DanielCKennedy makes sense! Fixed!

[prateekgogia]

I looked into this last night, EKS users should be able to modify the mutating webhook in their cluster to skip certain pods (the change in PR).

Another option to achieve this is by using this existing annotation supported by webhook eks.amazonaws.com/skip-containers

Do any of these option help achieving the desired output?

[DerekTBrown]

Hey @prateekgogia! 👋🏻 Thanks for looking at this!

This discussion is a bit nuanced- I've tried my best to capture this in writing, but we can also discuss synchronously if that would be easier.

I looked into this last night, EKS users should be able to modify the mutating webhook in their cluster to skip certain pods (the change in PR).

There are a few reasons the PR is needed to solve this problem:

1. Kubernetes users at scale use IaC for managing Kubernetes resources. Users typically don't create these IaC manifests themselves, they instead rely on open-source Helm Charts, or addon/plugin managers (for example, we use Kops). At a minimum, applying your suggested change would require the user to own a fork of these manifests/helm charts (Helm Chart for Deployments #4) indefinitely. In our case, it would actually require we own a fork of the Kops codebase to maintain this one-line change.

2. The goal of this PR and related issue (Standard Label for Excluding Pod from Identity Mounting #215) is to create a set of canonical annotations that can be used to skip the pod-identity-webhook so that users don't need to figure this our manually. For example, Cilium should always skip the pod identity webhook, both because it doesn't need EKS identities, and because the pod-identity-webhook depends on the CNI being setup (thus creating a circular dependency). Therefore, the canonical annotation could be added to Cilium upstream repository to prevent this circular dependency for all users (instead of each user having to find their own solution).

Another option to achieve this is by using this existing annotation supported by webhook eks.amazonaws.com/skip-containers.

Unfortunately, this isn't a workable solution because of the circular dependency issue described here. It appears eks.amazonaws.com/skip-containers is implemented inside the logic of the eks-pod-identity-webhook itself, but the issue here is that container doesn't start because it has no network connectivity. We need the ability for Kubernetes to skip calling the webhook, since it is non-responsive.

[prateekgogia]

LGTM!

[modulitos]

@DerekTBrown this change is now live across EKS.

Thank you for the contribution!