Customizable Reinvocation Policy for the Amazon EKS Pod Identity Webhook during Cluster Creation

[sparsh-95]

What would you like to be added:
I would like to propose a feature request for the Amazon EKS Pod Identity Webhook to establish a consistent default value for the reinvocationPolicy parameter and introduce a configuration option for the same during EKS cluster creation via any means (AWS Console, CLI, Terraform, CloudFormation, etc.).

Why is this needed:
Currently, the reinvocationPolicy parameter of the webhook has a random default value of either "Never" or "IfNeeded" during EKS cluster creation, even though Kubernetes states here it should default to Never. At least that is what we have been experiencing in all EKS clusters >= 1.23 being provisioned. This randomness can lead to confusion and difficulty in managing the webhook's reinvocation behavior.

By introducing a configurable default value, users can define a consistent behavior for the reinvocationPolicy parameter that aligns with their specific requirements. This provides clarity and predictability when configuring the webhook.

Further by having a configuration option during EKS cluster creation to allow users to explicitly set the desired reinvocationPolicy for the webhook will give users the flexibility to define the reinvocation behavior based on their specific requirements.

Overall, these enhancements provide greater control and flexibility in managing the reinvocation policy of the Amazon EKS Pod Identity Webhook, leading to improved usability and easier management of EKS clusters using the webhook.

[dims]

@sparsh-95 when this happens next, can you please open a support ticket for us to investigate?

(Also a feature request like this is better for https://github.com/aws/containers-roadmap repository)