-
Notifications
You must be signed in to change notification settings - Fork 593
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Kong Ingress Controller Chart] High Load on k8s API with KongUpstreamPolicy and Multiple Ingress Controllers #6270
Comments
Problem: Analysis: This causes a lot of API request to k8s API and the Kong admin API as ALL ingress controllers in a single k8s cluster are fighting over the same resource, even when not responsible. The reason why this anyway works when translating KongUpsteramPolicies, is as in https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/dataplane/kongstate/kongstate.go#L243 only services where the current ingress controller is responsible for are available and the service namespace is used at https://github.com/Kong/kubernetes-ingress-controller/blob/main/internal/dataplane/kongstate/kongupstreampolicy.go#L58 for fetching the Suggestion: Does this sound right? This would leave at least one question open: If for some reason the reference to a PR: |
Don't know if the |
We where now forced to take action but the suggested workaround by using
We are testing now if we can use something like
and bind this role only to services accounts responsible for the resources. This is cumbersome and needs whitelist management of resources. Getting back if this worked. |
The above workaround worked by removing the first apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kong-ingress-namespaced-dev
rules:
- apiGroups:
- configuration.konghq.com
resources:
- kongupstreampolicies
verbs:
- get
- list
- watch
- apiGroups:
- configuration.konghq.com
resources:
- kongupstreampolicies/status
verbs:
- get
- apiGroups:
- configuration.konghq.com
resources:
- kongupstreampolicies/status
resourceNames:
- ch-kong-dev-session-stickiness
verbs:
- patch
- update This is still throwing errors like {"level":"error","time":"2024-07-29T11:09:19Z","logger":"controllers.KongUpstreamPolicy","msg":"Reconciler error","reconcileID":"2998f359-de26-4e0a-bb06-680521652fd4","error":"kongupstreampolicies.configuration.konghq.com \"ch-kong-int-session-stickiness\" is forbidden: User \"system:serviceaccount:ch-kong-test:kong-serviceaccount\" cannot patch resource \"kongupstreampolicies/status\" in API group \"configuration.konghq.com\" in the namespace \"ch-kong-httpbin-int\""} But this error happens only once every few minutes (around 5m) and the load on the k8s API server is reduced. |
@dweber019 Thanks for pointing it out and your work on resolving it. I'll create the PR following your approach. |
Awesome thanks @randmonkey. |
@dweber019 PR is here: #6421. We will include it in the upcoming KIC 3.3.0 release. |
Is there an existing issue for this?
Current Behavior
When using UpstreamPolicy in a Kubernetes cluster with two ingress controllers, there is a cyclic configuration overwrite. Both ingress controllers compete for the UpstreamPolicy resource, leading to high load on the k8s API.
Expected Behavior
The ingress controllers should handle the UpstreamPolicy resource without cyclic overwrites, avoiding unnecessary load on the k8s API.
Steps To Reproduce
Kong Ingress Controller version
Kubernetes version
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: