-
Notifications
You must be signed in to change notification settings - Fork 83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix an issue where getReadyIngresses was not loading ready ingresses at startup #1079
Fix an issue where getReadyIngresses was not loading ready ingresses at startup #1079
Conversation
Indeed, we can't use label selector as kourier class is not a label, but an annotation. As a result, kourier does not sync ingresses at startup
Codecov Report
@@ Coverage Diff @@
## main #1079 +/- ##
=======================================
Coverage 80.95% 80.95%
=======================================
Files 18 18
Lines 1355 1355
=======================================
Hits 1097 1097
Misses 205 205
Partials 53 53 |
@@ -59,6 +59,10 @@ spec: | |||
value: "kourier-system" | |||
- name: ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID | |||
value: "false" | |||
- name: KUBE_API_BURST |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For reviewers: this is based on #1066, but I couldn't find any doc on these environment variables; is there a doc somewhere that I've missed? If so, I think it would be worth adding a small comment there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we should add a small comment. Would you like to add a comment? Or I can send a follow-up PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done ✔️ 8af7a3f. Feel free to tell me if comment is not clear enough 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! Looks good to me.
/lgtm Thank you so much @norbjd Please feel free to unhold if you will not include the comment about |
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: norbjd The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/cherry-pick release-1.11 |
@nak3: once the present PR merges, I will cherry-pick it on top of release-1.11 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/lgtm |
@nak3: new pull request created: #1081 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Changes
getReadyIngresses
not loading ingresses at startup/kind bug
I've noticed that with latest kourier release (1.11.0), kourier was not loading at all ready ingresses at startup. This was confirmed by looking at the logs, after restarting the controller on a cluster with a few ready ingresses:
This log is generated by https://github.com/knative-sandbox/net-kourier/blob/knative-v1.11.0/pkg/reconciler/ingress/controller.go#L210. I would have expected to see
Priming the config with 100 ingresses
, as my cluster contained 100 ingresses.After investigation, it turns out that the PR #1075 introduced a regression on
getReadyIngresses
method (https://github.com/knative-sandbox/net-kourier/blob/knative-v1.11.0/pkg/reconciler/ingress/controller.go#L313-L327):This can't work, because we're passing an annotation (
networking.knative.dev/ingress.class: kourier.ingress.networking.knative.dev
), whileList
method is actually filtering using labels. As a result, thisList
operation will always return an empty slice.So technically, #1075 solves the issue where kourier takes too long to start, but it's due to a bug: since there are always 0 ingress returned, there is no initial reconciliation; thus, the xds management server is started very quickly, and the readiness probe succeeds, marking kourier as ready pretty fast.
This PR aims to solves this bug. But, as I've rollbacked to the previous version of
getReadyIngresses
, I've "reintroduced" the slow startup issue (because it's loading all ingresses, as it should do I guess). This is why I've also copied here the changes done in #1066 (on kube api QPS and burst params).With these 2 changes, kourier loads all ready ingresses at startup and is pretty quick to start.
I'm not sure if introducing
KUBE_API_{BURST,QPM}
on this PR is a good thing or not; but, I've done so because otherwise, I couldn't even restart kourier on a kind cluster with 500 ingresses (got errors, probably because I was rate limited by Kubernetes API).Tests
With these changes, on a Kind cluster with 500 ingresses, Kourier took around 1s to restart and to be ready (after reloading the 500 ingresses, I can see the correct log
Priming the config with 500 ingresses
).It's worth noting that increasing
KUBE_API_{BURST,QPM}
has a side effect on kourier in general: it allows to speed up the ingresses creation. I've tried creating 500 ingresses on a Kind cluster (in parallel, with a concurrency limit of 20), here are the results:main
branch: it took 8m18s to create the ingresses. Creating 1 ingress took around 20s, as I were creating 20 ingresses at a time in parallelKUBE_API_{BURST,QPM}=200
): it takes 20s (25x less) to create all 500 ingresses, with the same parallelismRelease Note